2021-11-01 03:29:52 +00:00
|
|
|
Quality definition
|
|
|
|
|
==================
|
2021-10-31 11:53:58 +00:00
|
|
|
|
|
|
|
|
The **martiLQ** definition allows for the inclusion of load quality
|
|
|
|
|
definitions. The load quality definition is intended to be
|
|
|
|
|
able to be applied universally with common tools. Not
|
|
|
|
|
all needs are covered with the base definition but can be extended.
|
|
|
|
|
|
2021-11-01 03:29:52 +00:00
|
|
|
Defined load quality metrics
|
|
|
|
|
----------------------------
|
2021-10-31 11:53:58 +00:00
|
|
|
|
|
|
|
|
* Sequential batch number - This is a decimal number defined at the **martiLQ** definition
|
|
|
|
|
and applies to all resources. The integer portion is for new batches and the fraction
|
|
|
|
|
part can be used for issues with the same data extract. such as requiring resend because
|
|
|
|
|
a resource was missing.
|
|
|
|
|
|
|
|
|
|
* Number of records in the document - This is the number of data primary records not the
|
|
|
|
|
count of end of lines and is agreed between parties. XML record counts could be based
|
|
|
|
|
on the number of primary segments under root. JSON records can be counted in a similar way.
|
|
|
|
|
The headers or trailling records are not counted
|
|
|
|
|
|
2021-11-01 03:29:52 +00:00
|
|
|
Addresses deficiencies
|
|
|
|
|
----------------------
|
2021-10-31 11:53:58 +00:00
|
|
|
|
|
|
|
|
The **martiLQ** objective is to address deficiencies with alternative
|
|
|
|
|
data load quality approaches such as:
|
|
|
|
|
|
|
|
|
|
* magic formats in file names
|
|
|
|
|
* identifying the number of files
|
|
|
|
|
* knowing when all files are ready
|
|
|
|
|
* separate documentation that is unlinked
|
|
|
|
|
* securing the data
|
|
|
|
|
* adding footers to the data, requiring custom file handlers
|
|
|
|
|
|
2021-11-01 03:29:52 +00:00
|
|
|
Extending load quality metrics
|
|
|
|
|
------------------------------
|
2021-10-31 11:53:58 +00:00
|
|
|
|
|
|
|
|
**martLQ** framewokr and importantly is open to extension so that extra
|
|
|
|
|
load metrics appropriate to the situation can be included.
|
|
|
|
|
|
|
|
|
|
### Extension ideas
|
|
|
|
|
|
|
|
|
|
The following extensions for load quality can easily be included:
|
|
|
|
|
|
|
|
|
|
* Mandatory data in column
|
|
|
|
|
* Uniqueness of data values
|
|
|
|
|
* Data values are within defined tolerances
|
|
|
|
|
* Check for data exclusions
|
|
|
|
|
|
|
|
|
|
And all this information is included in the **martiLQ** definition
|
|
|
|
|
allowing for self describing load quality.
|
