Make MD compatible with RtD
parent
28fe1b006c
commit
dc44c0d660
|
|
@ -1,11 +1,13 @@
|
||||||
# Attribute definition
|
Attribute definition
|
||||||
|
====================
|
||||||
|
|
||||||
A Resource can list attributes related to the document / file.
|
A Resource can list attributes related to the document / file.
|
||||||
|
|
||||||
An attribute is a generic definition and conventions are
|
An attribute is a generic definition and conventions are
|
||||||
observed in the definitions that are captured here.
|
observed in the definitions that are captured here.
|
||||||
|
|
||||||
## Attribute definition
|
Attribute definition
|
||||||
|
--------------------
|
||||||
|
|
||||||
The Attribute consists of:
|
The Attribute consists of:
|
||||||
|
|
||||||
|
|
@ -29,4 +31,4 @@ number of records in the file for the given format.
|
||||||
"value": "9"
|
"value": "9"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,5 @@
|
||||||
# CKAN definition
|
CKAN definition
|
||||||
|
===============
|
||||||
|
|
||||||
The **martiLQ** has used similar terms and structures that are found in the
|
The **martiLQ** has used similar terms and structures that are found in the
|
||||||
CKAN API describing the resources. This similarity allows for simple mapping of values
|
CKAN API describing the resources. This similarity allows for simple mapping of values
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,5 @@
|
||||||
# Comparison of martiLQ document definition
|
Comparison of martiLQ document definition
|
||||||
|
=========================================
|
||||||
|
|
||||||
The use of metadata definitions is not unique and examples
|
The use of metadata definitions is not unique and examples
|
||||||
exist in many different situations. Some are standard and open
|
exist in many different situations. Some are standard and open
|
||||||
|
|
@ -22,7 +23,8 @@ a CKAN source into a **martiLQ** document definition and then process the data
|
||||||
through the pipeline as you would for any other data file that had a
|
through the pipeline as you would for any other data file that had a
|
||||||
**martiLQ** document definition.
|
**martiLQ** document definition.
|
||||||
|
|
||||||
## Benefit of CKAN and martiLQ
|
Benefit of CKAN and martiLQ
|
||||||
|
---------------------------
|
||||||
|
|
||||||
The CKAN is excellent at defining the data source details but it lacks information
|
The CKAN is excellent at defining the data source details but it lacks information
|
||||||
for load quality. If you have CKAN deployed in your organisation and wish
|
for load quality. If you have CKAN deployed in your organisation and wish
|
||||||
|
|
@ -31,7 +33,8 @@ CKAN and marti.
|
||||||
|
|
||||||
Samples exist on CKAN integration.
|
Samples exist on CKAN integration.
|
||||||
|
|
||||||
## Magda and martiLQ
|
Magda and martiLQ
|
||||||
|
-----------------
|
||||||
|
|
||||||
Another source of data is [Magda](https://magda.io/) which has API metadata
|
Another source of data is [Magda](https://magda.io/) which has API metadata
|
||||||
definitions. Magda is more about data federation and as such provides
|
definitions. Magda is more about data federation and as such provides
|
||||||
|
|
|
||||||
|
|
@ -13,7 +13,7 @@
|
||||||
# import os
|
# import os
|
||||||
# import sys
|
# import sys
|
||||||
# sys.path.insert(0, os.path.abspath('.'))
|
# sys.path.insert(0, os.path.abspath('.'))
|
||||||
|
from recommonmark.parser import CommonMarkParser
|
||||||
|
|
||||||
# -- Project information -----------------------------------------------------
|
# -- Project information -----------------------------------------------------
|
||||||
|
|
||||||
|
|
@ -43,12 +43,13 @@ exclude_patterns = [
|
||||||
'samples/powershell/test/*',
|
'samples/powershell/test/*',
|
||||||
]
|
]
|
||||||
|
|
||||||
source_suffix = {
|
source_parsers = {
|
||||||
'.rst': 'restructuredtext',
|
'.md': CommonMarkParser
|
||||||
'.txt': 'restructuredtext',
|
|
||||||
'.md': 'markdown',
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
source_suffix = ['.rst', '.md']
|
||||||
|
|
||||||
|
|
||||||
# -- Options for HTML output -------------------------------------------------
|
# -- Options for HTML output -------------------------------------------------
|
||||||
|
|
||||||
# The theme to use for HTML and HTML Help pages. See the documentation for
|
# The theme to use for HTML and HTML Help pages. See the documentation for
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,5 @@
|
||||||
# Custom Definition
|
Custom Definition
|
||||||
|
=================
|
||||||
|
|
||||||
The custome definition section allows the inclusion of extensions
|
The custome definition section allows the inclusion of extensions
|
||||||
to the standard. To demonstrate the inclusion, there are three
|
to the standard. To demonstrate the inclusion, there are three
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,5 @@
|
||||||
# Magda definitions
|
Magda definitions
|
||||||
|
=================
|
||||||
|
|
||||||
https://magda.io/
|
https://magda.io/
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,5 @@
|
||||||
# MartiLQ document
|
MartiLQ document
|
||||||
|
================
|
||||||
|
|
||||||
The metadata reconciliation transfer information is referred
|
The metadata reconciliation transfer information is referred
|
||||||
to as the **martiLQ** document throughout this documentation.
|
to as the **martiLQ** document throughout this documentation.
|
||||||
|
|
@ -6,9 +7,8 @@ to as the **martiLQ** document throughout this documentation.
|
||||||
The **martiLQ** document can be part of a message or a file
|
The **martiLQ** document can be part of a message or a file
|
||||||
in its own right. The definition is curently a Json file.
|
in its own right. The definition is curently a Json file.
|
||||||
|
|
||||||
|
Mandatory information
|
||||||
|
---------------------
|
||||||
### Mandatory information
|
|
||||||
|
|
||||||
The mandatory information is:
|
The mandatory information is:
|
||||||
|
|
||||||
|
|
@ -16,7 +16,8 @@ The mandatory information is:
|
||||||
* Unique identifier
|
* Unique identifier
|
||||||
* Resource list - See Resource section summary below or detailed document [Resource](resources.md)
|
* Resource list - See Resource section summary below or detailed document [Resource](resources.md)
|
||||||
|
|
||||||
### Optional information
|
Optional information
|
||||||
|
++++++++++++++++++++
|
||||||
|
|
||||||
The optional information is:
|
The optional information is:
|
||||||
|
|
||||||
|
|
@ -36,12 +37,14 @@ The optional information is:
|
||||||
* Custom list - List of custom entries, one being the **martiLQ** software details
|
* Custom list - List of custom entries, one being the **martiLQ** software details
|
||||||
see [custom](custom.md)
|
see [custom](custom.md)
|
||||||
|
|
||||||
### Information extension
|
Information extension
|
||||||
|
+++++++++++++++++++++
|
||||||
|
|
||||||
The information supplied can be extended by party agreement and there
|
The information supplied can be extended by party agreement and there
|
||||||
are place holders in the defintion.
|
are place holders in the defintion.
|
||||||
|
|
||||||
### Resource
|
Resource
|
||||||
|
--------
|
||||||
|
|
||||||
The resource section is a list of documents or files that are to be grouped
|
The resource section is a list of documents or files that are to be grouped
|
||||||
together are listed under the same **martiLQ** definition.
|
together are listed under the same **martiLQ** definition.
|
||||||
|
|
@ -59,7 +62,8 @@ items:
|
||||||
* Size of document - The document size in bytes
|
* Size of document - The document size in bytes
|
||||||
* URL - This can be ``file://``, ``https://``, ``ftp://``, etc resource location
|
* URL - This can be ``file://``, ``https://``, ``ftp://``, etc resource location
|
||||||
|
|
||||||
### Resource optional
|
Resource optional
|
||||||
|
+++++++++++++++++
|
||||||
|
|
||||||
The following are some of the optional items in the resource section. See [Resource](resources.md)
|
The following are some of the optional items in the resource section. See [Resource](resources.md)
|
||||||
for more details
|
for more details
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,5 @@
|
||||||
# MartiLQ objective
|
MartiLQ objective
|
||||||
|
=================
|
||||||
|
|
||||||
The objective of **martLQ** is to define a simle standard for
|
The objective of **martLQ** is to define a simle standard for
|
||||||
capturing the data files being transferred. It is not for
|
capturing the data files being transferred. It is not for
|
||||||
|
|
|
||||||
|
|
@ -1,11 +1,13 @@
|
||||||
# Quality definition
|
Quality definition
|
||||||
|
==================
|
||||||
|
|
||||||
The **martiLQ** definition allows for the inclusion of load quality
|
The **martiLQ** definition allows for the inclusion of load quality
|
||||||
definitions. The load quality definition is intended to be
|
definitions. The load quality definition is intended to be
|
||||||
able to be applied universally with common tools. Not
|
able to be applied universally with common tools. Not
|
||||||
all needs are covered with the base definition but can be extended.
|
all needs are covered with the base definition but can be extended.
|
||||||
|
|
||||||
## Defined load quality metrics
|
Defined load quality metrics
|
||||||
|
----------------------------
|
||||||
|
|
||||||
* Sequential batch number - This is a decimal number defined at the **martiLQ** definition
|
* Sequential batch number - This is a decimal number defined at the **martiLQ** definition
|
||||||
and applies to all resources. The integer portion is for new batches and the fraction
|
and applies to all resources. The integer portion is for new batches and the fraction
|
||||||
|
|
@ -17,7 +19,8 @@ all needs are covered with the base definition but can be extended.
|
||||||
on the number of primary segments under root. JSON records can be counted in a similar way.
|
on the number of primary segments under root. JSON records can be counted in a similar way.
|
||||||
The headers or trailling records are not counted
|
The headers or trailling records are not counted
|
||||||
|
|
||||||
## Addresses deficiencies
|
Addresses deficiencies
|
||||||
|
----------------------
|
||||||
|
|
||||||
The **martiLQ** objective is to address deficiencies with alternative
|
The **martiLQ** objective is to address deficiencies with alternative
|
||||||
data load quality approaches such as:
|
data load quality approaches such as:
|
||||||
|
|
@ -29,7 +32,8 @@ data load quality approaches such as:
|
||||||
* securing the data
|
* securing the data
|
||||||
* adding footers to the data, requiring custom file handlers
|
* adding footers to the data, requiring custom file handlers
|
||||||
|
|
||||||
## Extending load quality metrics
|
Extending load quality metrics
|
||||||
|
------------------------------
|
||||||
|
|
||||||
**martLQ** framewokr and importantly is open to extension so that extra
|
**martLQ** framewokr and importantly is open to extension so that extra
|
||||||
load metrics appropriate to the situation can be included.
|
load metrics appropriate to the situation can be included.
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,5 @@
|
||||||
# References
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
|
||||||
https://dex.dss.gov.au/sites/default/files/documents/2021-06/data-exchange-protocols-june-2021.pdf
|
https://dex.dss.gov.au/sites/default/files/documents/2021-06/data-exchange-protocols-june-2021.pdf
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,5 @@
|
||||||
# Resources definition
|
Resources definition
|
||||||
|
====================
|
||||||
|
|
||||||
The resources section defines the files that are grouped
|
The resources section defines the files that are grouped
|
||||||
together by association. This association is not defined but can
|
together by association. This association is not defined but can
|
||||||
|
|
@ -38,7 +39,8 @@ The following are optional in the resource section.
|
||||||
* Encryption - Type of encryption used if any
|
* Encryption - Type of encryption used if any
|
||||||
|
|
||||||
|
|
||||||
## Compression
|
Compression
|
||||||
|
-----------
|
||||||
|
|
||||||
Files can be compressed using a utility. A single compressed file can contain
|
Files can be compressed using a utility. A single compressed file can contain
|
||||||
multiple files. The **martiLQ** definition document applies to the compressed file
|
multiple files. The **martiLQ** definition document applies to the compressed file
|
||||||
|
|
@ -49,7 +51,8 @@ compressed file.
|
||||||
|
|
||||||
Compression of files always occur before encryption.
|
Compression of files always occur before encryption.
|
||||||
|
|
||||||
### martiLQ definition for Compressed File
|
martiLQ definition for Compressed File
|
||||||
|
++++++++++++++++++++++++++++++++++++++
|
||||||
|
|
||||||
For a compressed file that is not encrypted, the distribution definition will be:
|
For a compressed file that is not encrypted, the distribution definition will be:
|
||||||
|
|
||||||
|
|
@ -68,7 +71,8 @@ The reason for this approach is it allows a generic tool to be deployed to
|
||||||
check the validity of the contents without unpacking the received /fetched
|
check the validity of the contents without unpacking the received /fetched
|
||||||
file. That is you can perform load quality pipeline processing.
|
file. That is you can perform load quality pipeline processing.
|
||||||
|
|
||||||
## Encryption
|
Encryption
|
||||||
|
----------
|
||||||
|
|
||||||
The encryption of content is always applied after compression not before, if
|
The encryption of content is always applied after compression not before, if
|
||||||
you are not using the compression tool native encryption. WinZIP and 7ZIP
|
you are not using the compression tool native encryption. WinZIP and 7ZIP
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,5 @@
|
||||||
# What is marti
|
What is marti
|
||||||
|
=============
|
||||||
|
|
||||||
The foundation pillar for the **martiLQ** framework is the [martiLQ document](martiLQ.md)
|
The foundation pillar for the **martiLQ** framework is the [martiLQ document](martiLQ.md)
|
||||||
that defines the reconciliation and other metadata of the document / file being transferred.
|
that defines the reconciliation and other metadata of the document / file being transferred.
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,5 @@
|
||||||
# When would you use martiLQ
|
When would you use martiLQ
|
||||||
|
==========================
|
||||||
|
|
||||||
You are likely to start using the **martiLQ** framework when:
|
You are likely to start using the **martiLQ** framework when:
|
||||||
|
|
||||||
|
|
@ -11,7 +12,8 @@ If you already have a standard and it works for you, and you have no upcoming (l
|
||||||
initiative that would benefit from the framework, then stick with what you have. The benefits
|
initiative that would benefit from the framework, then stick with what you have. The benefits
|
||||||
of the framework are unlikely to weigh in your framework.
|
of the framework are unlikely to weigh in your framework.
|
||||||
|
|
||||||
## Read the material
|
Read the material
|
||||||
|
-----------------
|
||||||
|
|
||||||
Please read the material before jumping in and make sure the **martiLQ** framework will
|
Please read the material before jumping in and make sure the **martiLQ** framework will
|
||||||
provide you with the expected benefits. Implementing the framework and then changing
|
provide you with the expected benefits. Implementing the framework and then changing
|
||||||
|
|
|
||||||
|
|
@ -1,11 +1,13 @@
|
||||||
# Who is likely to use martiLQ
|
Who is likely to use martiLQ
|
||||||
|
============================
|
||||||
|
|
||||||
You are likely to find the **martiLQ** framework relevant if you:
|
You are likely to find the **martiLQ** framework relevant if you:
|
||||||
|
|
||||||
1. Have many document exchanges, such as End of Day batches
|
1. Have many document exchanges, such as End of Day batches
|
||||||
2. Need to verify or reconcile the documents
|
2. Need to verify or reconcile the documents
|
||||||
|
|
||||||
## Data exchanges
|
Data exchanges
|
||||||
|
--------------
|
||||||
|
|
||||||
If you are creating or receiving many documents or files on a regular basis
|
If you are creating or receiving many documents or files on a regular basis
|
||||||
then you probably have some framework defined. The framework may be as simple as:
|
then you probably have some framework defined. The framework may be as simple as:
|
||||||
|
|
@ -22,7 +24,8 @@ Simple framework such as the above have limitations, such as:
|
||||||
* Lower automation prospects and alignment to DataSecOps
|
* Lower automation prospects and alignment to DataSecOps
|
||||||
* Poor fit to web applications (they tend to be designed for FTP and LAN)
|
* Poor fit to web applications (they tend to be designed for FTP and LAN)
|
||||||
|
|
||||||
## Framework Sidecar files
|
Framework Sidecar files
|
||||||
|
-----------------------
|
||||||
|
|
||||||
The **martiLQ** framework addresses the issues and limitations by using sidecar
|
The **martiLQ** framework addresses the issues and limitations by using sidecar
|
||||||
or shadow files. The [concept of sidecar files](https://en.wikipedia.org/wiki/Sidecar_file) is
|
or shadow files. The [concept of sidecar files](https://en.wikipedia.org/wiki/Sidecar_file) is
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,5 @@
|
||||||
# Why use marti
|
Why use marti
|
||||||
|
=============
|
||||||
|
|
||||||
**martiLQ** is a framework for providing a degree of auditability and reconciliation of
|
**martiLQ** is a framework for providing a degree of auditability and reconciliation of
|
||||||
documents transferred between systems in an organisation and externally. It does not intend
|
documents transferred between systems in an organisation and externally. It does not intend
|
||||||
|
|
@ -13,7 +14,8 @@ define the format or content of the document. It defines controls that can be u
|
||||||
|
|
||||||
You would use **martiLQ** if any of the controls are a requirement for you.
|
You would use **martiLQ** if any of the controls are a requirement for you.
|
||||||
|
|
||||||
## Documents
|
Documents
|
||||||
|
---------
|
||||||
|
|
||||||
Documents in this context are digital storage objects such as operating system files,
|
Documents in this context are digital storage objects such as operating system files,
|
||||||
cloud storage objects or blobs. The document content can have structure and contains multiple
|
cloud storage objects or blobs. The document content can have structure and contains multiple
|
||||||
|
|
@ -25,7 +27,8 @@ in single web transactions. It is for providing controls when moving large amou
|
||||||
data as one event. This data are commonly referred to as batch extracts and performed
|
data as one event. This data are commonly referred to as batch extracts and performed
|
||||||
at scheduled times such as end of day.
|
at scheduled times such as end of day.
|
||||||
|
|
||||||
## Security
|
Security
|
||||||
|
--------
|
||||||
|
|
||||||
The framework does not replace your security, inflight encryption or encryption at rest.
|
The framework does not replace your security, inflight encryption or encryption at rest.
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue