Make MD compatible with RtD

draft_specifications
meerkat 2021-11-01 14:29:52 +11:00
parent 28fe1b006c
commit dc44c0d660
15 changed files with 73 additions and 41 deletions

View File

@ -1,11 +1,13 @@
# Attribute definition Attribute definition
====================
A Resource can list attributes related to the document / file. A Resource can list attributes related to the document / file.
An attribute is a generic definition and conventions are An attribute is a generic definition and conventions are
observed in the definitions that are captured here. observed in the definitions that are captured here.
## Attribute definition Attribute definition
--------------------
The Attribute consists of: The Attribute consists of:
@ -29,4 +31,4 @@ number of records in the file for the given format.
"value": "9" "value": "9"
} }
] ]
``` ```

View File

@ -1,4 +1,5 @@
# CKAN definition CKAN definition
===============
The **martiLQ** has used similar terms and structures that are found in the The **martiLQ** has used similar terms and structures that are found in the
CKAN API describing the resources. This similarity allows for simple mapping of values CKAN API describing the resources. This similarity allows for simple mapping of values

View File

@ -1,4 +1,5 @@
# Comparison of martiLQ document definition Comparison of martiLQ document definition
=========================================
The use of metadata definitions is not unique and examples The use of metadata definitions is not unique and examples
exist in many different situations. Some are standard and open exist in many different situations. Some are standard and open
@ -22,7 +23,8 @@ a CKAN source into a **martiLQ** document definition and then process the data
through the pipeline as you would for any other data file that had a through the pipeline as you would for any other data file that had a
**martiLQ** document definition. **martiLQ** document definition.
## Benefit of CKAN and martiLQ Benefit of CKAN and martiLQ
---------------------------
The CKAN is excellent at defining the data source details but it lacks information The CKAN is excellent at defining the data source details but it lacks information
for load quality. If you have CKAN deployed in your organisation and wish for load quality. If you have CKAN deployed in your organisation and wish
@ -31,7 +33,8 @@ CKAN and marti.
Samples exist on CKAN integration. Samples exist on CKAN integration.
## Magda and martiLQ Magda and martiLQ
-----------------
Another source of data is [Magda](https://magda.io/) which has API metadata Another source of data is [Magda](https://magda.io/) which has API metadata
definitions. Magda is more about data federation and as such provides definitions. Magda is more about data federation and as such provides

View File

@ -13,7 +13,7 @@
# import os # import os
# import sys # import sys
# sys.path.insert(0, os.path.abspath('.')) # sys.path.insert(0, os.path.abspath('.'))
from recommonmark.parser import CommonMarkParser
# -- Project information ----------------------------------------------------- # -- Project information -----------------------------------------------------
@ -43,12 +43,13 @@ exclude_patterns = [
'samples/powershell/test/*', 'samples/powershell/test/*',
] ]
source_suffix = { source_parsers = {
'.rst': 'restructuredtext', '.md': CommonMarkParser
'.txt': 'restructuredtext',
'.md': 'markdown',
} }
source_suffix = ['.rst', '.md']
# -- Options for HTML output ------------------------------------------------- # -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for # The theme to use for HTML and HTML Help pages. See the documentation for

View File

@ -1,4 +1,5 @@
# Custom Definition Custom Definition
=================
The custome definition section allows the inclusion of extensions The custome definition section allows the inclusion of extensions
to the standard. To demonstrate the inclusion, there are three to the standard. To demonstrate the inclusion, there are three

View File

@ -1,4 +1,5 @@
# Magda definitions Magda definitions
=================
https://magda.io/ https://magda.io/

View File

@ -1,4 +1,5 @@
# MartiLQ document MartiLQ document
================
The metadata reconciliation transfer information is referred The metadata reconciliation transfer information is referred
to as the **martiLQ** document throughout this documentation. to as the **martiLQ** document throughout this documentation.
@ -6,9 +7,8 @@ to as the **martiLQ** document throughout this documentation.
The **martiLQ** document can be part of a message or a file The **martiLQ** document can be part of a message or a file
in its own right. The definition is curently a Json file. in its own right. The definition is curently a Json file.
Mandatory information
---------------------
### Mandatory information
The mandatory information is: The mandatory information is:
@ -16,7 +16,8 @@ The mandatory information is:
* Unique identifier * Unique identifier
* Resource list - See Resource section summary below or detailed document [Resource](resources.md) * Resource list - See Resource section summary below or detailed document [Resource](resources.md)
### Optional information Optional information
++++++++++++++++++++
The optional information is: The optional information is:
@ -36,12 +37,14 @@ The optional information is:
* Custom list - List of custom entries, one being the **martiLQ** software details * Custom list - List of custom entries, one being the **martiLQ** software details
see [custom](custom.md) see [custom](custom.md)
### Information extension Information extension
+++++++++++++++++++++
The information supplied can be extended by party agreement and there The information supplied can be extended by party agreement and there
are place holders in the defintion. are place holders in the defintion.
### Resource Resource
--------
The resource section is a list of documents or files that are to be grouped The resource section is a list of documents or files that are to be grouped
together are listed under the same **martiLQ** definition. together are listed under the same **martiLQ** definition.
@ -59,7 +62,8 @@ items:
* Size of document - The document size in bytes * Size of document - The document size in bytes
* URL - This can be ``file://``, ``https://``, ``ftp://``, etc resource location * URL - This can be ``file://``, ``https://``, ``ftp://``, etc resource location
### Resource optional Resource optional
+++++++++++++++++
The following are some of the optional items in the resource section. See [Resource](resources.md) The following are some of the optional items in the resource section. See [Resource](resources.md)
for more details for more details

View File

@ -1,4 +1,5 @@
# MartiLQ objective MartiLQ objective
=================
The objective of **martLQ** is to define a simle standard for The objective of **martLQ** is to define a simle standard for
capturing the data files being transferred. It is not for capturing the data files being transferred. It is not for

View File

@ -1,11 +1,13 @@
# Quality definition Quality definition
==================
The **martiLQ** definition allows for the inclusion of load quality The **martiLQ** definition allows for the inclusion of load quality
definitions. The load quality definition is intended to be definitions. The load quality definition is intended to be
able to be applied universally with common tools. Not able to be applied universally with common tools. Not
all needs are covered with the base definition but can be extended. all needs are covered with the base definition but can be extended.
## Defined load quality metrics Defined load quality metrics
----------------------------
* Sequential batch number - This is a decimal number defined at the **martiLQ** definition * Sequential batch number - This is a decimal number defined at the **martiLQ** definition
and applies to all resources. The integer portion is for new batches and the fraction and applies to all resources. The integer portion is for new batches and the fraction
@ -17,7 +19,8 @@ all needs are covered with the base definition but can be extended.
on the number of primary segments under root. JSON records can be counted in a similar way. on the number of primary segments under root. JSON records can be counted in a similar way.
The headers or trailling records are not counted The headers or trailling records are not counted
## Addresses deficiencies Addresses deficiencies
----------------------
The **martiLQ** objective is to address deficiencies with alternative The **martiLQ** objective is to address deficiencies with alternative
data load quality approaches such as: data load quality approaches such as:
@ -29,7 +32,8 @@ data load quality approaches such as:
* securing the data * securing the data
* adding footers to the data, requiring custom file handlers * adding footers to the data, requiring custom file handlers
## Extending load quality metrics Extending load quality metrics
------------------------------
**martLQ** framewokr and importantly is open to extension so that extra **martLQ** framewokr and importantly is open to extension so that extra
load metrics appropriate to the situation can be included. load metrics appropriate to the situation can be included.

View File

@ -1,4 +1,5 @@
# References References
==========
https://dex.dss.gov.au/sites/default/files/documents/2021-06/data-exchange-protocols-june-2021.pdf https://dex.dss.gov.au/sites/default/files/documents/2021-06/data-exchange-protocols-june-2021.pdf

View File

@ -1,4 +1,5 @@
# Resources definition Resources definition
====================
The resources section defines the files that are grouped The resources section defines the files that are grouped
together by association. This association is not defined but can together by association. This association is not defined but can
@ -38,7 +39,8 @@ The following are optional in the resource section.
* Encryption - Type of encryption used if any * Encryption - Type of encryption used if any
## Compression Compression
-----------
Files can be compressed using a utility. A single compressed file can contain Files can be compressed using a utility. A single compressed file can contain
multiple files. The **martiLQ** definition document applies to the compressed file multiple files. The **martiLQ** definition document applies to the compressed file
@ -49,7 +51,8 @@ compressed file.
Compression of files always occur before encryption. Compression of files always occur before encryption.
### martiLQ definition for Compressed File martiLQ definition for Compressed File
++++++++++++++++++++++++++++++++++++++
For a compressed file that is not encrypted, the distribution definition will be: For a compressed file that is not encrypted, the distribution definition will be:
@ -68,7 +71,8 @@ The reason for this approach is it allows a generic tool to be deployed to
check the validity of the contents without unpacking the received /fetched check the validity of the contents without unpacking the received /fetched
file. That is you can perform load quality pipeline processing. file. That is you can perform load quality pipeline processing.
## Encryption Encryption
----------
The encryption of content is always applied after compression not before, if The encryption of content is always applied after compression not before, if
you are not using the compression tool native encryption. WinZIP and 7ZIP you are not using the compression tool native encryption. WinZIP and 7ZIP

View File

@ -1,4 +1,5 @@
# What is marti What is marti
=============
The foundation pillar for the **martiLQ** framework is the [martiLQ document](martiLQ.md) The foundation pillar for the **martiLQ** framework is the [martiLQ document](martiLQ.md)
that defines the reconciliation and other metadata of the document / file being transferred. that defines the reconciliation and other metadata of the document / file being transferred.

View File

@ -1,4 +1,5 @@
# When would you use martiLQ When would you use martiLQ
==========================
You are likely to start using the **martiLQ** framework when: You are likely to start using the **martiLQ** framework when:
@ -11,7 +12,8 @@ If you already have a standard and it works for you, and you have no upcoming (l
initiative that would benefit from the framework, then stick with what you have. The benefits initiative that would benefit from the framework, then stick with what you have. The benefits
of the framework are unlikely to weigh in your framework. of the framework are unlikely to weigh in your framework.
## Read the material Read the material
-----------------
Please read the material before jumping in and make sure the **martiLQ** framework will Please read the material before jumping in and make sure the **martiLQ** framework will
provide you with the expected benefits. Implementing the framework and then changing provide you with the expected benefits. Implementing the framework and then changing

View File

@ -1,11 +1,13 @@
# Who is likely to use martiLQ Who is likely to use martiLQ
============================
You are likely to find the **martiLQ** framework relevant if you: You are likely to find the **martiLQ** framework relevant if you:
1. Have many document exchanges, such as End of Day batches 1. Have many document exchanges, such as End of Day batches
2. Need to verify or reconcile the documents 2. Need to verify or reconcile the documents
## Data exchanges Data exchanges
--------------
If you are creating or receiving many documents or files on a regular basis If you are creating or receiving many documents or files on a regular basis
then you probably have some framework defined. The framework may be as simple as: then you probably have some framework defined. The framework may be as simple as:
@ -22,7 +24,8 @@ Simple framework such as the above have limitations, such as:
* Lower automation prospects and alignment to DataSecOps * Lower automation prospects and alignment to DataSecOps
* Poor fit to web applications (they tend to be designed for FTP and LAN) * Poor fit to web applications (they tend to be designed for FTP and LAN)
## Framework Sidecar files Framework Sidecar files
-----------------------
The **martiLQ** framework addresses the issues and limitations by using sidecar The **martiLQ** framework addresses the issues and limitations by using sidecar
or shadow files. The [concept of sidecar files](https://en.wikipedia.org/wiki/Sidecar_file) is or shadow files. The [concept of sidecar files](https://en.wikipedia.org/wiki/Sidecar_file) is

View File

@ -1,4 +1,5 @@
# Why use marti Why use marti
=============
**martiLQ** is a framework for providing a degree of auditability and reconciliation of **martiLQ** is a framework for providing a degree of auditability and reconciliation of
documents transferred between systems in an organisation and externally. It does not intend documents transferred between systems in an organisation and externally. It does not intend
@ -13,7 +14,8 @@ define the format or content of the document. It defines controls that can be u
You would use **martiLQ** if any of the controls are a requirement for you. You would use **martiLQ** if any of the controls are a requirement for you.
## Documents Documents
---------
Documents in this context are digital storage objects such as operating system files, Documents in this context are digital storage objects such as operating system files,
cloud storage objects or blobs. The document content can have structure and contains multiple cloud storage objects or blobs. The document content can have structure and contains multiple
@ -25,7 +27,8 @@ in single web transactions. It is for providing controls when moving large amou
data as one event. This data are commonly referred to as batch extracts and performed data as one event. This data are commonly referred to as batch extracts and performed
at scheduled times such as end of day. at scheduled times such as end of day.
## Security Security
--------
The framework does not replace your security, inflight encryption or encryption at rest. The framework does not replace your security, inflight encryption or encryption at rest.