Make MD compatible with RtD
parent
28fe1b006c
commit
dc44c0d660
|
|
@ -1,11 +1,13 @@
|
|||
# Attribute definition
|
||||
Attribute definition
|
||||
====================
|
||||
|
||||
A Resource can list attributes related to the document / file.
|
||||
|
||||
An attribute is a generic definition and conventions are
|
||||
observed in the definitions that are captured here.
|
||||
|
||||
## Attribute definition
|
||||
Attribute definition
|
||||
--------------------
|
||||
|
||||
The Attribute consists of:
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
# CKAN definition
|
||||
CKAN definition
|
||||
===============
|
||||
|
||||
The **martiLQ** has used similar terms and structures that are found in the
|
||||
CKAN API describing the resources. This similarity allows for simple mapping of values
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
# Comparison of martiLQ document definition
|
||||
Comparison of martiLQ document definition
|
||||
=========================================
|
||||
|
||||
The use of metadata definitions is not unique and examples
|
||||
exist in many different situations. Some are standard and open
|
||||
|
|
@ -22,7 +23,8 @@ a CKAN source into a **martiLQ** document definition and then process the data
|
|||
through the pipeline as you would for any other data file that had a
|
||||
**martiLQ** document definition.
|
||||
|
||||
## Benefit of CKAN and martiLQ
|
||||
Benefit of CKAN and martiLQ
|
||||
---------------------------
|
||||
|
||||
The CKAN is excellent at defining the data source details but it lacks information
|
||||
for load quality. If you have CKAN deployed in your organisation and wish
|
||||
|
|
@ -31,7 +33,8 @@ CKAN and marti.
|
|||
|
||||
Samples exist on CKAN integration.
|
||||
|
||||
## Magda and martiLQ
|
||||
Magda and martiLQ
|
||||
-----------------
|
||||
|
||||
Another source of data is [Magda](https://magda.io/) which has API metadata
|
||||
definitions. Magda is more about data federation and as such provides
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@
|
|||
# import os
|
||||
# import sys
|
||||
# sys.path.insert(0, os.path.abspath('.'))
|
||||
|
||||
from recommonmark.parser import CommonMarkParser
|
||||
|
||||
# -- Project information -----------------------------------------------------
|
||||
|
||||
|
|
@ -43,12 +43,13 @@ exclude_patterns = [
|
|||
'samples/powershell/test/*',
|
||||
]
|
||||
|
||||
source_suffix = {
|
||||
'.rst': 'restructuredtext',
|
||||
'.txt': 'restructuredtext',
|
||||
'.md': 'markdown',
|
||||
source_parsers = {
|
||||
'.md': CommonMarkParser
|
||||
}
|
||||
|
||||
source_suffix = ['.rst', '.md']
|
||||
|
||||
|
||||
# -- Options for HTML output -------------------------------------------------
|
||||
|
||||
# The theme to use for HTML and HTML Help pages. See the documentation for
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
# Custom Definition
|
||||
Custom Definition
|
||||
=================
|
||||
|
||||
The custome definition section allows the inclusion of extensions
|
||||
to the standard. To demonstrate the inclusion, there are three
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
# Magda definitions
|
||||
Magda definitions
|
||||
=================
|
||||
|
||||
https://magda.io/
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
# MartiLQ document
|
||||
MartiLQ document
|
||||
================
|
||||
|
||||
The metadata reconciliation transfer information is referred
|
||||
to as the **martiLQ** document throughout this documentation.
|
||||
|
|
@ -6,9 +7,8 @@ to as the **martiLQ** document throughout this documentation.
|
|||
The **martiLQ** document can be part of a message or a file
|
||||
in its own right. The definition is curently a Json file.
|
||||
|
||||
|
||||
|
||||
### Mandatory information
|
||||
Mandatory information
|
||||
---------------------
|
||||
|
||||
The mandatory information is:
|
||||
|
||||
|
|
@ -16,7 +16,8 @@ The mandatory information is:
|
|||
* Unique identifier
|
||||
* Resource list - See Resource section summary below or detailed document [Resource](resources.md)
|
||||
|
||||
### Optional information
|
||||
Optional information
|
||||
++++++++++++++++++++
|
||||
|
||||
The optional information is:
|
||||
|
||||
|
|
@ -36,12 +37,14 @@ The optional information is:
|
|||
* Custom list - List of custom entries, one being the **martiLQ** software details
|
||||
see [custom](custom.md)
|
||||
|
||||
### Information extension
|
||||
Information extension
|
||||
+++++++++++++++++++++
|
||||
|
||||
The information supplied can be extended by party agreement and there
|
||||
are place holders in the defintion.
|
||||
|
||||
### Resource
|
||||
Resource
|
||||
--------
|
||||
|
||||
The resource section is a list of documents or files that are to be grouped
|
||||
together are listed under the same **martiLQ** definition.
|
||||
|
|
@ -59,7 +62,8 @@ items:
|
|||
* Size of document - The document size in bytes
|
||||
* URL - This can be ``file://``, ``https://``, ``ftp://``, etc resource location
|
||||
|
||||
### Resource optional
|
||||
Resource optional
|
||||
+++++++++++++++++
|
||||
|
||||
The following are some of the optional items in the resource section. See [Resource](resources.md)
|
||||
for more details
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
# MartiLQ objective
|
||||
MartiLQ objective
|
||||
=================
|
||||
|
||||
The objective of **martLQ** is to define a simle standard for
|
||||
capturing the data files being transferred. It is not for
|
||||
|
|
|
|||
|
|
@ -1,11 +1,13 @@
|
|||
# Quality definition
|
||||
Quality definition
|
||||
==================
|
||||
|
||||
The **martiLQ** definition allows for the inclusion of load quality
|
||||
definitions. The load quality definition is intended to be
|
||||
able to be applied universally with common tools. Not
|
||||
all needs are covered with the base definition but can be extended.
|
||||
|
||||
## Defined load quality metrics
|
||||
Defined load quality metrics
|
||||
----------------------------
|
||||
|
||||
* Sequential batch number - This is a decimal number defined at the **martiLQ** definition
|
||||
and applies to all resources. The integer portion is for new batches and the fraction
|
||||
|
|
@ -17,7 +19,8 @@ all needs are covered with the base definition but can be extended.
|
|||
on the number of primary segments under root. JSON records can be counted in a similar way.
|
||||
The headers or trailling records are not counted
|
||||
|
||||
## Addresses deficiencies
|
||||
Addresses deficiencies
|
||||
----------------------
|
||||
|
||||
The **martiLQ** objective is to address deficiencies with alternative
|
||||
data load quality approaches such as:
|
||||
|
|
@ -29,7 +32,8 @@ data load quality approaches such as:
|
|||
* securing the data
|
||||
* adding footers to the data, requiring custom file handlers
|
||||
|
||||
## Extending load quality metrics
|
||||
Extending load quality metrics
|
||||
------------------------------
|
||||
|
||||
**martLQ** framewokr and importantly is open to extension so that extra
|
||||
load metrics appropriate to the situation can be included.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
# References
|
||||
References
|
||||
==========
|
||||
|
||||
|
||||
https://dex.dss.gov.au/sites/default/files/documents/2021-06/data-exchange-protocols-june-2021.pdf
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
# Resources definition
|
||||
Resources definition
|
||||
====================
|
||||
|
||||
The resources section defines the files that are grouped
|
||||
together by association. This association is not defined but can
|
||||
|
|
@ -38,7 +39,8 @@ The following are optional in the resource section.
|
|||
* Encryption - Type of encryption used if any
|
||||
|
||||
|
||||
## Compression
|
||||
Compression
|
||||
-----------
|
||||
|
||||
Files can be compressed using a utility. A single compressed file can contain
|
||||
multiple files. The **martiLQ** definition document applies to the compressed file
|
||||
|
|
@ -49,7 +51,8 @@ compressed file.
|
|||
|
||||
Compression of files always occur before encryption.
|
||||
|
||||
### martiLQ definition for Compressed File
|
||||
martiLQ definition for Compressed File
|
||||
++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
For a compressed file that is not encrypted, the distribution definition will be:
|
||||
|
||||
|
|
@ -68,7 +71,8 @@ The reason for this approach is it allows a generic tool to be deployed to
|
|||
check the validity of the contents without unpacking the received /fetched
|
||||
file. That is you can perform load quality pipeline processing.
|
||||
|
||||
## Encryption
|
||||
Encryption
|
||||
----------
|
||||
|
||||
The encryption of content is always applied after compression not before, if
|
||||
you are not using the compression tool native encryption. WinZIP and 7ZIP
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
# What is marti
|
||||
What is marti
|
||||
=============
|
||||
|
||||
The foundation pillar for the **martiLQ** framework is the [martiLQ document](martiLQ.md)
|
||||
that defines the reconciliation and other metadata of the document / file being transferred.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
# When would you use martiLQ
|
||||
When would you use martiLQ
|
||||
==========================
|
||||
|
||||
You are likely to start using the **martiLQ** framework when:
|
||||
|
||||
|
|
@ -11,7 +12,8 @@ If you already have a standard and it works for you, and you have no upcoming (l
|
|||
initiative that would benefit from the framework, then stick with what you have. The benefits
|
||||
of the framework are unlikely to weigh in your framework.
|
||||
|
||||
## Read the material
|
||||
Read the material
|
||||
-----------------
|
||||
|
||||
Please read the material before jumping in and make sure the **martiLQ** framework will
|
||||
provide you with the expected benefits. Implementing the framework and then changing
|
||||
|
|
|
|||
|
|
@ -1,11 +1,13 @@
|
|||
# Who is likely to use martiLQ
|
||||
Who is likely to use martiLQ
|
||||
============================
|
||||
|
||||
You are likely to find the **martiLQ** framework relevant if you:
|
||||
|
||||
1. Have many document exchanges, such as End of Day batches
|
||||
2. Need to verify or reconcile the documents
|
||||
|
||||
## Data exchanges
|
||||
Data exchanges
|
||||
--------------
|
||||
|
||||
If you are creating or receiving many documents or files on a regular basis
|
||||
then you probably have some framework defined. The framework may be as simple as:
|
||||
|
|
@ -22,7 +24,8 @@ Simple framework such as the above have limitations, such as:
|
|||
* Lower automation prospects and alignment to DataSecOps
|
||||
* Poor fit to web applications (they tend to be designed for FTP and LAN)
|
||||
|
||||
## Framework Sidecar files
|
||||
Framework Sidecar files
|
||||
-----------------------
|
||||
|
||||
The **martiLQ** framework addresses the issues and limitations by using sidecar
|
||||
or shadow files. The [concept of sidecar files](https://en.wikipedia.org/wiki/Sidecar_file) is
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
# Why use marti
|
||||
Why use marti
|
||||
=============
|
||||
|
||||
**martiLQ** is a framework for providing a degree of auditability and reconciliation of
|
||||
documents transferred between systems in an organisation and externally. It does not intend
|
||||
|
|
@ -13,7 +14,8 @@ define the format or content of the document. It defines controls that can be u
|
|||
|
||||
You would use **martiLQ** if any of the controls are a requirement for you.
|
||||
|
||||
## Documents
|
||||
Documents
|
||||
---------
|
||||
|
||||
Documents in this context are digital storage objects such as operating system files,
|
||||
cloud storage objects or blobs. The document content can have structure and contains multiple
|
||||
|
|
@ -25,7 +27,8 @@ in single web transactions. It is for providing controls when moving large amou
|
|||
data as one event. This data are commonly referred to as batch extracts and performed
|
||||
at scheduled times such as end of day.
|
||||
|
||||
## Security
|
||||
Security
|
||||
--------
|
||||
|
||||
The framework does not replace your security, inflight encryption or encryption at rest.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue