Wardrobe
/
martiLQ
mirror of https://github.com/meerkat-manor/martiLQ.git
3
0
Fork 0
Code Issues Projects Releases Wiki Activity

Refactored to use mkdocs

draft_specifications
meerkat 2021-11-03 00:13:27 +11:00
parent dc44c0d660
commit d8dee9f825
23 changed files with 983 additions and 248 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
.gitignore vendored
View File

@ -21,4 +21,6 @@ source/python/client/__pycache__
source/golang/client/src/test source/golang/client/src/test
source/golang/client/src/martilq_client* source/golang/client/src/martilq_client*
source/golang/client/src/*.exe source/golang/client/src/*.exe
docs/site/

28
docs/mkdocs.yml 100644
View File

@ -0,0 +1,28 @@
site_name: martiLQ
site_url: https://martilq.readthedocs.com/
docs_dir: source
nav:
- Home: README.md
- martiLQ: martiLQ.md
- Why: why.md
- What: what.md
- When: when.md
- Who: who.md
- Resource: resources.md
- Attributes: attributes.md
- Custom: custom.md
- Quality: quality.md
- Samples: samples\json\README.md
- Comparison: comparison.md
- CKAN: ckan.md
- MAGDA: magda.md
- References: references.md
theme: readthedocs
extra_css:
- stylesheets/extra.css
site_author: meerkat
repo_url: https://github.com/meerkat-manor/marti/

43
docs/source/README.md
View File

@ -9,20 +9,35 @@ documents.
There is no quickstart document to get you started as each use case and There is no quickstart document to get you started as each use case and
organisation is different. There are sample implementations which you organisation is different. There are sample implementations which you
can adjust if they resonate with your circumstances, can adjust if they resonate with your circumstances,
see [sample implementations](samples/) see [sample implementations](samples/)
## Index
1. [Objective](objective.md) There are sample implementations which you
2. [martiLQ](martiLQ.md) can adjust if they resonate with your circumstances.
3. [Why you would use martiLQ](why.md)
4. [What is martiLQ](what.md)
5. [When would you start using martiLQ](when.md)
6. [Who would use martiLQ](who.md)
7. [Resources definition](resources.md)
8. [Attributes definition](attributes.md)
9. [Custom definition](custom.md)
10. [Load quality metrics support](quality.md)
11. [Comparison of martiLQ definition](comparison.md)
12. [References](references.md)
The source, documentation and samples are available at `<https://github.com/meerkat-manor/marti>`_
## MartiLQ objective
The objective of **martiLQ** is to define a simple standard for
capturing the data files being transferred. It is not for
real time web service transactions.
**martiLQ** is about file and document transfer and reconciling
that the all files have arrived and have not changed, and if so
required are also encrypted.
The proposition is to have a common, machine readable format
for file exchange that:
* ensures data load quality and reconciles
* can be used on Linux or Windows or Mac
* can be used with Python, Java, PowerShell, Golang, etc
* can be used by web services
* uses a text based format (JSON)
* can form part of the data processing pipeline
And finally is easy to understand.
To get a better understanding have a look at the definition
in [martiLQ](martiLQ.md)

26
docs/source/attributes.md
View File

@ -1,22 +1,24 @@
Attribute definition # Attribute definition
====================
A Resource can list attributes related to the document / file. A Resource can list attributes related to the document / file.
An attribute is a generic definition and conventions are An attribute is a generic definition and conventions are
observed in the definitions that are captured here. observed in the definitions that are captured here. The attribute
section is where load quality metrics are defined.
Attribute definition ## Attribute definition
--------------------
The Attribute consists of: The Attribute is described by the table below. Recommended
values are listed but custom values can also be defined, just be
certain the recipient is able to understand them.
* category - A value of "dataset", Name|Description|Values or Default
* name ---|---|---
* function - A value such as "count" category|A type of attribute|dataset, format
* comparison - A comparisn value or NA. Values are "EQ", name|A name for the attribute|records,columns,header,footer,separator,quote, escape
"NE", "GT", "LT function|A function to perform|count,sum,unique
* value - The value for the attribute based on the above complex key, excluding comparison comparison|A comparison value or NA|NA, EQ, NE, GT, GE, LT,LE
value|The value for the attribute based on the above complex key, excluding comparison|numeric
A sample JSON is shown below which describes the A sample JSON is shown below which describes the
number of records in the file for the given format. number of records in the file for the given format.

9
docs/source/ckan.md
View File

@ -1,16 +1,15 @@
CKAN definition # CKAN definition
===============
The **martiLQ** has used similar terms and structures that are found in the The **martiLQ** has used similar terms and structures that are found in the
CKAN API describing the resources. This similarity allows for simple mapping of values CKAN API describing the resources. This similarity allows for simple mapping of values
from the CKAN format to **martiLQ** format. from the CKAN format to **martiLQ** format.
What **martiLQ** brings above the CKAN definition is attributes that can be What **martiLQ** extends above the CKAN definition is attributes that can be
used to reconcile with the recieved data, plus the ability to define used to reconcile with the received data, plus the ability to define
compressed and encrypted resources. compressed and encrypted resources.
For more information on CKAN see https://ckan.org/ For more information on CKAN see https://ckan.org/
A sample Json to compare against the **martiLQ** definition A sample JSON to compare against the **martiLQ** document
is https://data.gov.au/data/dataset/f2b7c2c1-f4ef-4ae9-aba5-45c19e4d3038 is https://data.gov.au/data/dataset/f2b7c2c1-f4ef-4ae9-aba5-45c19e4d3038

65
docs/source/conf.py
View File

@ -1,65 +0,0 @@
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
from recommonmark.parser import CommonMarkParser
# -- Project information -----------------------------------------------------
project = 'MartiLQ'
copyright = '2021, meerkat@merebox.coom'
author = 'meerkat@merebox.coom'
# The full version, including alpha/beta/rc tags
release = '0.0.1'
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
#extensions = ['myst_parser']
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = [
'samples/python/test/*',
'samples/powershell/test/*',
]
source_parsers = {
'.md': CommonMarkParser
}
source_suffix = ['.rst', '.md']
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
#html_theme = 'alabaster'
html_theme = 'sphinx_rtd_theme'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']

37
docs/source/index.rst
View File

@ -1,37 +0,0 @@
.. MartiLQ documentation master file, created by
sphinx-quickstart on Sun Oct 31 14:30:06 2021.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to MartiLQ's documentation!
===================================
.. toctree::
:hidden:
:maxdepth: 2
objective.md
martiLQ.md
why.md
what.md
when.md
who.md
resources.md
attributes.md
custom.md
quality.md
comparison.md
ckan.md
magda.md
references.md
**martiLQ** stands for metadata reconcilation for transfer information, load quality.
Before starting with **martiLQ** it is advisable to understand if it is right for
your organisation's needs. Information is available in a number of short
documents. Start with the (objectives)[objective.md]
There are sample implementations which you
can adjust if they resonate with your circumstances.
The source, documentation and samples are available at `<https://github.com/meerkat-manor/marti>`_

175
docs/source/martiLQ.md
View File

@ -1,33 +1,47 @@
MartiLQ document # MartiLQ document
================
The metadata reconciliation transfer information is referred The metadata reconciliation transfer information is referred
to as the **martiLQ** document throughout this documentation. to as the **martiLQ** document throughout this documentation.
The **martiLQ** document can be part of a message or a file The **martiLQ** document can be part of a message or a file
in its own right. The definition is curently a Json file. in its own right. The definition is currently a JSON file.
Mandatory information ## Structure
---------------------
The mandatory information is: The JSON is composed of:
* Title * A root definition that contains information applicable to all reosurces
* Unique identifier * A [resource](resources.md) list that contains information related
* Resource list - See Resource section summary below or detailed document [Resource](resources.md) to each document or file
* [Attribute](attributes.md) list as a child to each resource
* A [custom](custom.md) list
Optional information ## Mandatory root information
++++++++++++++++++++
The optional information is: The mandatory information in the root of the **martiLQ** document is:
Name|Description|Default or values
---|---|---
title|A title or batch identifier. Use a name that is easy to understand or relate to|None
uid|Unique identifier for the document. If the same document is reproduced this may not change but the minor number of the batch must change|Auto generated
resources|Resource list. See Resource section summary below or detailed document [Resource](resources.md)|At least one required
### Optional information
The optional information is described in the table below. A number of elements can be configured for their
default value(s) in a configuration file.
Name|Description|Default or values
---|---|--
description|Long description of the purpose, background or files included|None
modified|Modified date and time of the **martiLQ** document|Now
tags|List of tags or keywords|
publisher|Publisher name|
contactPoint|Contact point of a person or team|
accessLevel|Acces level|
rights|Rights|
* Description
* Modified
* Tags or keywords
* Publisher
* Contact point
* Acces level
* Rights
* Batch * Batch
* License * License
* Described By - A link to the metadata describing the document. * Described By - A link to the metadata describing the document.
@ -37,14 +51,12 @@ The optional information is:
* Custom list - List of custom entries, one being the **martiLQ** software details * Custom list - List of custom entries, one being the **martiLQ** software details
see [custom](custom.md) see [custom](custom.md)
Information extension ### Information extension
+++++++++++++++++++++
The information supplied can be extended by party agreement and there The information supplied can be extended by party agreement and there
are place holders in the defintion. are place holders in the defintion.
Resource ## Resource
--------
The resource section is a list of documents or files that are to be grouped The resource section is a list of documents or files that are to be grouped
together are listed under the same **martiLQ** definition. together are listed under the same **martiLQ** definition.
@ -54,26 +66,107 @@ it will commonly be for definiting multiple formats, with each file having a
different extension. Commonly the definition includes at least the following different extension. Commonly the definition includes at least the following
items: items:
* Title - A summary description of the document contents Name|Description|Default or values
* Unique identifier - A unique identifier, commonly a GUID ---|---|--
* Document name - A name of the document such as thefile name title|Title for the resource|Document name
* Issued date - When the document was made available. The date can include time uid|A unique identifier, commonly a GUID|Auto generated
* Modified - When the document was created or modified. This is the data and time documentName|A name of the document such as the file name
* Size of document - The document size in bytes issueDate|Issued date - When the document was made available. The date can include time
* URL - This can be ``file://``, ``https://``, ``ftp://``, etc resource location modified|Modified - When the document was created or modified. This is the data and time
size|Size of document - The document size in bytes
url|URL - This can be ``file://``, ``https://``, ``ftp://``, etc resource location
Resource optional ### Resource optional
+++++++++++++++++
The following are some of the optional items in the resource section. See [Resource](resources.md) The following are some of the optional items in the resource section. See [Resource](resources.md)
for more details for more details
* Hash of document - The hash of the document, which can be blank especially for large documents Name|Description|Default or values
* Hash algorithm - Algoroithm used to generate the hash value or sign it ---|---|--
* Description - A more detailed description hash|Hash of document - The hash of the document, which can be blank especially for large documents
* Version - A document version algo|Hash algorithm - Algoroithm used to generate the hash value or sign it
* Encoding description|Description - A more detailed description
* Content Type version|Version - A document version
* Compression encoding|Encoding
* Encryption contentType|Content Type
* Author compression|Compression|None
encryption|Encryption|None
author|Author
attributes|List of attributes for the resource|Record count
## Simple sample
A sample of a single resource **martiLQ** document is shown below. The
sample can be generated using the GOLANG client program with parameters:
```
-t GEN -m Sample.json -s ./docs/source/martilq.md --title "GEN001" --description "Simple example"
```
```json
{
"content-type": "application/vnd.martilq.json",
"title": "GEN001",
"uid": "9a0a7edb-dd81-4fc5-a6cb-c5716eda7b51",
"description": "Simple example",
"modified": "2021-11-02T22:44:29.6887001+11:00",
"publisher": "",
"contactPoint": "",
"accessLevel": "Confidential",
"rights": "Restricted",
"tags": null,
"license": "",
"state": "active",
"batch": 1.001,
"describedBy": "",
"landingPage": "",
"theme": "",
"resources": [
{
"title": "martilq.md",
"uid": "a88b4e5f-66b7-4003-ac24-831c95d0da07",
"documentName": "martilq.md",
"issueDate": "2021-11-02T22:44:29.6881663+11:00",
"modified": "2021-11-02T07:47:13.9410018+11:00",
"expires": "2023-11-02T00:00:00+11:00",
"state": "active",
"author": "",
"length": 3654,
"hash": {
"algo": "SHA256",
"value": "213a6254ddc02423b6c3bb3d977892678258539d37f06410ef18d27c14ffa821",
"signed": false
},
"description": "",
"url": "http://localhost/martilq/martilq.md",
"version": "",
"content-type": "",
"encoding": "UTF-8",
"compression": "",
"encryption": "",
"describedBy": "",
"attributes": [
{
"category": "dataset",
"name": "records",
"function": "count",
"comparison": "EQ",
"value": "95"
}
]
}
],
"custom": [
{
"extension": "software",
"softwareName": "MARTILQREFERENCE",
"author": "Meerkat@merebox.com",
"version": "0.0.1"
}
]
}
```
You can view a more complete sample [samples/json/sample_02.md](samples/json/sample_02.md)
which has been generated using a configuration file to supply default values.

23
docs/source/objective.md
View File

@ -1,23 +0,0 @@
MartiLQ objective
=================
The objective of **martLQ** is to define a simle standard for
capturing the data files being transferred. It is not for
real time web service transactions.
**martiLQ** is about file and document transfer and reconciling
that the all files have arrived and have not changed, and if so
required are also encrypted.
The proposition is to have a common, machine readable format
for file exchange that:
* ensures data load quality and reconciles
* can be used on Linux or Windows or Mac
* can be used with Python, Java, PowerShell, Golang, etc
* can be used by web services
* uses a text based format (JSON)
* can form part of the data processing pipeline
And finally is easy to understand.

28
docs/source/quality.md
View File

@ -1,16 +1,18 @@
Quality definition # Load Quality
==================
The **martiLQ** definition allows for the inclusion of load quality The **martiLQ** document allows for the inclusion of load quality
definitions. The load quality definition is intended to be metrics. The load quality metrics is intended to be
able to be applied universally with common tools. Not able to be applied universally with common tools. Not
all needs are covered with the base definition but can be extended. all needs are covered with the base definition but can be extended.
Defined load quality metrics The load quality metrics are in the majority defined in the [attributes](attributes.md)
---------------------------- list attached to each resource. Therefore each resource can have different
load quality metrics.
* Sequential batch number - This is a decimal number defined at the **martiLQ** definition ## Defined load quality metrics
and applies to all resources. The integer portion is for new batches and the fraction
* Sequential batch number - This is a decimal number defined at the **martiLQ** document
header and applies to all resources. The integer portion is for new batches and the fraction
part can be used for issues with the same data extract. such as requiring resend because part can be used for issues with the same data extract. such as requiring resend because
a resource was missing. a resource was missing.
@ -19,8 +21,7 @@ Defined load quality metrics
on the number of primary segments under root. JSON records can be counted in a similar way. on the number of primary segments under root. JSON records can be counted in a similar way.
The headers or trailling records are not counted The headers or trailling records are not counted
Addresses deficiencies ## Addresses deficiencies
----------------------
The **martiLQ** objective is to address deficiencies with alternative The **martiLQ** objective is to address deficiencies with alternative
data load quality approaches such as: data load quality approaches such as:
@ -32,10 +33,9 @@ data load quality approaches such as:
* securing the data * securing the data
* adding footers to the data, requiring custom file handlers * adding footers to the data, requiring custom file handlers
Extending load quality metrics ## Extending load quality metrics
------------------------------
**martLQ** framewokr and importantly is open to extension so that extra **martLQ** document is open to extension so that extra
load metrics appropriate to the situation can be included. load metrics appropriate to the situation can be included.
### Extension ideas ### Extension ideas
@ -47,5 +47,5 @@ The following extensions for load quality can easily be included:
* Data values are within defined tolerances * Data values are within defined tolerances
* Check for data exclusions * Check for data exclusions
And all this information is included in the **martiLQ** definition And all this information is included in the **martiLQ** document
allowing for self describing load quality. allowing for self describing load quality.

5
docs/source/references.md
View File

@ -1,6 +1,7 @@
References # References
==========
The following are references to documents that inspired the creation of **martiLQ**
document and associatd framework.
https://dex.dss.gov.au/sites/default/files/documents/2021-06/data-exchange-protocols-june-2021.pdf https://dex.dss.gov.au/sites/default/files/documents/2021-06/data-exchange-protocols-june-2021.pdf

12
docs/source/resources.md
View File

@ -1,5 +1,4 @@
Resources definition # Resources definition
====================
The resources section defines the files that are grouped The resources section defines the files that are grouped
together by association. This association is not defined but can together by association. This association is not defined but can
@ -39,8 +38,7 @@ The following are optional in the resource section.
* Encryption - Type of encryption used if any * Encryption - Type of encryption used if any
Compression ## Compression
-----------
Files can be compressed using a utility. A single compressed file can contain Files can be compressed using a utility. A single compressed file can contain
multiple files. The **martiLQ** definition document applies to the compressed file multiple files. The **martiLQ** definition document applies to the compressed file
@ -51,8 +49,7 @@ compressed file.
Compression of files always occur before encryption. Compression of files always occur before encryption.
martiLQ definition for Compressed File ### martiLQ definition for Compressed File
++++++++++++++++++++++++++++++++++++++
For a compressed file that is not encrypted, the distribution definition will be: For a compressed file that is not encrypted, the distribution definition will be:
@ -71,8 +68,7 @@ The reason for this approach is it allows a generic tool to be deployed to
check the validity of the contents without unpacking the received /fetched check the validity of the contents without unpacking the received /fetched
file. That is you can perform load quality pipeline processing. file. That is you can perform load quality pipeline processing.
Encryption ## Encryption
----------
The encryption of content is always applied after compression not before, if The encryption of content is always applied after compression not before, if
you are not using the compression tool native encryption. WinZIP and 7ZIP you are not using the compression tool native encryption. WinZIP and 7ZIP

11
docs/source/samples/json/README.md 100644
View File

@ -0,0 +1,11 @@
# Samples
This folder contains a number of sample **martiLQ** documents
or configuration files.
File|Content
---|---
[sample_01.md](sample_01.md)|Basic output
[sample_02.md](sample_02.md)|Basic ouput with configuration file
[sample_03.md](sample_03.md)|Directory output

55
docs/source/samples/json/martilq_02.ini 100644
View File

@ -0,0 +1,55 @@
[General]
logPath =
tempPath =
dataPath =
[MartiLQ]
tags = sample
publisher = meerkat@merebox.com
contactPoint = Your friendly Meerkat
accessLevel = Confidential
rights = Public
license = MIT
batch = @./config/batch.no
theme = Documentation
[Resources]
author = Hive
title = {{documentName}}
state = expired
expires = 2:0:0
encoding = UTF-8
version = 1.0
urlPrefix = http://localhost/martilq/
[Hash]
hashAlgorithm =
signKey_File =
signKey_Password =
[Network]
proxy =
username =
password =
[Custom_Spatial]
enabled = true
country = Netherland
region =
town = Amsterdam
[Custom_Temporal]
enabled = true
businessDate = {{yesterday}}
runDate = {{today}}

3
docs/source/samples/json/sample_01.md 100644
View File

@ -0,0 +1,3 @@
# Basic sample

122
docs/source/samples/json/sample_02.md 100644
View File

@ -0,0 +1,122 @@
# Sample with configuration file
The sample **martiLQ** document below has been generated
using the client GOLANG program and a configuration file
with changed values from the default
The source is the project folder /docs/source
The configuration file is
```ini
[MartiLQ]
tags = sample
publisher = meerkat@merebox.com
contactPoint = Your friendly Meerkat
accessLevel = Confidential
rights = Public
license = MIT
batch = @./config/batch.no
theme = Documentation
[Resources]
author = Hive
title = {{documentName}}
state = expired
expires = 2:0:0
encoding = UTF-8
urlPrefix = http://localhost/martilq/
[Custom_Spatial]
enabled = true
country = Netherland
region =
town = Amsterdam
[Custom_Temporal]
enabled = true
businessDate = {{yesterday}}
runDate = {{today}}
```
The **martiLQ** document
```json
{
"content-type": "application/vnd.martilq.json",
"title": "GEN001",
"uid": "369fb6c4-3628-4f61-a3ee-7b03a0fc8e25",
"description": "Simple example",
"modified": "2021-11-02T22:56:03.5897714+11:00",
"publisher": "meerkat@merebox.com",
"contactPoint": "Your friendly Meerkat",
"accessLevel": "Confidential",
"rights": "Public",
"tags": [
"sample"
],
"license": "MIT",
"state": "expired",
"batch": 1.001,
"describedBy": "",
"landingPage": "",
"theme": "Documentation",
"resources": [
{
"title": "martilq.md",
"uid": "b0206363-5dcb-485d-83e9-9495e75662c1",
"documentName": "martilq.md",
"issueDate": "2021-11-02T22:56:03.5892511+11:00",
"modified": "2021-11-02T22:47:49.7108132+11:00",
"expires": "2023-11-02T00:00:00+11:00",
"state": "expired",
"author": "Hive",
"length": 5873,
"hash": {
"algo": "SHA256",
"value": "38714907ced5ff5efbf939f6404634dfa51762cec075b82187c0c6fa880aa37b",
"signed": false
},
"description": "",
"url": "http://localhost/martilq/martilq.md",
"version": "",
"content-type": "",
"encoding": "UTF-8",
"compression": "",
"encryption": "",
"describedBy": "",
"attributes": [
{
"category": "dataset",
"name": "records",
"function": "count",
"comparison": "EQ",
"value": "169"
}
]
}
],
"custom": [
{
"extension": "software",
"softwareName": "MARTILQREFERENCE",
"author": "Meerkat@merebox.com",
"version": "0.0.1"
},
{
"extension": "spatial",
"country": "Netherland",
"region": "",
"town": "Amsterdam"
},
{
"extension": "temporal",
"businessDate": "2021-11-01T00:00:00+11:00",
"runDate": "2021-11-02T00:00:00+11:00"
}
]
}
```

515
docs/source/samples/json/sample_03.md 100644
View File

@ -0,0 +1,515 @@
# Sample with entire directory
The sample **martiLQ** document below has been generated
using the client GOLANG program and including all files
in a directory.
The source is the project folder /docs/source and is recursive
to include child folders and documents.
```json
{
"content-type": "application/vnd.martilq.json",
"title": "GEN003",
"uid": "67152b70-368e-4186-ab06-398f4c98a079",
"description": "Directory example",
"modified": "2021-11-02T23:48:07.0787735+11:00",
"publisher": "meerkat@merebox.com",
"contactPoint": "Your friendly Meerkat",
"accessLevel": "Confidential",
"rights": "Public",
"tags": [
"sample"
],
"license": "MIT",
"state": "expired",
"batch": 1.001,
"describedBy": "",
"landingPage": "",
"theme": "Documentation",
"resources": [
{
"title": "README.md",
"uid": "e32b8a62-5571-41a5-beba-071844a80082",
"documentName": "README.md",
"issueDate": "2021-11-02T23:48:07.0480017+11:00",
"modified": "2021-11-01T22:33:07.3888826+11:00",
"expires": "2023-11-02T00:00:00+11:00",
"state": "expired",
"author": "Hive",
"length": 1566,
"hash": {
"algo": "SHA256",
"value": "cf7cbf4186779984144fd022f4ba77b9773ffdc799b4a87f74e0ad2c52a7a261",
"signed": false
},
"description": "",
"url": "http://localhost/martilq/README.md",
"version": "",
"content-type": "",
"encoding": "UTF-8",
"compression": "",
"encryption": "",
"describedBy": "",
"attributes": [
{
"category": "dataset",
"name": "records",
"function": "count",
"comparison": "EQ",
"value": "43"
}
]
},
{
"title": "attributes.md",
"uid": "2d431e9b-9906-44bc-8b0b-6746ca9e5c4c",
"documentName": "attributes.md",
"issueDate": "2021-11-02T23:48:07.0497323+11:00",
"modified": "2021-11-01T22:56:49.7231238+11:00",
"expires": "2023-11-02T00:00:00+11:00",
"state": "expired",
"author": "Hive",
"length": 1204,
"hash": {
"algo": "SHA256",
"value": "38b27aa1302da2abfb1ed6e654f93681571158bb28586919e4cdf771d49d2ed5",
"signed": false
},
"description": "",
"url": "http://localhost/martilq/attributes.md",
"version": "",
"content-type": "",
"encoding": "UTF-8",
"compression": "",
"encryption": "",
"describedBy": "",
"attributes": [
{
"category": "dataset",
"name": "records",
"function": "count",
"comparison": "EQ",
"value": "36"
}
]
},
{
"title": "ckan.md",
"uid": "4141a67a-dcb3-4b0c-b819-66555d0699b4",
"documentName": "ckan.md",
"issueDate": "2021-11-02T23:48:07.0509128+11:00",
"modified": "2021-11-01T22:59:14.8560535+11:00",
"expires": "2023-11-02T00:00:00+11:00",
"state": "expired",
"author": "Hive",
"length": 613,
"hash": {
"algo": "SHA256",
"value": "17e886292b873937f4ff6bb18c32196ffd19311c4adb00ad989d13d01bd738d7",
"signed": false
},
"description": "",
"url": "http://localhost/martilq/ckan.md",
"version": "",
"content-type": "",
"encoding": "UTF-8",
"compression": "",
"encryption": "",
"describedBy": "",
"attributes": [
{
"category": "dataset",
"name": "records",
"function": "count",
"comparison": "EQ",
"value": "15"
}
]
},
{
"title": "comparison.md",
"uid": "b76ca118-b01e-4f26-9395-c5c07051d0e2",
"documentName": "comparison.md",
"issueDate": "2021-11-02T23:48:07.0521334+11:00",
"modified": "2021-11-01T14:27:20.7769396+11:00",
"expires": "2023-11-02T00:00:00+11:00",
"state": "expired",
"author": "Hive",
"length": 1974,
"hash": {
"algo": "SHA256",
"value": "2c11419e22124990c68399fd332bb42508221cd6daeb4bc61b25ab44a12e6d68",
"signed": false
},
"description": "",
"url": "http://localhost/martilq/comparison.md",
"version": "",
"content-type": "",
"encoding": "UTF-8",
"compression": "",
"encryption": "",
"describedBy": "",
"attributes": [
{
"category": "dataset",
"name": "records",
"function": "count",
"comparison": "EQ",
"value": "46"
}
]
},
{
"title": "custom.md",
"uid": "e6829ac7-e0bd-413b-b035-2cf2aee1da5d",
"documentName": "custom.md",
"issueDate": "2021-11-02T23:48:07.0532497+11:00",
"modified": "2021-11-01T13:52:22.4420815+11:00",
"expires": "2023-11-02T00:00:00+11:00",
"state": "expired",
"author": "Hive",
"length": 397,
"hash": {
"algo": "SHA256",
"value": "c55ca715521b98fdad99e020c9a983d694557db84a25a9b5e2b0fbce7ece3549",
"signed": false
},
"description": "",
"url": "http://localhost/martilq/custom.md",
"version": "",
"content-type": "",
"encoding": "UTF-8",
"compression": "",
"encryption": "",
"describedBy": "",
"attributes": [
{
"category": "dataset",
"name": "records",
"function": "count",
"comparison": "EQ",
"value": "10"
}
]
},
{
"title": "magda.md",
"uid": "39879fe6-3525-4152-9b93-acf20aefd683",
"documentName": "magda.md",
"issueDate": "2021-11-02T23:48:07.0554587+11:00",
"modified": "2021-11-01T14:26:31.8900363+11:00",
"expires": "2023-11-02T00:00:00+11:00",
"state": "expired",
"author": "Hive",
"length": 120,
"hash": {
"algo": "SHA256",
"value": "74dcb357989485a7e98ab5c2b718460a0bb81c50b4c3afd8aa72091c4cd62e48",
"signed": false
},
"description": "",
"url": "http://localhost/martilq/magda.md",
"version": "",
"content-type": "",
"encoding": "UTF-8",
"compression": "",
"encryption": "",
"describedBy": "",
"attributes": [
{
"category": "dataset",
"name": "records",
"function": "count",
"comparison": "EQ",
"value": "8"
}
]
},
{
"title": "martiLQ.md",
"uid": "d259ef9f-38a4-4af2-bfdb-baa29e5ea8c9",
"documentName": "martiLQ.md",
"issueDate": "2021-11-02T23:48:07.0569604+11:00",
"modified": "2021-11-02T22:57:00.0061831+11:00",
"expires": "2023-11-02T00:00:00+11:00",
"state": "expired",
"author": "Hive",
"length": 6046,
"hash": {
"algo": "SHA256",
"value": "111f3ad34d94dc346ac282c9a1cc9a3e5802706b5274684b660d2f9b1721abcf",
"signed": false
},
"description": "",
"url": "http://localhost/martilq/martiLQ.md",
"version": "",
"content-type": "",
"encoding": "UTF-8",
"compression": "",
"encryption": "",
"describedBy": "",
"attributes": [
{
"category": "dataset",
"name": "records",
"function": "count",
"comparison": "EQ",
"value": "172"
}
]
},
{
"title": "quality.md",
"uid": "08966d7f-3f6b-4187-9fb8-432fa78cb43d",
"documentName": "quality.md",
"issueDate": "2021-11-02T23:48:07.0581721+11:00",
"modified": "2021-11-01T22:43:35.410177+11:00",
"expires": "2023-11-02T00:00:00+11:00",
"state": "expired",
"author": "Hive",
"length": 2019,
"hash": {
"algo": "SHA256",
"value": "e87c546fae45363d9cd1db0228885290aecc89105c238d203d41253f08d1699f",
"signed": false
},
"description": "",
"url": "http://localhost/martilq/quality.md",
"version": "",
"content-type": "",
"encoding": "UTF-8",
"compression": "",
"encryption": "",
"describedBy": "",
"attributes": [
{
"category": "dataset",
"name": "records",
"function": "count",
"comparison": "EQ",
"value": "51"
}
]
},
{
"title": "references.md",
"uid": "842066e0-2e81-487b-9f48-0df972aa6f12",
"documentName": "references.md",
"issueDate": "2021-11-02T23:48:07.0593388+11:00",
"modified": "2021-11-01T23:25:15.048493+11:00",
"expires": "2023-11-02T00:00:00+11:00",
"state": "expired",
"author": "Hive",
"length": 722,
"hash": {
"algo": "SHA256",
"value": "659db48c3b80a7c1ca3443fc43c45fe76eaa09c151e15f4dc1825fc84653f065",
"signed": false
},
"description": "",
"url": "http://localhost/martilq/references.md",
"version": "",
"content-type": "",
"encoding": "UTF-8",
"compression": "",
"encryption": "",
"describedBy": "",
"attributes": [
{
"category": "dataset",
"name": "records",
"function": "count",
"comparison": "EQ",
"value": "24"
}
]
},
{
"title": "resources.md",
"uid": "21659556-bf8b-486e-868f-c739f3f9cee8",
"documentName": "resources.md",
"issueDate": "2021-11-02T23:48:07.0604619+11:00",
"modified": "2021-11-01T23:23:55.517501+11:00",
"expires": "2023-11-02T00:00:00+11:00",
"state": "expired",
"author": "Hive",
"length": 4646,
"hash": {
"algo": "SHA256",
"value": "d2918d7b373a8b3bf43f377bc3d8df21f623b3c2a61bf5a16e19602fa763b8fc",
"signed": false
},
"description": "",
"url": "http://localhost/martilq/resources.md",
"version": "",
"content-type": "",
"encoding": "UTF-8",
"compression": "",
"encryption": "",
"describedBy": "",
"attributes": [
{
"category": "dataset",
"name": "records",
"function": "count",
"comparison": "EQ",
"value": "98"
}
]
},
{
"title": "what.md",
"uid": "60b0ddad-d860-415e-a3da-4fdfb9d7dc91",
"documentName": "what.md",
"issueDate": "2021-11-02T23:48:07.0638034+11:00",
"modified": "2021-11-01T23:22:55.6854301+11:00",
"expires": "2023-11-02T00:00:00+11:00",
"state": "expired",
"author": "Hive",
"length": 864,
"hash": {
"algo": "SHA256",
"value": "d80c5445b62a2aefd670b0a5fd6f25031f67cc858a1a9826afdbfb82985c6450",
"signed": false
},
"description": "",
"url": "http://localhost/martilq/what.md",
"version": "",
"content-type": "",
"encoding": "UTF-8",
"compression": "",
"encryption": "",
"describedBy": "",
"attributes": [
{
"category": "dataset",
"name": "records",
"function": "count",
"comparison": "EQ",
"value": "18"
}
]
},
{
"title": "when.md",
"uid": "add6d143-e602-4066-b804-3ce7fc1214e4",
"documentName": "when.md",
"issueDate": "2021-11-02T23:48:07.0650335+11:00",
"modified": "2021-11-01T23:22:49.8615445+11:00",
"expires": "2023-11-02T00:00:00+11:00",
"state": "expired",
"author": "Hive",
"length": 1106,
"hash": {
"algo": "SHA256",
"value": "605c97bb103d352c6d8fefe5c86c7e521b5998048a9ae876d6f4c8cd551ec9d1",
"signed": false
},
"description": "",
"url": "http://localhost/martilq/when.md",
"version": "",
"content-type": "",
"encoding": "UTF-8",
"compression": "",
"encryption": "",
"describedBy": "",
"attributes": [
{
"category": "dataset",
"name": "records",
"function": "count",
"comparison": "EQ",
"value": "22"
}
]
},
{
"title": "who.md",
"uid": "f281360b-b9ed-4485-b0d7-20f92a677085",
"documentName": "who.md",
"issueDate": "2021-11-02T23:48:07.0748937+11:00",
"modified": "2021-11-01T14:28:37.7765182+11:00",
"expires": "2023-11-02T00:00:00+11:00",
"state": "expired",
"author": "Hive",
"length": 1943,
"hash": {
"algo": "SHA256",
"value": "c28ece0930b5671896b8ada02b13e85aa5c050d824c3843d557e6ef48579df55",
"signed": false
},
"description": "",
"url": "http://localhost/martilq/who.md",
"version": "",
"content-type": "",
"encoding": "UTF-8",
"compression": "",
"encryption": "",
"describedBy": "",
"attributes": [
{
"category": "dataset",
"name": "records",
"function": "count",
"comparison": "EQ",
"value": "41"
}
]
},
{
"title": "why.md",
"uid": "f4e8b691-3117-4864-a4a9-24737b7fcad0",
"documentName": "why.md",
"issueDate": "2021-11-02T23:48:07.0771793+11:00",
"modified": "2021-11-01T14:29:02.7419541+11:00",
"expires": "2023-11-02T00:00:00+11:00",
"state": "expired",
"author": "Hive",
"length": 1614,
"hash": {
"algo": "SHA256",
"value": "1dbf3221b4a3bdb39e073e9e75dd3218bb45ae6e083bf5b02f63481e58fc8f91",
"signed": false
},
"description": "",
"url": "http://localhost/martilq/why.md",
"version": "",
"content-type": "",
"encoding": "UTF-8",
"compression": "",
"encryption": "",
"describedBy": "",
"attributes": [
{
"category": "dataset",
"name": "records",
"function": "count",
"comparison": "EQ",
"value": "37"
}
]
}
],
"custom": [
{
"extension": "software",
"softwareName": "MARTILQREFERENCE",
"author": "Meerkat@merebox.com",
"version": "0.0.1"
},
{
"extension": "spatial",
"country": "Netherland",
"region": "",
"town": "Amsterdam"
},
{
"extension": "temporal",
"businessDate": "2021-11-01T00:00:00+11:00",
"runDate": "2021-11-02T00:00:00+11:00"
}
]
}
```

0
docs/source/stylesheets/extra.css 100644
View File

5
docs/source/what.md
View File

@ -1,12 +1,11 @@
What is marti # What is marti
=============
The foundation pillar for the **martiLQ** framework is the [martiLQ document](martiLQ.md) The foundation pillar for the **martiLQ** framework is the [martiLQ document](martiLQ.md)
that defines the reconciliation and other metadata of the document / file being transferred. that defines the reconciliation and other metadata of the document / file being transferred.
A definition, while fundamental, benefits from having tools that can create, read and A definition, while fundamental, benefits from having tools that can create, read and
interpret the definition. The **martiLQ** framework is about providing those tools interpret the definition. The **martiLQ** framework is about providing those tools
and a ecosystem that can be added to. and an ecosystem that can be added to.
The majority of the effort in creating **martiLQ** is in the tools. There are tools The majority of the effort in creating **martiLQ** is in the tools. There are tools
for various programming languages and situations. As many programming languages for various programming languages and situations. As many programming languages

6
docs/source/when.md
View File

@ -1,5 +1,4 @@
When would you use martiLQ # When would you use martiLQ
==========================
You are likely to start using the **martiLQ** framework when: You are likely to start using the **martiLQ** framework when:
@ -12,8 +11,7 @@ If you already have a standard and it works for you, and you have no upcoming (l
initiative that would benefit from the framework, then stick with what you have. The benefits initiative that would benefit from the framework, then stick with what you have. The benefits
of the framework are unlikely to weigh in your framework. of the framework are unlikely to weigh in your framework.
Read the material ## Read the material
-----------------
Please read the material before jumping in and make sure the **martiLQ** framework will Please read the material before jumping in and make sure the **martiLQ** framework will
provide you with the expected benefits. Implementing the framework and then changing provide you with the expected benefits. Implementing the framework and then changing

22
source/golang/client/src/config/martilq.ini
View File

@ -7,21 +7,21 @@ dataPath =
[MartiLQ] [MartiLQ]
tags = test,sample tags =
publisher = meerkat@merebox.com publisher =
contactPoint = Meerkat contactPoint =
accessLevel = Confidential accessLevel =
rights = None rights =
license = MIT license =
batch = @./config/batch.no batch = @./config/batch.no
theme = GOLANG theme =
[Resources] [Resources]
author = Mobidick author =
title = {{documentName}} title = {{documentName}}
state = active state =
expires = 2:0:0 expires = 2:0:0
encoding = UTF-8 encoding = UTF-8
version = version =
@ -43,13 +43,13 @@ password =
[Custom_Spatial] [Custom_Spatial]
enabled = true enabled = false
country = Netherland country = Netherland
region = region =
town = Amsterdam town = Amsterdam
[Custom_Temporal] [Custom_Temporal]
enabled = true enabled = false
businessDate = {{yesterday}} businessDate = {{yesterday}}
runDate = {{today}} runDate = {{today}}

17
source/golang/client/src/main.go
View File

@ -17,6 +17,7 @@ type Parameters struct {
task string task string
sourcePath string sourcePath string
recursive bool recursive bool
update bool
urlPrefix string urlPrefix string
configPath string configPath string
definitionPath string definitionPath string
@ -99,6 +100,17 @@ func loadArguments(args []string) {
} }
if args[ix] == "-R" || args[ix] == "--recursive" {
matched = true
params.recursive = true
}
if args[ix] == "--update" {
matched = true
params.update = true
}
if args[ix] == "--title" { if args[ix] == "--title" {
matched = true matched = true
@ -227,6 +239,11 @@ func main () {
panic("Missing 'output' parameter") panic("Missing 'output' parameter")
} }
_, err := os.Stat(params.definitionPath)
if err == nil && params.update == false {
panic("MartiLQ document '"+ params.definitionPath+"' already exists and update not specified")
}
m := martilq.ProcessFilePath(params.configPath, params.sourcePath, params.recursive, params.urlPrefix, params.definitionPath ) m := martilq.ProcessFilePath(params.configPath, params.sourcePath, params.recursive, params.urlPrefix, params.definitionPath )
if params.title != "" { if params.title != "" {
m.Title = params.title m.Title = params.title

20
source/golang/client/src/martilq/marti.go
View File

@ -210,6 +210,7 @@ func ProcessFilePath(ConfigPath string, SourcePath string, Recursive bool, UrlPr
} }
fileStat, err := os.Stat(SourcePath) fileStat, err := os.Stat(SourcePath)
fileAbs, err :=filepath.Abs(SourcePath)
if err != nil { if err != nil {
panic("Source path does not exist or is inaccessible: " + SourcePath) panic("Source path does not exist or is inaccessible: " + SourcePath)
} else { } else {
@ -222,24 +223,27 @@ func ProcessFilePath(ConfigPath string, SourcePath string, Recursive bool, UrlPr
} }
if fileStat.IsDir() { if fileStat.IsDir() {
diffCheck := fileAbs+string(os.PathSeparator)
filepath.Walk(SourcePath, func(path string, info os.FileInfo, err error) error { filepath.Walk(fileAbs, func(path string, info os.FileInfo, err error) error {
if err != nil { if err != nil {
log.Fatalf(err.Error()) log.Fatalf(err.Error())
} }
if info.IsDir() { if info.IsDir() == false {
if Recursive { diff := strings.Replace(path, diffCheck, "", -1)
if Recursive || diff == info.Name() {
url := UrlPrefix+strings.Replace(diff, "\\", "/", -1)
if UrlPrefix[0:6] == "file://" || UrlPrefix[0:1] == "\\\\" {
url = UrlPrefix+diff
}
m.AddResource(info.Name(), path, url)
} }
} else {
url := UrlPrefix+info.Name()
m.AddResource(info.Name(), path, url)
} }
return nil return nil
}) })
} else { } else {
url := UrlPrefix+fileStat.Name() url := UrlPrefix+fileStat.Name()
m.AddResource(fileStat.Name(), SourcePath, url) m.AddResource(fileStat.Name(), fileAbs, url)
} }
} }