Continued work on documentation

Marti.md

@@ -1,9 +1,9 @@
 # Marti document
 
 The metadata reconciliation transfer information is referred
-to as the **Marti** document throughout this documentation.
+to as the **marti** document throughout this documentation.
 
-The **Marti** document can be part of a message or a document
+The **marti** document can be part of a message or a document
 in its own right.  If the document is a file then the recommended
 name for the document is the same name as the data file,
 including extension, with the added extension of ``.mti``

README.md

@@ -1,18 +1,18 @@
 # Marti
 
-Marti is for metadata reconcilation for transfer information.
+**marti** stands for metadata reconcilation for transfer information.
 
 The objective is the provide transfer information for high volume data such as
 in files.  The document (files) can be transferred via HTTPS, SFTP, message queue, 
 network share or other.  The transfer information being described here does not 
-need to arrive via the same channel and cluld be received via email or 
+need to arrive via the same channel and could be received via email or 
 even synchronous / asynchronous API. The transfer information does not dictate or 
 determine how the data is formatted.
 
 The transfer information can provide details on the document format, but in itself
 it does not understand the data fomrat.
 
-Marti is intended to provide minimum basic information on the transfer with  
+**marti** is intended to provide minimum basic information on the transfer with  
 ability to include additional optional information.  The metadata reconcilation 
 transfer document being decscribed here wil be referred to as the [Marti](Marti.md)
 document throughout this documentation.

docs/README.md

@@ -0,0 +1,22 @@
+# Documentation
+
+**marti** stands for metadata reconcilation for transfer information.
+
+Before starting with **marti** it is advisable to understand if it is right for
+your organisation's needs.  Information is available in a number of short 
+documents.
+
+There is no quickstart document to get you started as each use case and
+organisation is different. There are sample implementations which you
+can adjust if they resonate with your circumstances,
+ see [sample implementations](samples/)
+
+## Index
+
+1. [What is marti](what.md)
+2. [Who would use marti](who.md)
+3. [Why you would use marti](why.md)
+4. [When would you start using marti](when.md)
+5. [Load quality metrics support](quality.md)
+6. [Comparison of marti definition](comparison.md)
+7. [References](references.md)

docs/comparison.md

@@ -0,0 +1,4 @@
+# Comparison of marti definition
+
+
+

docs/quality.md

@@ -1,11 +1,11 @@
 # Quality definition
 
-The Marti definition allows for the inclusion of a load quality 
+The **marti** definition allows for the inclusion of a load quality 
 definition.  This load quality definition is intended to be
 able to be applied universally with common tools.  As such not 
 all needs are covered.
 
-
+## Defined laod quality metrics
 
 
 * Number of records in the document - This is the number of data primary records not the 

docs/samples/README.md

@@ -0,0 +1,4 @@
+# Sample implementations
+
+The following samples are provided 
+

docs/what.md

@@ -0,0 +1,18 @@
+# What is marti
+
+The foundation pillar for the **marti** framework is the [marti document](../marti.md) 
+that defines the reconciliation and other metadata of the document / file being transferred.
+
+A definition, while fundamental, benefits from having tools that can create, read and 
+interpret the definition.  The **marti** framework is about providing those tools
+and a ecosystem that can be added to.
+
+The majority of the effort in creating **marti** is in the tools.  There are tools
+for various programming languages and situations.  As many programming languages
+generate portable programs that can execute on multiple operating systems, the
+likelihood is that a tools exists for you.
+
+The source for tools is provided in this repository and some have precompiled
+images.
+
+See the [source](../source/) directory for more details.

docs/when.md

@@ -0,0 +1,24 @@
+# When would you use marti
+
+You are likely to start using the **marti** framework when:
+1. you have no existing standard or framework or;
+2. your existing standards no longer meets you needs or;
+3. you are starting document exchange with another business or division that uses the framework or;
+4. there is a benefit to further automating your document processing in regard to audit and reconciliation
+
+If you already have a standard and it works for you, and you have no upcoming (large)
+initiative that would benefit from the framework, then stick with what you have.  The benefits
+of the framework are unlikely to weigh in your framework
+
+## Read the material
+
+Please read the material before jumping in and make sure the **marti** framework will
+provide you with the expected benefits.  Implementing the framework and then changing 
+your direction will more than likely add to your support costs.
+
+**Note**: As the framework is based on open structure and can be processed independently 
+without the tools, you can export the definitions to other tools / future methods.
+
+See also:
+1. [What the framework contains](what.md)
+2. [Quality definition](quality.md)

docs/who.md

@@ -0,0 +1,40 @@
+# Who is likely to use marti
+
+You are likely to find the **marti** framework relevant if you:
+
+1. Have many document exchanges
+2. Need to verify or reconcile the documents
+
+## Data exchanges
+
+If you are creating or receiving many documents or files on a regular basis
+then you probably have some framework defined.  The framework may be as simple as:
+
+1. The files are placed in given folders that have significance, such as the source or topic
+2. File names have a naming standard, such as subject domain and date of extract
+
+Simple framework such as the above have limitations, such as:
+
+* File names becoming long and need special parsing, with associated testing
+* Risk of overwriting 
+* New folders need to be created for new sources
+* Require constant polling, if passive
+* Lower automation prospects and alignment to DataSecOps
+* Poor fit to web applications (they tend to be designed for FTP and LAN)
+
+## Framework Sidecar files
+
+The **marti** framework addresses the issues and limitations by using sidecar
+or shadow files. The [concept of sidecar files](https://en.wikipedia.org/wiki/Sidecar_file) is 
+not new and are commonly found associated to media file processing.
+
+Sidecar files can also be implemented as ``forks`` and built into the operating system, such as 
+in Mac OS X HFS.  The Microsoft NTFS supports Alternate Data Streams to achieve a similar outcome.
+Unfortunately this information is not transferrable to other systems. 
+
+The proposition is to define a format for the sidecare file and provide common library tools that
+can be be used on multiple platforms when exchanging documents / files.
+
+See also:
+1. [When to use the framework](when.md)
+2. [What the framework contains](what.md)

docs/why.md

@@ -0,0 +1,37 @@
+# Why use marti
+
+**marti** is a framework for providing a degree of auditability and reconciliation of
+documents transferred between systems in an organisation and externally.  It does not intend
+define the format or content of the document.  It defines controls that can be used to:
+
+1. verify source
+2. identify unauthorized alterations (tampering)
+3. reconcile agreed metrics, such as the number of records
+4. ensure sequential processing
+5. describe metadata including format, extract time
+6. link to further information
+
+You would use **marti** if any of the controls are a requirement for you.
+
+## Documents
+
+Documents in this context are digital storage objects such as operating system files,
+cloud storage objects or blobs.  The document content has structure and contains multiple
+records.
+
+The **marti** framework is not intended to be used for single record transfers such as 
+in single web transactions.  It is for providing controls when moving large amounts of
+data as one event.  This data are commonly referred to as batch extracts and performed 
+at scheduled times such as end of day.
+
+## Security
+
+The framework does not replace your security, inflight encryption or encryption at rest.
+
+You are encouraged to use TLS or SSH to connect devices and transfer documents.  Storage 
+encryption and access controls for your documents is also relevant as part of the bigger 
+picture.
+
+See also:
+1. [Who would use the framework](who.md)
+2. [When to use the framework](when.md)

source/README.md

@@ -0,0 +1,5 @@
+# Contents
+
+This is the parent directory for **mati** tools written in various languages.
+
+Please browse the folder for the programming language of interest.