Wardrobe
/
martiLQ
mirror of https://github.com/meerkat-manor/martiLQ.git
3
0
Fork 0
Code Issues Projects Releases Wiki Activity

Refactored to martiLQ

draft_specifications
meerkat 2021-10-14 21:50:55 +11:00
parent f360bdd824
commit d515a6c4a6
25 changed files with 353 additions and 485 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
README.md
View File

@ -1,6 +1,6 @@
# Marti # Marti
**marti** stands for metadata reconcilation for transfer information. **martiLQ** stands for metadata reconcilation for transfer information, load quality.
The objective is to provide transfer information for high volume data such as The objective is to provide transfer information for high volume data such as
in files. The files can be transferred via HTTPS, SFTP, message queue, in files. The files can be transferred via HTTPS, SFTP, message queue,
@ -15,9 +15,9 @@ it does not understand the data format.
**Note**: The terms file and document are intended to be interchangeable **Note**: The terms file and document are intended to be interchangeable
through out this documentation. through out this documentation.
**marti** is intended to provide minimum basic information on the transfer with **martiLQ** is intended to provide minimum basic information on the transfer with
ability to include optional information. The metadata reconcilation ability to include optional information. The metadata reconcilation
transfer document being decscribed here wil be referred to as the [marti document](Marti.md) transfer document being decscribed here wil be referred to as the [martiLQ document](MartiLQ.md)
throughout this documentation. throughout this documentation.
The transfer information is supplied as a separate document which could be another file The transfer information is supplied as a separate document which could be another file
@ -39,8 +39,8 @@ information
## Transfer information ## Transfer information
The information in the **marti** document is summarised below. For more detailed The information in the **martiLQ** document is summarised below. For more detailed
information see [marti definition](marti.md) information see [martiLQ definition](martiLQ.md)
### Mandatory information ### Mandatory information
@ -61,6 +61,7 @@ The optional information is:
* Contact point * Contact point
* Acces level * Acces level
* Rights * Rights
* Batch
* License * License
* Spatial (*) * Spatial (*)
* Temporal (*) * Temporal (*)

16
docs/README.md
View File

@ -1,8 +1,8 @@
# Documentation # Documentation
**marti** stands for metadata reconcilation for transfer information. **martiLQ** stands for metadata reconcilation for transfer information.
Before starting with **marti** it is advisable to understand if it is right for Before starting with **martiLQ** it is advisable to understand if it is right for
your organisation's needs. Information is available in a number of short your organisation's needs. Information is available in a number of short
documents. documents.
@ -13,12 +13,12 @@ can adjust if they resonate with your circumstances,
## Index ## Index
1. [What is marti](what.md) 1. [What is martiLQ](what.md)
2. [Who would use marti](who.md) 2. [Who would use martiLQ](who.md)
3. [Why you would use marti](why.md) 3. [Why you would use martiLQ](why.md)
4. [When would you start using marti](when.md) 4. [When would you start using martiLQ](when.md)
5. [Load quality metrics support](quality.md) 5. [Load quality metrics support](quality.md)
6. [Comparison of marti definition](comparison.md) 6. [Comparison of martiLQ definition](comparison.md)
7. [References](references.md) 7. [References](references.md)
[!INCLUDE [marti High Level Definition](../marti.md)] [!INCLUDE [martiLQ High Level Definition](../martiLQ.md)]

15
docs/comparison.md
View File

@ -1,4 +1,4 @@
# Comparison of marti document definition # Comparison of martiLQ document definition
The use of metadata definitions is not unique and examples The use of metadata definitions is not unique and examples
exist in many different situations. Some are standard and open exist in many different situations. Some are standard and open
@ -8,20 +8,21 @@ Some open standards are EXIF data for pictures, SQL DDL defintions
for databases, the XMP definition and web header responses before the for databases, the XMP definition and web header responses before the
web content. web content.
The **marti** document definition is intended to cover the situation The **martiLQ** document definition is intended to cover the situation
where data files are being transferred and reconciliation is required. where data files are being transferred and reconciliation is required.
The **marti** document definition is modelled on the [CKAN API metadata](https://docs.ckan.org/en/2.9/api/index.html) The **martiLQ** document definition is modelled on
the [CKAN API metadata](https://docs.ckan.org/en/2.9/api/index.html)
which has been adapted to included additional elements relevant to when which has been adapted to included additional elements relevant to when
you are exchanging data files. This includes the reconciliation elements you are exchanging data files. This includes the reconciliation elements
such as number of records and file hash. such as number of records and file hash.
As the definition is based on the CKAN API, there are tools to import As the definition is based on the CKAN API, there are tools to import
a CKAN source into a **marti** document definition and then process the data a CKAN source into a **martiLQ** document definition and then process the data
through the pipeline as you would for any other data file that had a through the pipeline as you would for any other data file that had a
**marti** document definition. **martiLQ** document definition.
## Benefit of CKAN and marti ## Benefit of CKAN and martiLQ
The CKAN is excellent at defining the data source details but it lacks information The CKAN is excellent at defining the data source details but it lacks information
for load quality. If you have CKAN deployed in your organisation and wish for load quality. If you have CKAN deployed in your organisation and wish
@ -30,7 +31,7 @@ CKAN and marti.
Samples exist on CKAN integration. Samples exist on CKAN integration.
## Magda and marti ## Magda and martiLQ
Another source of data is [Magda](https://magda.io/) which has API metadata Another source of data is [Magda](https://magda.io/) which has API metadata
definitions. Magda is more about data fedaration and as such provides definitions. Magda is more about data fedaration and as such provides

8
docs/distribution.md
View File

@ -8,7 +8,7 @@ such as end of day.
Some files may expand to multiple files if they are Some files may expand to multiple files if they are
compressed with a utility such as WinZIP or 7ZIP. In the situation compressed with a utility such as WinZIP or 7ZIP. In the situation
where a ZIP file expands to multiple documents, then the expectation is where a ZIP file expands to multiple documents, then the expectation is
that the ZIP file contains a **marti** document describing its contents. that the ZIP file contains a **martiLQ** document describing its contents.
The elements in the distribution section are: The elements in the distribution section are:
@ -41,15 +41,15 @@ The following are optional in the distribution section.
## Compression ## Compression
Files can be compressed using a utility. A single compressed file can contain Files can be compressed using a utility. A single compressed file can contain
multiple files. The **marti** definition document applies to the compressed file multiple files. The **martiLQ** definition document applies to the compressed file
and not to the contents, which could be multiple files. and not to the contents, which could be multiple files.
In the case of a compressed files, there should be a **marti** definition document in the In the case of a compressed files, there should be a **martiLQ** definition document in the
compressed file. compressed file.
Compression of files always occur before encryption. Compression of files always occur before encryption.
### Marti definition for Compressed File ### martiLQ definition for Compressed File
For a compressed file that is not encrypted, the distribution definition will be: For a compressed file that is not encrypted, the distribution definition will be:

2
docs/quality.md
View File

@ -1,6 +1,6 @@
# Quality definition # Quality definition
The **marti** definition allows for the inclusion of a load quality The **martiLQ** definition allows for the inclusion of a load quality
definition. This load quality definition is intended to be definition. This load quality definition is intended to be
able to be applied universally with common tools. As such not able to be applied universally with common tools. As such not
all needs are covered. all needs are covered.

41
docs/samples/powershell/Invoke-SampleGenerateBsb.ps1
View File

@ -1,10 +1,7 @@
. ..\..\..\source\powershell\New-Marti.ps1 . ..\..\..\source\powershell\MartiLQ.ps1
. ..\..\..\source\powershell\New-MartiChildItem.ps1 . ..\..\..\source\powershell\MartiLQItem.ps1
. ..\..\..\source\powershell\New-MartiResource.ps1 . ..\..\..\source\powershell\Compress-MartiLQ.ps1
. ..\..\..\source\powershell\Get-Marti.ps1
. ..\..\..\source\powershell\Compress-Marti.ps1
. ..\..\..\source\powershell\Get-MartiResourceAttributes.ps1
function PullFtpFile { function PullFtpFile {
@ -89,7 +86,7 @@ $fileList = ListFtpDirectory -Username "anonymous" -Password "anon@merebox.com"
Write-Host "File list size: $($fileList.count)" -ForegroundColor Gray Write-Host "File list size: $($fileList.count)" -ForegroundColor Gray
Write-Host "Now iterate through the remote files and build remote marti list " -ForeGroundColor Green Write-Host "Now iterate through the remote files and build remote martiLQ list " -ForeGroundColor Green
$oMarti = New-MartiDefinition $oMarti = New-MartiDefinition
$oMarti.title = "Remote_BSB_data" $oMarti.title = "Remote_BSB_data"
@ -101,7 +98,7 @@ $oMarti.theme = "payment"
ForEach ($item in $fileList) { ForEach ($item in $fileList) {
if ($item -ne "" -and $item.startswith("BSBDirectory")) { if ($item -ne "" -and $item.startswith("BSBDirectory")) {
PullFtpFile -Username "anonymous" -Password "anon@merebox.com" -RemoteFile ($remoteDirectory + $item) -OutputPath (Join-Path -Path $localDirectory -ChildPath $item) PullFtpFile -Username "anonymous" -Password "anon@merebox.com" -RemoteFile ($remoteDirectory + $item) -OutputPath (Join-Path -Path $localDirectory -ChildPath $item)
Write-Host "Add BSB $item file to Remote marti metadata sample " -ForeGroundColor Yellow Write-Host "Add BSB $item file to Remote martiLQ metadata sample " -ForeGroundColor Yellow
$oResource = New-MartiResource -SourcePath (Join-Path -Path $localDirectory -ChildPath $item) -UrlPath $remoteDirectory -LogPath ".\test\Logs" -ExtendAttributes $oResource = New-MartiResource -SourcePath (Join-Path -Path $localDirectory -ChildPath $item) -UrlPath $remoteDirectory -LogPath ".\test\Logs" -ExtendAttributes
if ($item.endswith(".txt")) { if ($item.endswith(".txt")) {
Set-AttributeValueNumber -Attributes $oResource.attributes -Key "header" -Category "dataset" -Function "count" -Value 1 Set-AttributeValueNumber -Attributes $oResource.attributes -Key "header" -Category "dataset" -Function "count" -Value 1
@ -116,13 +113,12 @@ ForEach ($item in $fileList) {
} }
} }
$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBRemote.mri.json" $fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBRemote.mti"
$oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson $oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson
Write-Host "Remote marti definition file is $fileJson " -ForeGroundColor Green Write-Host "Remote martiLQ definition file is $fileJson " -ForeGroundColor Green
Write-Host "Note that file has been given JSON extension for sample purposes" -ForeGroundColor Gray
Write-Host "Now iterate through the local files and build marti ZIP " -ForeGroundColor Green Write-Host "Now iterate through the local files and build martiLQ ZIP " -ForeGroundColor Green
$oMarti = New-MartiDefinition $oMarti = New-MartiDefinition
$oMarti.title = "Zip_BSB_data" $oMarti.title = "Zip_BSB_data"
@ -139,7 +135,7 @@ if (Test-Path -Path $zipFile) {
foreach($file in Get-ChildItem $localDirectory) foreach($file in Get-ChildItem $localDirectory)
{ {
if ($file.Name.startswith("BSBDirectory") -and !($file.Name.EndsWith(".zip")) -and !($file.Name.EndsWith(".7z")) ) { if ($file.Name.startswith("BSBDirectory") -and !($file.Name.EndsWith(".zip")) -and !($file.Name.EndsWith(".7z")) ) {
Write-Host "Add BSB file $file to ZIP marti metadata sample " -ForeGroundColor Yellow Write-Host "Add BSB file $file to ZIP martiLQ metadata sample " -ForeGroundColor Yellow
Compress-Archive -Path $file.FullName -DestinationPath $zipFile -Update Compress-Archive -Path $file.FullName -DestinationPath $zipFile -Update
$oResource = New-MartiResource -SourcePath $file.FullName -UrlPath $localDirectory -LogPath ".\test\Logs" -ExtendAttributes $oResource = New-MartiResource -SourcePath $file.FullName -UrlPath $localDirectory -LogPath ".\test\Logs" -ExtendAttributes
if ($file.Extension -eq ".txt") { if ($file.Extension -eq ".txt") {
@ -159,10 +155,9 @@ $oResource = New-MartiResource -SourcePath $zipFile -UrlPath $localDirectory -Lo
Set-AttributeValueString -Attributes $oResource.attributes -Key "compression" -Category "format" -Function "algorithm" -Value "WINZIP" Set-AttributeValueString -Attributes $oResource.attributes -Key "compression" -Category "format" -Function "algorithm" -Value "WINZIP"
$oMarti.resources += $oResource $oMarti.resources += $oResource
$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBZip.mri.json" $fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBZip.mti"
$oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson $oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson
Write-Host "ZIP marti definition file is $fileJson " -ForeGroundColor Green Write-Host "ZIP martiLQ definition file is $fileJson " -ForeGroundColor Green
Write-Host "Note that file has been given JSON extension for sample purposes" -ForeGroundColor Gray
@ -175,10 +170,9 @@ $oMarti.contactPoint = "meerkat@merebox.com"
$oMarti.landingPage = "https://github.com/meerkat-manor/marti/blob/draft_specifications/docs/samples/powershell/Invoke-BSBSample.ps1" $oMarti.landingPage = "https://github.com/meerkat-manor/marti/blob/draft_specifications/docs/samples/powershell/Invoke-BSBSample.ps1"
$oMarti.theme = "payment" $oMarti.theme = "payment"
$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBLocal.mri.json" $fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBLocal.mti"
$oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson $oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson
Write-Host "Local marti definition file is $fileJson " -ForeGroundColor Green Write-Host "Local martiLQ definition file is $fileJson " -ForeGroundColor Green
Write-Host "Note that file has been given JSON extension for sample purposes" -ForeGroundColor Gray
Write-Host "Now create an encrypted 7ZIP file with asymmetric password protection" -ForeGroundColor Green Write-Host "Now create an encrypted 7ZIP file with asymmetric password protection" -ForeGroundColor Green
@ -198,7 +192,7 @@ if (Test-Path -Path $zipFile) {
foreach($file in Get-ChildItem $localDirectory) foreach($file in Get-ChildItem $localDirectory)
{ {
if ($file.Name.startswith("BSBDirectory") -and !($file.Name.EndsWith(".zip")) -and !($file.Name.EndsWith(".7z")) ) { if ($file.Name.startswith("BSBDirectory") -and !($file.Name.EndsWith(".zip")) -and !($file.Name.EndsWith(".7z")) ) {
Write-Host "Add BSB file $file to 7ZIP marti metadata sample " -ForeGroundColor Yellow Write-Host "Add BSB file $file to 7ZIP martiLQ metadata sample " -ForeGroundColor Yellow
if (Test-Path -Path $zipFile) { if (Test-Path -Path $zipFile) {
Compress-7Zip -Path $file.FullName -ArchiveFileName $zipFile -Format SevenZip -Append Compress-7Zip -Path $file.FullName -ArchiveFileName $zipFile -Format SevenZip -Append
} else { } else {
@ -220,7 +214,7 @@ foreach($file in Get-ChildItem $localDirectory)
} }
$noticeFile = Join-Path -Path $localDirectory -ChildPath "README.txt" $noticeFile = Join-Path -Path $localDirectory -ChildPath "README.txt"
Set-Content -Path $noticeFile -Value "Generated by marti Samples" Set-Content -Path $noticeFile -Value "Generated by martiLQ Samples"
$oResource = New-MartiResource -SourcePath $noticeFile -UrlPath $localDirectory -LogPath ".\test\Logs" $oResource = New-MartiResource -SourcePath $noticeFile -UrlPath $localDirectory -LogPath ".\test\Logs"
$oMarti.resources += $oResource $oMarti.resources += $oResource
@ -233,10 +227,9 @@ $oResource.encryption = New-Encryption -Algorithm "Passphrase" -Value $secret
Set-AttributeValueString -Attributes $oResource.attributes -Key "compression" -Category "format" -Function "algorithm" -Value "7ZIP" Set-AttributeValueString -Attributes $oResource.attributes -Key "compression" -Category "format" -Function "algorithm" -Value "7ZIP"
$oMarti.resources += $oResource $oMarti.resources += $oResource
$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBSecure.mri.json" $fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBSecure.mti"
$oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson $oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson
Write-Host "Secure 7ZIP marti definition file is $fileJson " -ForeGroundColor Green Write-Host "Secure 7ZIP martiLQ definition file is $fileJson " -ForeGroundColor Green
Write-Host "Note that file has been given JSON extension for sample purposes" -ForeGroundColor Gray
Write-Host "Sample execution completed" -ForeGroundColor Green Write-Host "Sample execution completed" -ForeGroundColor Green

20
docs/samples/powershell/Invoke-SampleGenerateBsbSecure.ps1
View File

@ -1,10 +1,7 @@
. ..\..\..\source\powershell\New-Marti.ps1 . ..\..\..\source\powershell\MartiLQ.ps1
. ..\..\..\source\powershell\New-MartiChildItem.ps1 . ..\..\..\source\powershell\MartiLQItem.ps1
. ..\..\..\source\powershell\New-MartiResource.ps1 . ..\..\..\source\powershell\Compress-MartiLQ.ps1
. ..\..\..\source\powershell\Get-Marti.ps1
. ..\..\..\source\powershell\Compress-Marti.ps1
. ..\..\..\source\powershell\Get-MartiResourceAttributes.ps1
function PullFtpFile { function PullFtpFile {
@ -127,7 +124,7 @@ if (Test-Path -Path $zipFile) {
foreach($file in Get-ChildItem $localDirectory) foreach($file in Get-ChildItem $localDirectory)
{ {
if ($file.Name.startswith("BSBDirectory") -and $file.Name.EndsWith(".csv") ) { if ($file.Name.startswith("BSBDirectory") -and $file.Name.EndsWith(".csv") ) {
Write-Host "Add BSB file $file to 7ZIP marti metadata sample " -ForeGroundColor Yellow Write-Host "Add BSB file $file to 7ZIP martiLQ metadata sample " -ForeGroundColor Yellow
if (Test-Path -Path $zipFile) { if (Test-Path -Path $zipFile) {
Compress-7Zip -Path $file.FullName -ArchiveFileName $zipFile -Format SevenZip -Append Compress-7Zip -Path $file.FullName -ArchiveFileName $zipFile -Format SevenZip -Append
} else { } else {
@ -149,7 +146,7 @@ foreach($file in Get-ChildItem $localDirectory)
} }
$noticeFile = Join-Path -Path $localDirectory -ChildPath "README.txt" $noticeFile = Join-Path -Path $localDirectory -ChildPath "README.txt"
Set-Content -Path $noticeFile -Value "Generated by marti Samples" Set-Content -Path $noticeFile -Value "Generated by martiLQ Samples"
$oResource = New-MartiResource -SourcePath $noticeFile -UrlPath $localDirectory -LogPath ".\test\Logs" $oResource = New-MartiResource -SourcePath $noticeFile -UrlPath $localDirectory -LogPath ".\test\Logs"
$oMarti.resources += $oResource $oMarti.resources += $oResource
@ -167,9 +164,8 @@ Write-Debug "Secret: $secret"
Set-AttributeValueString -Attributes $oResource.attributes -Key "compression" -Category "format" -Function "algorithm" -Value "7ZIP" Set-AttributeValueString -Attributes $oResource.attributes -Key "compression" -Category "format" -Function "algorithm" -Value "7ZIP"
$oMarti.resources += $oResource $oMarti.resources += $oResource
$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSB7ZipPKI.mri.json" $fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSB7ZipPKI.mti"
$oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson $oMarti | ConvertTo-Json -depth 50 | Out-File $fileJson
Write-Host "7ZIP marti definition file is $fileJson " -ForeGroundColor Green Write-Host "7ZIP martiLQ definition file is $fileJson " -ForeGroundColor Green
Write-Host "Note that file has been given JSON extension for sample purposes" -ForeGroundColor Gray
Write-Host "Sample secure execution completed" -ForeGroundColor Green Write-Host "Sample secure execution completed" -ForeGroundColor Green

10
docs/samples/python/SampleGenerateBsb.py
View File

@ -59,9 +59,9 @@ for file_name in files:
if file_name.endswith(".csv") | file_name.endswith(".txt"): if file_name.endswith(".csv") | file_name.endswith(".txt"):
file_remote = remote_dir + file_name file_remote = remote_dir + file_name
file_local = "./test/" + file_name file_local = "./test/" + file_name
# ftpPull(remote_host, file_remote, file_local) ftpPull(remote_host, file_remote, file_local)
print("Creating marti definition") print("Creating martiLQ definition")
mlq = martiLQ() mlq = martiLQ()
oMarti = mlq.NewMartiDefinition() oMarti = mlq.NewMartiDefinition()
@ -72,15 +72,15 @@ for file_name in files:
oMarti["resources"].append(oResource) oMarti["resources"].append(oResource)
mlq.CloseLog() mlq.CloseLog()
print("Save marti definition") print("Save martiLQ definition")
jd = json.dumps(oMarti, indent=5) jd = json.dumps(oMarti, indent=5)
jsonFile = open("./test/BSBDirectoryPlain.mri.json", "w") jsonFile = open("./test/BSBDirectoryPlain.mti", "w")
jsonFile.write(jd) jsonFile.write(jd)
jsonFile.close() jsonFile.close()
print("Sample completed: SampleGenerateBsb.py") print("Sample completed: SampleGenerateBsb.py")
lqresults, testError = mlq.TestMartiDefinition(oMarti, "./test/BSBDirectoryPlain.mri.json") lqresults, testError = mlq.TestMartiDefinition(oMarti, "./test/BSBDirectoryPlain.mti")
testfile = open("./test/LoadQualityTest01.csv", "w+", newline ="") testfile = open("./test/LoadQualityTest01.csv", "w+", newline ="")
with testfile: with testfile:

6
docs/what.md
View File

@ -1,13 +1,13 @@
# What is marti # What is marti
The foundation pillar for the **marti** framework is the [marti document](../marti.md) The foundation pillar for the **martiLQ** framework is the [martiLQ document](../martiLQ.md)
that defines the reconciliation and other metadata of the document / file being transferred. that defines the reconciliation and other metadata of the document / file being transferred.
A definition, while fundamental, benefits from having tools that can create, read and A definition, while fundamental, benefits from having tools that can create, read and
interpret the definition. The **marti** framework is about providing those tools interpret the definition. The **martiLQ** framework is about providing those tools
and a ecosystem that can be added to. and a ecosystem that can be added to.
The majority of the effort in creating **marti** is in the tools. There are tools The majority of the effort in creating **martiLQ** is in the tools. There are tools
for various programming languages and situations. As many programming languages for various programming languages and situations. As many programming languages
generate portable programs that can execute on multiple operating systems, the generate portable programs that can execute on multiple operating systems, the
likelihood is that a tools exists for you. likelihood is that a tools exists for you.

6
docs/when.md
View File

@ -1,6 +1,6 @@
# When would you use marti # When would you use martiLQ
You are likely to start using the **marti** framework when: You are likely to start using the **martiLQ** framework when:
1. you have no existing standard or framework or; 1. you have no existing standard or framework or;
2. your existing standards no longer meets you needs or; 2. your existing standards no longer meets you needs or;
3. you are starting document exchange with another business or division that uses the framework or; 3. you are starting document exchange with another business or division that uses the framework or;
@ -12,7 +12,7 @@ of the framework are unlikely to weigh in your framework
## Read the material ## Read the material
Please read the material before jumping in and make sure the **marti** framework will Please read the material before jumping in and make sure the **martiLQ** framework will
provide you with the expected benefits. Implementing the framework and then changing provide you with the expected benefits. Implementing the framework and then changing
your direction will more than likely add to your support costs. your direction will more than likely add to your support costs.

4
docs/who.md
View File

@ -1,6 +1,6 @@
# Who is likely to use marti # Who is likely to use marti
You are likely to find the **marti** framework relevant if you: You are likely to find the **martiLQ** framework relevant if you:
1. Have many document exchanges 1. Have many document exchanges
2. Need to verify or reconcile the documents 2. Need to verify or reconcile the documents
@ -24,7 +24,7 @@ Simple framework such as the above have limitations, such as:
## Framework Sidecar files ## Framework Sidecar files
The **marti** framework addresses the issues and limitations by using sidecar The **martiLQ** framework addresses the issues and limitations by using sidecar
or shadow files. The [concept of sidecar files](https://en.wikipedia.org/wiki/Sidecar_file) is or shadow files. The [concept of sidecar files](https://en.wikipedia.org/wiki/Sidecar_file) is
not new and are commonly found associated to media file processing. not new and are commonly found associated to media file processing.

6
docs/why.md
View File

@ -1,6 +1,6 @@
# Why use marti # Why use marti
**marti** is a framework for providing a degree of auditability and reconciliation of **martiLQ** is a framework for providing a degree of auditability and reconciliation of
documents transferred between systems in an organisation and externally. It does not intend documents transferred between systems in an organisation and externally. It does not intend
define the format or content of the document. It defines controls that can be used to: define the format or content of the document. It defines controls that can be used to:
@ -11,7 +11,7 @@ define the format or content of the document. It defines controls that can be u
5. describe metadata including format, extract time 5. describe metadata including format, extract time
6. link to further information 6. link to further information
You would use **marti** if any of the controls are a requirement for you. You would use **martiLQ** if any of the controls are a requirement for you.
## Documents ## Documents
@ -19,7 +19,7 @@ Documents in this context are digital storage objects such as operating system f
cloud storage objects or blobs. The document content has structure and contains multiple cloud storage objects or blobs. The document content has structure and contains multiple
records. records.
The **marti** framework is not intended to be used for single record transfers such as The **martiLQ** framework is not intended to be used for single record transfers such as
in single web transactions. It is for providing controls when moving large amounts of in single web transactions. It is for providing controls when moving large amounts of
data as one event. This data are commonly referred to as batch extracts and performed data as one event. This data are commonly referred to as batch extracts and performed
at scheduled times such as end of day. at scheduled times such as end of day.

6
Marti.md → martiLQ.md
View File

@ -1,9 +1,9 @@
# Marti document # MartiLQ document
The metadata reconciliation transfer information is referred The metadata reconciliation transfer information is referred
to as the **marti** document throughout this documentation. to as the **martiLQ** document throughout this documentation.
The **marti** document can be part of a message or a document The **martiLQ** document can be part of a message or a document
in its own right. If the document is a file then the recommended in its own right. If the document is a file then the recommended
name for the document is the same name as the data file, name for the document is the same name as the data file,
including extension, with the added extension of ``.mti`` including extension, with the added extension of ``.mti``

2
source/README.md
View File

@ -1,5 +1,5 @@
# Contents # Contents
This is the parent directory for **marti** tools written in various languages. This is the parent directory for **martiLQ** tools written in various languages.
Please browse the folder for the programming language of interest. Please browse the folder for the programming language of interest.

2
source/powershell/Compare-MartiResource.ps1
View File

@ -16,7 +16,7 @@ function Compare-MartiResource {
if ($null -eq $Resource) { if ($null -eq $Resource) {
$Global:MartiErrorId = "MRI2201" $Global:MartiErrorId = "MRI2201"
$message = "No Marti resource definition supplied" $message = "No resource definition supplied"
Write-Log ($message + " " + $Global:MartiErrorId) Write-Log ($message + " " + $Global:MartiErrorId)
Close-Log Close-Log
throw $message throw $message

66
source/powershell/Compress-Marti.ps1 → source/powershell/Compress-MartiLQ.ps1
View File

@ -1,68 +1,6 @@
$script:LogPathName = ""
$script:SoftwareVersion = "0.0.1"
$global:default_metaFile = "##marti##.mri"
function Get-LogName {
$date = Get-Date -f "yyyy-MM-dd"
if (($null -eq $script:LogPathName) -or ($script:LogPathName -eq ""))
{
return $null
}
if (!(Test-Path -Path $script:LogPathName)) {
$null = New-Item -Path $script:LogPathName -ItemType Directory
}
$logName = $(Get-SoftwareName) + "_$date.log"
return Join-Path -Path $script:LogPathName -ChildPath $logName
}
function Write-Log { function Compress-MartiLQ
param(
[String] $LogEntry
)
$sFullPath = Get-LogName
$dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss"
if ($null -ne $sFullPath -and $sFullPath -ne "") {
if (!(Test-Path -Path $sFullPath)) {
Write-Host "Log path: $sFullPath"
$null = New-Item -Path $sFullPath -ItemType File
}
Add-Content -Path $sFullPath -Value "[$dateTime]. $LogEntry"
}
Write-Debug "[$dateTime]. $LogEntry"
}
function Open-Log {
$dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss"
Write-Log "***********************************************************************************"
Write-Log "* Start of processing: [$dateTime]"
Write-Log "***********************************************************************************"
}
function Close-Log {
$dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss"
Write-Log "***********************************************************************************"
Write-Log "* End of processing: [$dateTime]"
Write-Log "***********************************************************************************"
}
function Get-SoftwareName {
return [String] "MARTIREFERENCE"
}
function Compress-Marti
{ {
Param( Param(
[Parameter(Mandatory)][String] $SourceFolder, [Parameter(Mandatory)][String] $SourceFolder,
@ -76,7 +14,7 @@ Param(
Write-Debug "Parameter: LogPath Value: $LogPath " Write-Debug "Parameter: LogPath Value: $LogPath "
Open-Log Open-Log
Write-Log "Function 'Compress-Marti' parameters follow" Write-Log "Function 'Compress-MartiLQ' parameters follow"
Write-Log "Parameter: SourceFolder Value: $SourceFolder " Write-Log "Parameter: SourceFolder Value: $SourceFolder "
Write-Log "Parameter: ArchiveFile Value: $ArchiveFile " Write-Log "Parameter: ArchiveFile Value: $ArchiveFile "
Write-Log "Parameter: Filter Value: $Filter " Write-Log "Parameter: Filter Value: $Filter "

87
source/powershell/Get-Marti.ps1 → source/powershell/MartiLQ.ps1
View File

@ -2,7 +2,7 @@
$script:LogPathName = "" $script:LogPathName = ""
$script:SoftwareVersion = "0.0.1" $script:SoftwareVersion = "0.0.1"
$global:default_metaFile = "##marti##.mri" $global:default_metaFile = "##marti##.mti"
function Get-LogName { function Get-LogName {
@ -63,6 +63,89 @@ function Get-SoftwareName {
function New-MartiDefinition
{
$oSoftware = [PSCustomObject]@{
extension = "software"
softwareName = "MartiReference"
author = "Meerkat@merebox.com"
version = "$script:SoftwareVersion"
}
$publisher = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name
[System.Collections.ArrayList]$lcustom = @()
$lcustom += $oSoftware
[System.Collections.ArrayList]$lresource = @()
$oMarti = [PSCustomObject]@{
title = ""
uid = (New-Guid).ToString()
resources = $lresource
description = ""
modified = Get-Date -f "yyyy-MM-ddTHH:mm:ss"
tags = @( "document", "marti")
publisher = $publisher
contactPoint = ""
accessLevel = "Confidential"
rights = "Restricted"
license = ""
state = "active"
batch = 1
describedBy = ""
landingPage = ""
theme =""
custom = $lCustom
}
return $oMarti
}
function Set-MartiAttribute
{
Param(
[System.Collections.ArrayList] $Attributes,
[String] $ACategory,
[String] $AName,
[String] $AFunction,
[String] $Comparison,
[String] $Value
)
$matched = $false
$Attributes | ForEach-Object {
if ($_.category -eq $ACategory -and $_.name -eq $AName -and $_.function -eq $AFunction) {
$matched = $true
$_.comparison = $comparison
$_.value = $value
}
}
if (!($matched)) {
$oAttribute = [PSCustomObject]@{
category = $Acategory
name = $AName
function = $Afunction
comparison = $comparison
value = $value
}
$Attributes += $oAttribute
}
return $Attributes
}
function Get-MartChildResource function Get-MartChildResource
{ {
@ -87,7 +170,7 @@ function Get-MartChildResource
if ($null -eq $Marti) { if ($null -eq $Marti) {
$Global:MartiErrorId = "MRI2101" $Global:MartiErrorId = "MRI2101"
$message = "No Marti definition supplied" $message = "No definition supplied"
Write-Log ($message + " " + $Global:MartiErrorId) Write-Log ($message + " " + $Global:MartiErrorId)
Close-Log Close-Log
throw $message throw $message

173
source/powershell/Get-MartiResourceAttributes.ps1 → source/powershell/MartiLQItem.ps1
View File

@ -1,4 +1,176 @@
function New-MartiResource {
Param(
[Parameter(Mandatory)][String] $SourcePath,
[String] $UrlPath = "",
[switch] $ExcludeHash,
[switch] $ExtendAttributes,
[String] $LogPath
)
$Global:MartiErrorId = ""
$script:LogPathName = $LogPath
Write-Debug "Parameter: LogPath Value: $LogPath "
Open-Log
Write-Log "Function 'New-MartiResource' parameters follow"
Write-Log "Parameter: UrlPath Value: $UrlPath "
Write-Log "Parameter: SourcePath Value: $SourcePath "
Write-Log "Parameter: ExcludeHash Value: $ExcludeHash "
Write-Log ""
if (Test-Path -Path $SourcePath -PathType Leaf) {
$item = Get-Item -Path $SourcePath -Force
Write-Log "Define file $($item.FullName) "
if ($ExcludeHash) {
$hash = $null
} else {
$hash = New-MartiHash -Algorithm "SHA256" -FilePath $item.FullName
}
$lattribute = Set-MartiResourceAttributes -Path $item.FullName -FileType $item.Extension.Substring(1) -ExtendedAttributes:$ExtendAttributes
$oResource = [PSCustomObject]@{
title = $item.Name.Replace($item.Extension, "")
uid = (New-Guid).ToString()
documentName = $item.Name
issuedDate = Get-Date -f "yyyy-MM-ddTHH:mm:ss"
modified = $item.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss")
state = "active"
author = ""
length = $item.Length
hash = $hash
description = ""
url = ""
version = $version
format = $item.Extension.Substring(1)
compression = $null
encryption = $null
attributes = $lattribute
}
if ($null -ne $UrlPath -and $UrlPath -ne "") {
if ($UrlPath[$UrlPath.Length-1] -eq "/" -or $UrlPath[$UrlPath.Length-1] -eq "\\") {
$oResource.url = $UrlPath.Replace("\\", "/") + $item.Name
} else {
$oResource.url = $UrlPath.Replace("\\", "/") + "/" + $item.Name
}
}
} else {
$Global:MartiErrorId = "MRI2001"
$message = "Document '$SourcePath' not found or is a folder"
Write-Log ($message + " " + $Global:MartiErrorId)
Close-Log
throw $message
}
Close-Log
return $oResource
}
function New-MartiHash{
Param(
[Parameter(Mandatory)][String] $Algorithm,
[String] $FilePath,
[String] $Value = ""
)
if ($Value -eq "" -and $FilePath -ne "") {
$Value = (Get-FileHash -Path $FilePath -Algorithm $Algorithm).Hash
}
$oHash = [PSCustomObject]@{
algo = $Algorithm
value = $Value
}
return $oHash
}
function New-Encryption{
Param(
[Parameter(Mandatory)][String] $Algorithm,
[String] $Value
)
$oEncryption = [PSCustomObject]@{
algo = $Algorithm
value = $Value
}
return $oEncryption
}
function New-MartiChildItem
{
Param(
[Parameter(Mandatory)][String] $SourceFolder,
[String] $Filter ="*",
[String] $UrlPath,
[switch] $Recurse,
[switch] $ExtendAttributes,
[switch] $ExcludeHash,
[String] $LogPath
)
$script:LogPathName = $LogPath
Write-Debug "Parameter: LogPath Value: $LogPath "
Open-Log
Write-Log "Function 'New-MartiDefinition' parameters follow"
Write-Log "Parameter: SourceFolder Value: $SourceFolder "
Write-Log "Parameter: Filter Value: $Filter "
Write-Log "Parameter: Recurse Value: $Recurse "
Write-Log "Parameter: ExtendAttributes Value: $ExtendAttributes "
Write-Log "Parameter: ExcludeHash Value: $ExcludeHash "
Write-Log ""
$oMarti = New-MartiDefinition
$lresource = $oMarti.resources
$SourceFullName = (Get-Item -Path $SourceFolder).FullName
Get-ChildItem $SourceFolder -Filter $Filter -Recurse:$Recurse -Force| Where-Object {!$_.PSIsContainer} | ForEach-Object {
$oResource = New-MartiResource -SourcePath $_.FullName -UrlPath $remoteDirectory -LogPath $LogPath -ExtendAttributes:$ExtendAttributes -ExcludeHash:$ExcludeHash
if ($null -ne $UrlPath -and $UrlPath -ne "") {
$postfixName = $_.FullName.Replace($SourceFullName, "")
if ($postfixName[0] -eq "/" -or $postfixName[0] -eq "`\" ){
$postfixName = $postfixName.Substring(1, ($postfixName.Length-1))
}
if ($UrlPath[$UrlPath.Length-1] -eq "/" -or $UrlPath[$UrlPath.Length-1] -eq "`\") {
$oResource.url = $UrlPath.Replace("`\", "/") + $postfixName.Replace("`\", "/")
} else {
$oResource.url = $UrlPath.Replace("`\", "/") + "/" + $postfixName.Replace("`\", "/")
}
}
$lresource += $oResource
}
Write-Log "Captured $($lresource.Count) items"
$oMarti.resources = $lresource
Close-Log
return $oMarti
}
function New-DefaultCsvAttributes { function New-DefaultCsvAttributes {
@ -306,3 +478,4 @@ function Set-MartiResourceAttributes {
} }

147
source/powershell/New-Marti.ps1
View File

@ -1,147 +0,0 @@
$script:LogPathName = ""
$script:SoftwareVersion = "0.0.1"
$global:default_metaFile = "##marti##.mri"
function Get-LogName {
$date = Get-Date -f "yyyy-MM-dd"
if (($null -eq $script:LogPathName) -or ($script:LogPathName -eq ""))
{
return $null
}
if (!(Test-Path -Path $script:LogPathName)) {
$null = New-Item -Path $script:LogPathName -ItemType Directory
}
$logName = $(Get-SoftwareName) + "_$date.log"
return Join-Path -Path $script:LogPathName -ChildPath $logName
}
function Write-Log {
param(
[String] $LogEntry
)
$sFullPath = Get-LogName
$dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss"
if ($null -ne $sFullPath -and $sFullPath -ne "") {
if (!(Test-Path -Path $sFullPath)) {
Write-Host "Log path: $sFullPath"
$null = New-Item -Path $sFullPath -ItemType File
}
Add-Content -Path $sFullPath -Value "[$dateTime]. $LogEntry"
}
Write-Debug "[$dateTime]. $LogEntry"
}
function Open-Log {
$dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss"
Write-Log "***********************************************************************************"
Write-Log "* Start of processing: [$dateTime]"
Write-Log "***********************************************************************************"
}
function Close-Log {
$dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss"
Write-Log "***********************************************************************************"
Write-Log "* End of processing: [$dateTime]"
Write-Log "***********************************************************************************"
}
function Get-SoftwareName {
return [String] "MARTIREFERENCE"
}
function New-MartiDefinition
{
$oSoftware = [PSCustomObject]@{
extension = "software"
softwareName = "MartiReference"
author = "Meerkat@merebox.com"
version = "$script:SoftwareVersion"
}
$publisher = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name
[System.Collections.ArrayList]$lcustom = @()
$lcustom += $oSoftware
[System.Collections.ArrayList]$lresource = @()
$oMarti = [PSCustomObject]@{
title = ""
uid = (New-Guid).ToString()
resources = $lresource
description = ""
modified = Get-Date -f "yyyy-MM-ddTHH:mm:ss"
tags = @( "document", "marti")
publisher = $publisher
contactPoint = ""
accessLevel = "Confidential"
rights = "Restricted"
license = ""
state = "active"
describedBy = ""
landingPage = ""
theme =""
custom = $lCustom
}
return $oMarti
}
function Set-MartiAttribute
{
Param(
[System.Collections.ArrayList] $Attributes,
[String] $ACategory,
[String] $AName,
[String] $AFunction,
[String] $Comparison,
[String] $Value
)
$matched = $false
$Attributes | ForEach-Object {
if ($_.category -eq $ACategory -and $_.name -eq $AName -and $_.function -eq $AFunction) {
$matched = $true
$_.comparison = $comparison
$_.value = $value
}
}
if (!($matched)) {
$oAttribute = [PSCustomObject]@{
category = $Acategory
name = $AName
function = $Afunction
comparison = $comparison
value = $value
}
$Attributes += $oAttribute
}
return $Attributes
}

58
source/powershell/New-MartiChildItem.ps1
View File

@ -1,58 +0,0 @@
function New-MartiChildItem
{
Param(
[Parameter(Mandatory)][String] $SourceFolder,
[String] $Filter ="*",
[String] $UrlPath,
[switch] $Recurse,
[switch] $ExtendAttributes,
[switch] $ExcludeHash,
[String] $LogPath
)
$script:LogPathName = $LogPath
Write-Debug "Parameter: LogPath Value: $LogPath "
Open-Log
Write-Log "Function 'New-MartiDefinition' parameters follow"
Write-Log "Parameter: SourceFolder Value: $SourceFolder "
Write-Log "Parameter: Filter Value: $Filter "
Write-Log "Parameter: Recurse Value: $Recurse "
Write-Log "Parameter: ExtendAttributes Value: $ExtendAttributes "
Write-Log "Parameter: ExcludeHash Value: $ExcludeHash "
Write-Log ""
$oMarti = New-MartiDefinition
$lresource = $oMarti.resources
$SourceFullName = (Get-Item -Path $SourceFolder).FullName
Get-ChildItem $SourceFolder -Filter $Filter -Recurse:$Recurse -Force| Where-Object {!$_.PSIsContainer} | ForEach-Object {
$oResource = New-MartiResource -SourcePath $_.FullName -UrlPath $remoteDirectory -LogPath $LogPath -ExtendAttributes:$ExtendAttributes -ExcludeHash:$ExcludeHash
if ($null -ne $UrlPath -and $UrlPath -ne "") {
$postfixName = $_.FullName.Replace($SourceFullName, "")
if ($postfixName[0] -eq "/" -or $postfixName[0] -eq "`\" ){
$postfixName = $postfixName.Substring(1, ($postfixName.Length-1))
}
if ($UrlPath[$UrlPath.Length-1] -eq "/" -or $UrlPath[$UrlPath.Length-1] -eq "`\") {
$oResource.url = $UrlPath.Replace("`\", "/") + $postfixName.Replace("`\", "/")
} else {
$oResource.url = $UrlPath.Replace("`\", "/") + "/" + $postfixName.Replace("`\", "/")
}
}
$lresource += $oResource
}
Write-Log "Captured $($lresource.Count) items"
$oMarti.resources = $lresource
Close-Log
return $oMarti
}

112
source/powershell/New-MartiItem.ps1
View File

@ -1,112 +0,0 @@
function New-MartiResource {
Param(
[Parameter(Mandatory)][String] $SourcePath,
[String] $UrlPath = "",
[switch] $ExcludeHash,
[switch] $ExtendAttributes,
[String] $LogPath
)
$Global:MartiErrorId = ""
$script:LogPathName = $LogPath
Write-Debug "Parameter: LogPath Value: $LogPath "
Open-Log
Write-Log "Function 'New-MartiResource' parameters follow"
Write-Log "Parameter: UrlPath Value: $UrlPath "
Write-Log "Parameter: SourcePath Value: $SourcePath "
Write-Log "Parameter: ExcludeHash Value: $ExcludeHash "
Write-Log ""
if (Test-Path -Path $SourcePath -PathType Leaf) {
$item = Get-Item -Path $SourcePath -Force
Write-Log "Define file $($item.FullName) "
if ($ExcludeHash) {
$hash = $null
} else {
$hash = New-MartiHash -Algorithm "SHA256" -FilePath $item.FullName
}
$lattribute = Set-MartiResourceAttributes -Path $item.FullName -FileType $item.Extension.Substring(1) -ExtendedAttributes:$ExtendAttributes
$oResource = [PSCustomObject]@{
title = $item.Name.Replace($item.Extension, "")
uid = (New-Guid).ToString()
documentName = $item.Name
issuedDate = Get-Date -f "yyyy-MM-ddTHH:mm:ss"
modified = $item.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss")
state = "active"
author = ""
length = $item.Length
hash = $hash
description = ""
url = ""
version = $version
format = $item.Extension.Substring(1)
compression = $null
encryption = $null
attributes = $lattribute
}
if ($null -ne $UrlPath -and $UrlPath -ne "") {
if ($UrlPath[$UrlPath.Length-1] -eq "/" -or $UrlPath[$UrlPath.Length-1] -eq "\\") {
$oResource.url = $UrlPath.Replace("\\", "/") + $item.Name
} else {
$oResource.url = $UrlPath.Replace("\\", "/") + "/" + $item.Name
}
}
} else {
$Global:MartiErrorId = "MRI2001"
$message = "Document '$SourcePath' not found or is a folder"
Write-Log ($message + " " + $Global:MartiErrorId)
Close-Log
throw $message
}
Close-Log
return $oResource
}
function New-MartiHash{
Param(
[Parameter(Mandatory)][String] $Algorithm,
[String] $FilePath
[String] $Value = ""
)
if ($Value -eq "" -and $FilePath -ne "") {
$Value = (Get-FileHash -Path $FilePath -Algorithm $Algorithm).Hash
}
$oHash = [PSCustomObject]@{
algo = $Algorithm
value = $Value
}
return $oHash
}
function New-Encryption{
Param(
[Parameter(Mandatory)][String] $Algorithm,
[String] $Value
)
$oEncryption = [PSCustomObject]@{
algo = $Algorithm
value = $Value
}
return $oEncryption
}

9
source/python/client/martiLQ.py
View File

@ -14,7 +14,7 @@ class martiLQ:
gLogPathName = "" gLogPathName = ""
gSoftwareVersion = "0.0.1" gSoftwareVersion = "0.0.1"
gdefault_metaFile = "##marti##.mri" gdefault_metaFile = "##marti##.mti"
gMartiErrorId = "" gMartiErrorId = ""
gLogOpen = False gLogOpen = False
@ -126,7 +126,7 @@ class martiLQ:
"rights": "Restricted", "rights": "Restricted",
"license": "", "license": "",
"state": "active", "state": "active",
"batch": 1,
"describedBy": "", "describedBy": "",
"landingPage": "", "landingPage": "",
"theme": "", "theme": "",
@ -157,6 +157,7 @@ class martiLQ:
item = os.path.basename(SourcePath) item = os.path.basename(SourcePath)
self.WriteLog("Define file {}".format(SourcePath)) self.WriteLog("Define file {}".format(SourcePath))
HashAlgorithm = "SHA256"
try: try:
mtime = os.path.getmtime(SourcePath) mtime = os.path.getmtime(SourcePath)
@ -167,7 +168,7 @@ class martiLQ:
if ExcludeHash: if ExcludeHash:
hash = None hash = None
else: else:
hash = self.NewMartiHash(Algorithm="SHA256", FilePath=SourcePath, Value="") hash = self.NewMartiHash(Algorithm=HashAlgorithm, FilePath=SourcePath, Value="")
lattribute = self.SetMartiResourceAttributes(SourcePath, os.path.splitext(SourcePath)[1][1:], ExtendAttributes) lattribute = self.SetMartiResourceAttributes(SourcePath, os.path.splitext(SourcePath)[1][1:], ExtendAttributes)
@ -561,7 +562,7 @@ class martiLQ:
return errorCount return errorCount
def TestMartiDefinition(self, oMarti, LQSourcePath, LogPath =""): def TestMartiDefinition(self, LQSourcePath, oMarti = None, LogPath =""):
self.gMartiErrorId = "" self.gMartiErrorId = ""
self.gLogPathName = LogPath self.gLogPathName = LogPath

19
test/powershell/test_Marti.ps1
View File

@ -1,18 +1,17 @@
. .\source\powershell\New-Marti.ps1 . .\source\powershell\MartiLQ.ps1
. .\source\powershell\Get-Marti.ps1 . .\source\powershell\Compress-MartiLQ.ps1
. .\source\powershell\Compress-Marti.ps1
Write-Host "Test case #1" Write-Host "Test case #1"
$oMarti = New-MartiChildItem -SourceFolder ".\docs" -Recurse -UrlPath ".\docs" -Filter "*" -LogPath ".\test\powershell\results\Logs" $oMarti = New-MartiChildItem -SourceFolder ".\docs" -Recurse -UrlPath ".\docs" -Filter "*" -LogPath ".\test\powershell\results\Logs"
$oMarti.description = "Sample execution" $oMarti.description = "Sample execution"
$x = ConvertTo-Json -InputObject $oMarti $x = ConvertTo-Json -InputObject $oMarti
Set-Content -Path ".\test\powershell\results\marti_test01.mri.json" -Value $x Set-Content -Path ".\test\powershell\results\marti_test01.mti" -Value $x
Write-Host "Test case #2" Write-Host "Test case #2"
$ArchiveFile = ".\test\powershell\results\marti_test02.zip" $ArchiveFile = ".\test\powershell\results\marti_test02.zip"
Compress-Marti -SourceFolder ".\docs" -Filter "*" -LogPath ".\test\powershell\results\Logs" -ArchiveFile $ArchiveFile Compress-MartiLQ -SourceFolder ".\docs" -Filter "*" -LogPath ".\test\powershell\results\Logs" -ArchiveFile $ArchiveFile
Write-Host "Test case #3" Write-Host "Test case #3"
$y = Get-MartiItem -MartiDefintiion $oMarti -Title "ckan" -Format "txt" -LogPath ".\test\powershell\results\Logs" $y = Get-MartiItem -MartiDefintiion $oMarti -Title "ckan" -Format "txt" -LogPath ".\test\powershell\results\Logs"
@ -24,22 +23,22 @@ $oMarti = New-MartiResource -SourcePath ".\docs\ckan.md" -LogPath ".\test\powers
$oMarti.description = "Sample execution for ckan" $oMarti.description = "Sample execution for ckan"
$x = ConvertTo-Json -InputObject $oMarti $x = ConvertTo-Json -InputObject $oMarti
Set-Content -Path ".\test\powershell\results\marti_test02.mri.json" -Value $x Set-Content -Path ".\test\powershell\results\marti_test02json.mti" -Value $x
$x = ConvertTo-Csv -InputObject $oMarti $x = ConvertTo-Csv -InputObject $oMarti
Set-Content -Path ".\test\powershell\results\marti_test02.mri.csv" -Value $x Set-Content -Path ".\test\powershell\results\marti_test02csv.mti." -Value $x
$x = ConvertTo-Xml -As String -InputObject $oMarti -Depth 6 $x = ConvertTo-Xml -As String -InputObject $oMarti -Depth 6
Set-Content -Path ".\test\powershell\results\marti_test02.mri.xml" -Value $x Set-Content -Path ".\test\powershell\results\marti_test02xml.mti" -Value $x
$x = ConvertTo-Html -InputObject $oMarti $x = ConvertTo-Html -InputObject $oMarti
Set-Content -Path ".\test\powershell\results\marti_test02.mri.html" -Value $x Set-Content -Path ".\test\powershell\results\marti_test02html.mti" -Value $x
Write-Host "Test case #5" Write-Host "Test case #5"
$oMarti = New-MartiResource -SourcePath ".\docs\eror" -LogPath ".\test\powershell\results\Logs" $oMarti = New-MartiResource -SourcePath ".\docs\eror" -LogPath ".\test\powershell\results\Logs"
$oMarti.description = "Sample execution with error" $oMarti.description = "Sample execution with error"
$x = ConvertTo-Json -InputObject $oMarti $x = ConvertTo-Json -InputObject $oMarti
Set-Content -Path ".\test\powershell\results\marti_test03.mri.json" -Value $x Set-Content -Path ".\test\powershell\results\marti_test03.mti" -Value $x

6
test/powershell/test_MartiCkan.ps1
View File

@ -7,7 +7,7 @@
$ckan = Get-Content -Path ".\docs\samples\asic_ckan_api.json" -Raw $ckan = Get-Content -Path ".\docs\samples\asic_ckan_api.json" -Raw
$oMarti = ConvertFrom-Ckan -InputObject $ckan $oMarti = ConvertFrom-Ckan -InputObject $ckan
$x = ConvertTo-Json -InputObject $oMarti $x = ConvertTo-Json -InputObject $oMarti
Set-Content -Path ".\test\powershell\results\marti_test05.mri.json" -Value $x Set-Content -Path ".\test\powershell\results\marti_test05.mti" -Value $x
$covid_1 = Invoke-WebRequest "https://data.nsw.gov.au/data/api/3/action/package_show?id=793ac07d-a5f4-4851-835c-3f7158c19d15" $covid_1 = Invoke-WebRequest "https://data.nsw.gov.au/data/api/3/action/package_show?id=793ac07d-a5f4-4851-835c-3f7158c19d15"
@ -18,7 +18,7 @@ $oMarti.tags += "gov"
$oMarti.tags += "nsw" $oMarti.tags += "nsw"
$oMarti.publisher = "NSW government (Australia)" $oMarti.publisher = "NSW government (Australia)"
$x = ConvertTo-Json -InputObject $oMarti $x = ConvertTo-Json -InputObject $oMarti
Set-Content -Path ".\test\powershell\results\marti_test06.mri.json" -Value $x Set-Content -Path ".\test\powershell\results\marti_test06.mti" -Value $x
# cases # cases
@ -32,5 +32,5 @@ $oMarti.tags += "gov"
$oMarti.tags += "nsw" $oMarti.tags += "nsw"
$oMarti.publisher = "NSW government (Australia)" $oMarti.publisher = "NSW government (Australia)"
$x = ConvertTo-Json -InputObject $oMarti $x = ConvertTo-Json -InputObject $oMarti
Set-Content -Path ".\test\powershell\results\marti_test07.mri.json" -Value $x Set-Content -Path ".\test\powershell\results\marti_test07.mti" -Value $x

6
tools.md
View File

@ -2,9 +2,9 @@
A number of tools are povided that can be incorporated into your A number of tools are povided that can be incorporated into your
projects that are want to use the metadata transfer reconciliation format projects that are want to use the metadata transfer reconciliation format
(Marti document). (martiLQ document).
You can combine these in different ways such as: You can combine these in different ways such as:
* Use Python extract program to generate the Marti document * Use Python extract program to generate the martiLQ document
* Use a Java program to reconcile the received data document with the Marti file * Use a Java program to reconcile the received data document with the martiLQ file