From d515a6c4a64807dd73e98ef0eaf14b1732fe04cd Mon Sep 17 00:00:00 2001 From: meerkat Date: Thu, 14 Oct 2021 21:50:55 +1100 Subject: [PATCH] Refactored to martiLQ --- README.md | 11 +- docs/README.md | 16 +- docs/comparison.md | 15 +- docs/distribution.md | 8 +- docs/quality.md | 2 +- .../powershell/Invoke-SampleGenerateBsb.ps1 | 41 ++--- .../Invoke-SampleGenerateBsbSecure.ps1 | 20 +- docs/samples/python/SampleGenerateBsb.py | 10 +- docs/what.md | 6 +- docs/when.md | 6 +- docs/who.md | 4 +- docs/why.md | 6 +- Marti.md => martiLQ.md | 6 +- source/README.md | 2 +- source/powershell/Compare-MartiResource.ps1 | 2 +- ...ompress-Marti.ps1 => Compress-MartiLQ.ps1} | 66 +------ .../powershell/{Get-Marti.ps1 => MartiLQ.ps1} | 87 ++++++++- ...ResourceAttributes.ps1 => MartiLQItem.ps1} | 173 ++++++++++++++++++ source/powershell/New-Marti.ps1 | 147 --------------- source/powershell/New-MartiChildItem.ps1 | 58 ------ source/powershell/New-MartiItem.ps1 | 112 ------------ source/python/client/martiLQ.py | 9 +- test/powershell/test_Marti.ps1 | 19 +- test/powershell/test_MartiCkan.ps1 | 6 +- tools.md | 6 +- 25 files changed, 353 insertions(+), 485 deletions(-) rename Marti.md => martiLQ.md (62%) rename source/powershell/{Compress-Marti.ps1 => Compress-MartiLQ.ps1} (52%) rename source/powershell/{Get-Marti.ps1 => MartiLQ.ps1} (67%) rename source/powershell/{Get-MartiResourceAttributes.ps1 => MartiLQItem.ps1} (62%) delete mode 100644 source/powershell/New-Marti.ps1 delete mode 100644 source/powershell/New-MartiChildItem.ps1 delete mode 100644 source/powershell/New-MartiItem.ps1 diff --git a/README.md b/README.md index 681472e..95c8ca3 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Marti -**marti** stands for metadata reconcilation for transfer information. +**martiLQ** stands for metadata reconcilation for transfer information, load quality. The objective is to provide transfer information for high volume data such as in files. The files can be transferred via HTTPS, SFTP, message queue, @@ -15,9 +15,9 @@ it does not understand the data format. **Note**: The terms file and document are intended to be interchangeable through out this documentation. -**marti** is intended to provide minimum basic information on the transfer with +**martiLQ** is intended to provide minimum basic information on the transfer with ability to include optional information. The metadata reconcilation -transfer document being decscribed here wil be referred to as the [marti document](Marti.md) +transfer document being decscribed here wil be referred to as the [martiLQ document](MartiLQ.md) throughout this documentation. The transfer information is supplied as a separate document which could be another file @@ -39,8 +39,8 @@ information ## Transfer information -The information in the **marti** document is summarised below. For more detailed -information see [marti definition](marti.md) +The information in the **martiLQ** document is summarised below. For more detailed +information see [martiLQ definition](martiLQ.md) ### Mandatory information @@ -61,6 +61,7 @@ The optional information is: * Contact point * Acces level * Rights +* Batch * License * Spatial (*) * Temporal (*) diff --git a/docs/README.md b/docs/README.md index e7b1620..bebfe9e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,8 +1,8 @@ # Documentation -**marti** stands for metadata reconcilation for transfer information. +**martiLQ** stands for metadata reconcilation for transfer information. -Before starting with **marti** it is advisable to understand if it is right for +Before starting with **martiLQ** it is advisable to understand if it is right for your organisation's needs. Information is available in a number of short documents. @@ -13,12 +13,12 @@ can adjust if they resonate with your circumstances, ## Index -1. [What is marti](what.md) -2. [Who would use marti](who.md) -3. [Why you would use marti](why.md) -4. [When would you start using marti](when.md) +1. [What is martiLQ](what.md) +2. [Who would use martiLQ](who.md) +3. [Why you would use martiLQ](why.md) +4. [When would you start using martiLQ](when.md) 5. [Load quality metrics support](quality.md) -6. [Comparison of marti definition](comparison.md) +6. [Comparison of martiLQ definition](comparison.md) 7. [References](references.md) -[!INCLUDE [marti High Level Definition](../marti.md)] +[!INCLUDE [martiLQ High Level Definition](../martiLQ.md)] diff --git a/docs/comparison.md b/docs/comparison.md index f90489c..a0d7cef 100644 --- a/docs/comparison.md +++ b/docs/comparison.md @@ -1,4 +1,4 @@ -# Comparison of marti document definition +# Comparison of martiLQ document definition The use of metadata definitions is not unique and examples exist in many different situations. Some are standard and open @@ -8,20 +8,21 @@ Some open standards are EXIF data for pictures, SQL DDL defintions for databases, the XMP definition and web header responses before the web content. -The **marti** document definition is intended to cover the situation +The **martiLQ** document definition is intended to cover the situation where data files are being transferred and reconciliation is required. -The **marti** document definition is modelled on the [CKAN API metadata](https://docs.ckan.org/en/2.9/api/index.html) +The **martiLQ** document definition is modelled on +the [CKAN API metadata](https://docs.ckan.org/en/2.9/api/index.html) which has been adapted to included additional elements relevant to when you are exchanging data files. This includes the reconciliation elements such as number of records and file hash. As the definition is based on the CKAN API, there are tools to import -a CKAN source into a **marti** document definition and then process the data +a CKAN source into a **martiLQ** document definition and then process the data through the pipeline as you would for any other data file that had a -**marti** document definition. +**martiLQ** document definition. -## Benefit of CKAN and marti +## Benefit of CKAN and martiLQ The CKAN is excellent at defining the data source details but it lacks information for load quality. If you have CKAN deployed in your organisation and wish @@ -30,7 +31,7 @@ CKAN and marti. Samples exist on CKAN integration. -## Magda and marti +## Magda and martiLQ Another source of data is [Magda](https://magda.io/) which has API metadata definitions. Magda is more about data fedaration and as such provides diff --git a/docs/distribution.md b/docs/distribution.md index eec2284..d588bde 100644 --- a/docs/distribution.md +++ b/docs/distribution.md @@ -8,7 +8,7 @@ such as end of day. Some files may expand to multiple files if they are compressed with a utility such as WinZIP or 7ZIP. In the situation where a ZIP file expands to multiple documents, then the expectation is -that the ZIP file contains a **marti** document describing its contents. +that the ZIP file contains a **martiLQ** document describing its contents. The elements in the distribution section are: @@ -41,15 +41,15 @@ The following are optional in the distribution section. ## Compression Files can be compressed using a utility. A single compressed file can contain -multiple files. The **marti** definition document applies to the compressed file +multiple files. The **martiLQ** definition document applies to the compressed file and not to the contents, which could be multiple files. -In the case of a compressed files, there should be a **marti** definition document in the +In the case of a compressed files, there should be a **martiLQ** definition document in the compressed file. Compression of files always occur before encryption. -### Marti definition for Compressed File +### martiLQ definition for Compressed File For a compressed file that is not encrypted, the distribution definition will be: diff --git a/docs/quality.md b/docs/quality.md index 2a50d5c..be75f4e 100644 --- a/docs/quality.md +++ b/docs/quality.md @@ -1,6 +1,6 @@ # Quality definition -The **marti** definition allows for the inclusion of a load quality +The **martiLQ** definition allows for the inclusion of a load quality definition. This load quality definition is intended to be able to be applied universally with common tools. As such not all needs are covered. diff --git a/docs/samples/powershell/Invoke-SampleGenerateBsb.ps1 b/docs/samples/powershell/Invoke-SampleGenerateBsb.ps1 index 2535a38..21f9a4a 100644 --- a/docs/samples/powershell/Invoke-SampleGenerateBsb.ps1 +++ b/docs/samples/powershell/Invoke-SampleGenerateBsb.ps1 @@ -1,10 +1,7 @@ -. ..\..\..\source\powershell\New-Marti.ps1 -. ..\..\..\source\powershell\New-MartiChildItem.ps1 -. ..\..\..\source\powershell\New-MartiResource.ps1 -. ..\..\..\source\powershell\Get-Marti.ps1 -. ..\..\..\source\powershell\Compress-Marti.ps1 -. ..\..\..\source\powershell\Get-MartiResourceAttributes.ps1 +. ..\..\..\source\powershell\MartiLQ.ps1 +. ..\..\..\source\powershell\MartiLQItem.ps1 +. ..\..\..\source\powershell\Compress-MartiLQ.ps1 function PullFtpFile { @@ -89,7 +86,7 @@ $fileList = ListFtpDirectory -Username "anonymous" -Password "anon@merebox.com" Write-Host "File list size: $($fileList.count)" -ForegroundColor Gray -Write-Host "Now iterate through the remote files and build remote marti list " -ForeGroundColor Green +Write-Host "Now iterate through the remote files and build remote martiLQ list " -ForeGroundColor Green $oMarti = New-MartiDefinition $oMarti.title = "Remote_BSB_data" @@ -101,7 +98,7 @@ $oMarti.theme = "payment" ForEach ($item in $fileList) { if ($item -ne "" -and $item.startswith("BSBDirectory")) { PullFtpFile -Username "anonymous" -Password "anon@merebox.com" -RemoteFile ($remoteDirectory + $item) -OutputPath (Join-Path -Path $localDirectory -ChildPath $item) - Write-Host "Add BSB $item file to Remote marti metadata sample " -ForeGroundColor Yellow + Write-Host "Add BSB $item file to Remote martiLQ metadata sample " -ForeGroundColor Yellow $oResource = New-MartiResource -SourcePath (Join-Path -Path $localDirectory -ChildPath $item) -UrlPath $remoteDirectory -LogPath ".\test\Logs" -ExtendAttributes if ($item.endswith(".txt")) { Set-AttributeValueNumber -Attributes $oResource.attributes -Key "header" -Category "dataset" -Function "count" -Value 1 @@ -116,13 +113,12 @@ ForEach ($item in $fileList) { } } -$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBRemote.mri.json" +$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBRemote.mti" $oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson -Write-Host "Remote marti definition file is $fileJson " -ForeGroundColor Green -Write-Host "Note that file has been given JSON extension for sample purposes" -ForeGroundColor Gray +Write-Host "Remote martiLQ definition file is $fileJson " -ForeGroundColor Green -Write-Host "Now iterate through the local files and build marti ZIP " -ForeGroundColor Green +Write-Host "Now iterate through the local files and build martiLQ ZIP " -ForeGroundColor Green $oMarti = New-MartiDefinition $oMarti.title = "Zip_BSB_data" @@ -139,7 +135,7 @@ if (Test-Path -Path $zipFile) { foreach($file in Get-ChildItem $localDirectory) { if ($file.Name.startswith("BSBDirectory") -and !($file.Name.EndsWith(".zip")) -and !($file.Name.EndsWith(".7z")) ) { - Write-Host "Add BSB file $file to ZIP marti metadata sample " -ForeGroundColor Yellow + Write-Host "Add BSB file $file to ZIP martiLQ metadata sample " -ForeGroundColor Yellow Compress-Archive -Path $file.FullName -DestinationPath $zipFile -Update $oResource = New-MartiResource -SourcePath $file.FullName -UrlPath $localDirectory -LogPath ".\test\Logs" -ExtendAttributes if ($file.Extension -eq ".txt") { @@ -159,10 +155,9 @@ $oResource = New-MartiResource -SourcePath $zipFile -UrlPath $localDirectory -Lo Set-AttributeValueString -Attributes $oResource.attributes -Key "compression" -Category "format" -Function "algorithm" -Value "WINZIP" $oMarti.resources += $oResource -$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBZip.mri.json" +$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBZip.mti" $oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson -Write-Host "ZIP marti definition file is $fileJson " -ForeGroundColor Green -Write-Host "Note that file has been given JSON extension for sample purposes" -ForeGroundColor Gray +Write-Host "ZIP martiLQ definition file is $fileJson " -ForeGroundColor Green @@ -175,10 +170,9 @@ $oMarti.contactPoint = "meerkat@merebox.com" $oMarti.landingPage = "https://github.com/meerkat-manor/marti/blob/draft_specifications/docs/samples/powershell/Invoke-BSBSample.ps1" $oMarti.theme = "payment" -$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBLocal.mri.json" +$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBLocal.mti" $oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson -Write-Host "Local marti definition file is $fileJson " -ForeGroundColor Green -Write-Host "Note that file has been given JSON extension for sample purposes" -ForeGroundColor Gray +Write-Host "Local martiLQ definition file is $fileJson " -ForeGroundColor Green Write-Host "Now create an encrypted 7ZIP file with asymmetric password protection" -ForeGroundColor Green @@ -198,7 +192,7 @@ if (Test-Path -Path $zipFile) { foreach($file in Get-ChildItem $localDirectory) { if ($file.Name.startswith("BSBDirectory") -and !($file.Name.EndsWith(".zip")) -and !($file.Name.EndsWith(".7z")) ) { - Write-Host "Add BSB file $file to 7ZIP marti metadata sample " -ForeGroundColor Yellow + Write-Host "Add BSB file $file to 7ZIP martiLQ metadata sample " -ForeGroundColor Yellow if (Test-Path -Path $zipFile) { Compress-7Zip -Path $file.FullName -ArchiveFileName $zipFile -Format SevenZip -Append } else { @@ -220,7 +214,7 @@ foreach($file in Get-ChildItem $localDirectory) } $noticeFile = Join-Path -Path $localDirectory -ChildPath "README.txt" -Set-Content -Path $noticeFile -Value "Generated by marti Samples" +Set-Content -Path $noticeFile -Value "Generated by martiLQ Samples" $oResource = New-MartiResource -SourcePath $noticeFile -UrlPath $localDirectory -LogPath ".\test\Logs" $oMarti.resources += $oResource @@ -233,10 +227,9 @@ $oResource.encryption = New-Encryption -Algorithm "Passphrase" -Value $secret Set-AttributeValueString -Attributes $oResource.attributes -Key "compression" -Category "format" -Function "algorithm" -Value "7ZIP" $oMarti.resources += $oResource -$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBSecure.mri.json" +$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBSecure.mti" $oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson -Write-Host "Secure 7ZIP marti definition file is $fileJson " -ForeGroundColor Green -Write-Host "Note that file has been given JSON extension for sample purposes" -ForeGroundColor Gray +Write-Host "Secure 7ZIP martiLQ definition file is $fileJson " -ForeGroundColor Green Write-Host "Sample execution completed" -ForeGroundColor Green diff --git a/docs/samples/powershell/Invoke-SampleGenerateBsbSecure.ps1 b/docs/samples/powershell/Invoke-SampleGenerateBsbSecure.ps1 index 88f91a8..7bb4e98 100644 --- a/docs/samples/powershell/Invoke-SampleGenerateBsbSecure.ps1 +++ b/docs/samples/powershell/Invoke-SampleGenerateBsbSecure.ps1 @@ -1,10 +1,7 @@ -. ..\..\..\source\powershell\New-Marti.ps1 -. ..\..\..\source\powershell\New-MartiChildItem.ps1 -. ..\..\..\source\powershell\New-MartiResource.ps1 -. ..\..\..\source\powershell\Get-Marti.ps1 -. ..\..\..\source\powershell\Compress-Marti.ps1 -. ..\..\..\source\powershell\Get-MartiResourceAttributes.ps1 +. ..\..\..\source\powershell\MartiLQ.ps1 +. ..\..\..\source\powershell\MartiLQItem.ps1 +. ..\..\..\source\powershell\Compress-MartiLQ.ps1 function PullFtpFile { @@ -127,7 +124,7 @@ if (Test-Path -Path $zipFile) { foreach($file in Get-ChildItem $localDirectory) { if ($file.Name.startswith("BSBDirectory") -and $file.Name.EndsWith(".csv") ) { - Write-Host "Add BSB file $file to 7ZIP marti metadata sample " -ForeGroundColor Yellow + Write-Host "Add BSB file $file to 7ZIP martiLQ metadata sample " -ForeGroundColor Yellow if (Test-Path -Path $zipFile) { Compress-7Zip -Path $file.FullName -ArchiveFileName $zipFile -Format SevenZip -Append } else { @@ -149,7 +146,7 @@ foreach($file in Get-ChildItem $localDirectory) } $noticeFile = Join-Path -Path $localDirectory -ChildPath "README.txt" -Set-Content -Path $noticeFile -Value "Generated by marti Samples" +Set-Content -Path $noticeFile -Value "Generated by martiLQ Samples" $oResource = New-MartiResource -SourcePath $noticeFile -UrlPath $localDirectory -LogPath ".\test\Logs" $oMarti.resources += $oResource @@ -167,9 +164,8 @@ Write-Debug "Secret: $secret" Set-AttributeValueString -Attributes $oResource.attributes -Key "compression" -Category "format" -Function "algorithm" -Value "7ZIP" $oMarti.resources += $oResource -$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSB7ZipPKI.mri.json" -$oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson -Write-Host "7ZIP marti definition file is $fileJson " -ForeGroundColor Green -Write-Host "Note that file has been given JSON extension for sample purposes" -ForeGroundColor Gray +$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSB7ZipPKI.mti" +$oMarti | ConvertTo-Json -depth 50 | Out-File $fileJson +Write-Host "7ZIP martiLQ definition file is $fileJson " -ForeGroundColor Green Write-Host "Sample secure execution completed" -ForeGroundColor Green diff --git a/docs/samples/python/SampleGenerateBsb.py b/docs/samples/python/SampleGenerateBsb.py index bfeddce..27df27e 100644 --- a/docs/samples/python/SampleGenerateBsb.py +++ b/docs/samples/python/SampleGenerateBsb.py @@ -59,9 +59,9 @@ for file_name in files: if file_name.endswith(".csv") | file_name.endswith(".txt"): file_remote = remote_dir + file_name file_local = "./test/" + file_name -# ftpPull(remote_host, file_remote, file_local) + ftpPull(remote_host, file_remote, file_local) -print("Creating marti definition") +print("Creating martiLQ definition") mlq = martiLQ() oMarti = mlq.NewMartiDefinition() @@ -72,15 +72,15 @@ for file_name in files: oMarti["resources"].append(oResource) mlq.CloseLog() -print("Save marti definition") +print("Save martiLQ definition") jd = json.dumps(oMarti, indent=5) -jsonFile = open("./test/BSBDirectoryPlain.mri.json", "w") +jsonFile = open("./test/BSBDirectoryPlain.mti", "w") jsonFile.write(jd) jsonFile.close() print("Sample completed: SampleGenerateBsb.py") -lqresults, testError = mlq.TestMartiDefinition(oMarti, "./test/BSBDirectoryPlain.mri.json") +lqresults, testError = mlq.TestMartiDefinition(oMarti, "./test/BSBDirectoryPlain.mti") testfile = open("./test/LoadQualityTest01.csv", "w+", newline ="") with testfile: diff --git a/docs/what.md b/docs/what.md index 9c1c7d6..1b175fc 100644 --- a/docs/what.md +++ b/docs/what.md @@ -1,13 +1,13 @@ # What is marti -The foundation pillar for the **marti** framework is the [marti document](../marti.md) +The foundation pillar for the **martiLQ** framework is the [martiLQ document](../martiLQ.md) that defines the reconciliation and other metadata of the document / file being transferred. A definition, while fundamental, benefits from having tools that can create, read and -interpret the definition. The **marti** framework is about providing those tools +interpret the definition. The **martiLQ** framework is about providing those tools and a ecosystem that can be added to. -The majority of the effort in creating **marti** is in the tools. There are tools +The majority of the effort in creating **martiLQ** is in the tools. There are tools for various programming languages and situations. As many programming languages generate portable programs that can execute on multiple operating systems, the likelihood is that a tools exists for you. diff --git a/docs/when.md b/docs/when.md index f804780..cdac8ea 100644 --- a/docs/when.md +++ b/docs/when.md @@ -1,6 +1,6 @@ -# When would you use marti +# When would you use martiLQ -You are likely to start using the **marti** framework when: +You are likely to start using the **martiLQ** framework when: 1. you have no existing standard or framework or; 2. your existing standards no longer meets you needs or; 3. you are starting document exchange with another business or division that uses the framework or; @@ -12,7 +12,7 @@ of the framework are unlikely to weigh in your framework ## Read the material -Please read the material before jumping in and make sure the **marti** framework will +Please read the material before jumping in and make sure the **martiLQ** framework will provide you with the expected benefits. Implementing the framework and then changing your direction will more than likely add to your support costs. diff --git a/docs/who.md b/docs/who.md index 60a90ab..bc191d4 100644 --- a/docs/who.md +++ b/docs/who.md @@ -1,6 +1,6 @@ # Who is likely to use marti -You are likely to find the **marti** framework relevant if you: +You are likely to find the **martiLQ** framework relevant if you: 1. Have many document exchanges 2. Need to verify or reconcile the documents @@ -24,7 +24,7 @@ Simple framework such as the above have limitations, such as: ## Framework Sidecar files -The **marti** framework addresses the issues and limitations by using sidecar +The **martiLQ** framework addresses the issues and limitations by using sidecar or shadow files. The [concept of sidecar files](https://en.wikipedia.org/wiki/Sidecar_file) is not new and are commonly found associated to media file processing. diff --git a/docs/why.md b/docs/why.md index 081415a..050554d 100644 --- a/docs/why.md +++ b/docs/why.md @@ -1,6 +1,6 @@ # Why use marti -**marti** is a framework for providing a degree of auditability and reconciliation of +**martiLQ** is a framework for providing a degree of auditability and reconciliation of documents transferred between systems in an organisation and externally. It does not intend define the format or content of the document. It defines controls that can be used to: @@ -11,7 +11,7 @@ define the format or content of the document. It defines controls that can be u 5. describe metadata including format, extract time 6. link to further information -You would use **marti** if any of the controls are a requirement for you. +You would use **martiLQ** if any of the controls are a requirement for you. ## Documents @@ -19,7 +19,7 @@ Documents in this context are digital storage objects such as operating system f cloud storage objects or blobs. The document content has structure and contains multiple records. -The **marti** framework is not intended to be used for single record transfers such as +The **martiLQ** framework is not intended to be used for single record transfers such as in single web transactions. It is for providing controls when moving large amounts of data as one event. This data are commonly referred to as batch extracts and performed at scheduled times such as end of day. diff --git a/Marti.md b/martiLQ.md similarity index 62% rename from Marti.md rename to martiLQ.md index e55b430..c1e6230 100644 --- a/Marti.md +++ b/martiLQ.md @@ -1,9 +1,9 @@ -# Marti document +# MartiLQ document The metadata reconciliation transfer information is referred -to as the **marti** document throughout this documentation. +to as the **martiLQ** document throughout this documentation. -The **marti** document can be part of a message or a document +The **martiLQ** document can be part of a message or a document in its own right. If the document is a file then the recommended name for the document is the same name as the data file, including extension, with the added extension of ``.mti`` diff --git a/source/README.md b/source/README.md index 50645e1..73a8c35 100644 --- a/source/README.md +++ b/source/README.md @@ -1,5 +1,5 @@ # Contents -This is the parent directory for **marti** tools written in various languages. +This is the parent directory for **martiLQ** tools written in various languages. Please browse the folder for the programming language of interest. diff --git a/source/powershell/Compare-MartiResource.ps1 b/source/powershell/Compare-MartiResource.ps1 index 100ec13..82a9d50 100644 --- a/source/powershell/Compare-MartiResource.ps1 +++ b/source/powershell/Compare-MartiResource.ps1 @@ -16,7 +16,7 @@ function Compare-MartiResource { if ($null -eq $Resource) { $Global:MartiErrorId = "MRI2201" - $message = "No Marti resource definition supplied" + $message = "No resource definition supplied" Write-Log ($message + " " + $Global:MartiErrorId) Close-Log throw $message diff --git a/source/powershell/Compress-Marti.ps1 b/source/powershell/Compress-MartiLQ.ps1 similarity index 52% rename from source/powershell/Compress-Marti.ps1 rename to source/powershell/Compress-MartiLQ.ps1 index 7cb7bda..f7bb254 100644 --- a/source/powershell/Compress-Marti.ps1 +++ b/source/powershell/Compress-MartiLQ.ps1 @@ -1,68 +1,6 @@ -$script:LogPathName = "" -$script:SoftwareVersion = "0.0.1" - -$global:default_metaFile = "##marti##.mri" - -function Get-LogName { - - $date = Get-Date -f "yyyy-MM-dd" - - if (($null -eq $script:LogPathName) -or ($script:LogPathName -eq "")) - { - return $null - } - - if (!(Test-Path -Path $script:LogPathName)) { - $null = New-Item -Path $script:LogPathName -ItemType Directory - } - - $logName = $(Get-SoftwareName) + "_$date.log" - - return Join-Path -Path $script:LogPathName -ChildPath $logName -} - -function Write-Log { - param( - [String] $LogEntry - ) - - $sFullPath = Get-LogName - - $dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss" - if ($null -ne $sFullPath -and $sFullPath -ne "") { - - if (!(Test-Path -Path $sFullPath)) { - Write-Host "Log path: $sFullPath" - $null = New-Item -Path $sFullPath -ItemType File - } - Add-Content -Path $sFullPath -Value "[$dateTime]. $LogEntry" - } - Write-Debug "[$dateTime]. $LogEntry" - -} - -function Open-Log { - $dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss" - Write-Log "***********************************************************************************" - Write-Log "* Start of processing: [$dateTime]" - Write-Log "***********************************************************************************" -} - -function Close-Log { - $dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss" - Write-Log "***********************************************************************************" - Write-Log "* End of processing: [$dateTime]" - Write-Log "***********************************************************************************" -} - -function Get-SoftwareName { - return [String] "MARTIREFERENCE" -} - - -function Compress-Marti +function Compress-MartiLQ { Param( [Parameter(Mandatory)][String] $SourceFolder, @@ -76,7 +14,7 @@ Param( Write-Debug "Parameter: LogPath Value: $LogPath " Open-Log - Write-Log "Function 'Compress-Marti' parameters follow" + Write-Log "Function 'Compress-MartiLQ' parameters follow" Write-Log "Parameter: SourceFolder Value: $SourceFolder " Write-Log "Parameter: ArchiveFile Value: $ArchiveFile " Write-Log "Parameter: Filter Value: $Filter " diff --git a/source/powershell/Get-Marti.ps1 b/source/powershell/MartiLQ.ps1 similarity index 67% rename from source/powershell/Get-Marti.ps1 rename to source/powershell/MartiLQ.ps1 index 4534209..334a22c 100644 --- a/source/powershell/Get-Marti.ps1 +++ b/source/powershell/MartiLQ.ps1 @@ -2,7 +2,7 @@ $script:LogPathName = "" $script:SoftwareVersion = "0.0.1" -$global:default_metaFile = "##marti##.mri" +$global:default_metaFile = "##marti##.mti" function Get-LogName { @@ -63,6 +63,89 @@ function Get-SoftwareName { +function New-MartiDefinition +{ + + $oSoftware = [PSCustomObject]@{ + extension = "software" + softwareName = "MartiReference" + author = "Meerkat@merebox.com" + version = "$script:SoftwareVersion" + } + + $publisher = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name + + [System.Collections.ArrayList]$lcustom = @() + $lcustom += $oSoftware + + [System.Collections.ArrayList]$lresource = @() + + $oMarti = [PSCustomObject]@{ + title = "" + uid = (New-Guid).ToString() + resources = $lresource + + description = "" + modified = Get-Date -f "yyyy-MM-ddTHH:mm:ss" + tags = @( "document", "marti") + publisher = $publisher + contactPoint = "" + accessLevel = "Confidential" + rights = "Restricted" + license = "" + state = "active" + batch = 1 + describedBy = "" + landingPage = "" + theme ="" + + custom = $lCustom + } + + return $oMarti +} + + +function Set-MartiAttribute +{ +Param( + [System.Collections.ArrayList] $Attributes, + [String] $ACategory, + [String] $AName, + [String] $AFunction, + [String] $Comparison, + [String] $Value +) + + $matched = $false + + $Attributes | ForEach-Object { + + if ($_.category -eq $ACategory -and $_.name -eq $AName -and $_.function -eq $AFunction) { + $matched = $true + $_.comparison = $comparison + $_.value = $value + } + + } + + if (!($matched)) { + + $oAttribute = [PSCustomObject]@{ + category = $Acategory + name = $AName + function = $Afunction + comparison = $comparison + value = $value + } + + $Attributes += $oAttribute + } + + return $Attributes +} + + function Get-MartChildResource { @@ -87,7 +170,7 @@ function Get-MartChildResource if ($null -eq $Marti) { $Global:MartiErrorId = "MRI2101" - $message = "No Marti definition supplied" + $message = "No definition supplied" Write-Log ($message + " " + $Global:MartiErrorId) Close-Log throw $message diff --git a/source/powershell/Get-MartiResourceAttributes.ps1 b/source/powershell/MartiLQItem.ps1 similarity index 62% rename from source/powershell/Get-MartiResourceAttributes.ps1 rename to source/powershell/MartiLQItem.ps1 index cfa6e7f..b8f9b01 100644 --- a/source/powershell/Get-MartiResourceAttributes.ps1 +++ b/source/powershell/MartiLQItem.ps1 @@ -1,4 +1,176 @@ +function New-MartiResource { +Param( + [Parameter(Mandatory)][String] $SourcePath, + [String] $UrlPath = "", + [switch] $ExcludeHash, + [switch] $ExtendAttributes, + [String] $LogPath + +) + + $Global:MartiErrorId = "" + $script:LogPathName = $LogPath + + Write-Debug "Parameter: LogPath Value: $LogPath " + Open-Log + Write-Log "Function 'New-MartiResource' parameters follow" + Write-Log "Parameter: UrlPath Value: $UrlPath " + Write-Log "Parameter: SourcePath Value: $SourcePath " + Write-Log "Parameter: ExcludeHash Value: $ExcludeHash " + Write-Log "" + + + if (Test-Path -Path $SourcePath -PathType Leaf) { + + $item = Get-Item -Path $SourcePath -Force + + Write-Log "Define file $($item.FullName) " + + if ($ExcludeHash) { + $hash = $null + } else { + $hash = New-MartiHash -Algorithm "SHA256" -FilePath $item.FullName + } + + $lattribute = Set-MartiResourceAttributes -Path $item.FullName -FileType $item.Extension.Substring(1) -ExtendedAttributes:$ExtendAttributes + + $oResource = [PSCustomObject]@{ + title = $item.Name.Replace($item.Extension, "") + uid = (New-Guid).ToString() + documentName = $item.Name + issuedDate = Get-Date -f "yyyy-MM-ddTHH:mm:ss" + modified = $item.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss") + state = "active" + author = "" + length = $item.Length + hash = $hash + + description = "" + url = "" + version = $version + format = $item.Extension.Substring(1) + compression = $null + encryption = $null + + attributes = $lattribute + } + + if ($null -ne $UrlPath -and $UrlPath -ne "") { + if ($UrlPath[$UrlPath.Length-1] -eq "/" -or $UrlPath[$UrlPath.Length-1] -eq "\\") { + $oResource.url = $UrlPath.Replace("\\", "/") + $item.Name + } else { + $oResource.url = $UrlPath.Replace("\\", "/") + "/" + $item.Name + } + } + + } else { + $Global:MartiErrorId = "MRI2001" + $message = "Document '$SourcePath' not found or is a folder" + Write-Log ($message + " " + $Global:MartiErrorId) + Close-Log + throw $message + } + Close-Log + + return $oResource + +} + +function New-MartiHash{ + Param( + [Parameter(Mandatory)][String] $Algorithm, + [String] $FilePath, + [String] $Value = "" + ) + + if ($Value -eq "" -and $FilePath -ne "") { + $Value = (Get-FileHash -Path $FilePath -Algorithm $Algorithm).Hash + } + + $oHash = [PSCustomObject]@{ + algo = $Algorithm + value = $Value + } + + return $oHash +} + +function New-Encryption{ +Param( + [Parameter(Mandatory)][String] $Algorithm, + [String] $Value + +) + + $oEncryption = [PSCustomObject]@{ + algo = $Algorithm + value = $Value + } + + return $oEncryption +} + + + +function New-MartiChildItem +{ +Param( + [Parameter(Mandatory)][String] $SourceFolder, + [String] $Filter ="*", + [String] $UrlPath, + [switch] $Recurse, + [switch] $ExtendAttributes, + [switch] $ExcludeHash, + [String] $LogPath + +) + $script:LogPathName = $LogPath + + Write-Debug "Parameter: LogPath Value: $LogPath " + Open-Log + Write-Log "Function 'New-MartiDefinition' parameters follow" + Write-Log "Parameter: SourceFolder Value: $SourceFolder " + Write-Log "Parameter: Filter Value: $Filter " + Write-Log "Parameter: Recurse Value: $Recurse " + Write-Log "Parameter: ExtendAttributes Value: $ExtendAttributes " + Write-Log "Parameter: ExcludeHash Value: $ExcludeHash " + Write-Log "" + + $oMarti = New-MartiDefinition + $lresource = $oMarti.resources + + $SourceFullName = (Get-Item -Path $SourceFolder).FullName + + Get-ChildItem $SourceFolder -Filter $Filter -Recurse:$Recurse -Force| Where-Object {!$_.PSIsContainer} | ForEach-Object { + + $oResource = New-MartiResource -SourcePath $_.FullName -UrlPath $remoteDirectory -LogPath $LogPath -ExtendAttributes:$ExtendAttributes -ExcludeHash:$ExcludeHash + + if ($null -ne $UrlPath -and $UrlPath -ne "") { + $postfixName = $_.FullName.Replace($SourceFullName, "") + if ($postfixName[0] -eq "/" -or $postfixName[0] -eq "`\" ){ + $postfixName = $postfixName.Substring(1, ($postfixName.Length-1)) + } + if ($UrlPath[$UrlPath.Length-1] -eq "/" -or $UrlPath[$UrlPath.Length-1] -eq "`\") { + $oResource.url = $UrlPath.Replace("`\", "/") + $postfixName.Replace("`\", "/") + } else { + $oResource.url = $UrlPath.Replace("`\", "/") + "/" + $postfixName.Replace("`\", "/") + } + } + + $lresource += $oResource + + } + Write-Log "Captured $($lresource.Count) items" + $oMarti.resources = $lresource + + Close-Log + + return $oMarti + +} + + function New-DefaultCsvAttributes { @@ -306,3 +478,4 @@ function Set-MartiResourceAttributes { } + diff --git a/source/powershell/New-Marti.ps1 b/source/powershell/New-Marti.ps1 deleted file mode 100644 index 37df515..0000000 --- a/source/powershell/New-Marti.ps1 +++ /dev/null @@ -1,147 +0,0 @@ - -$script:LogPathName = "" -$script:SoftwareVersion = "0.0.1" - -$global:default_metaFile = "##marti##.mri" - -function Get-LogName { - - $date = Get-Date -f "yyyy-MM-dd" - - if (($null -eq $script:LogPathName) -or ($script:LogPathName -eq "")) - { - return $null - } - - if (!(Test-Path -Path $script:LogPathName)) { - $null = New-Item -Path $script:LogPathName -ItemType Directory - } - - $logName = $(Get-SoftwareName) + "_$date.log" - - return Join-Path -Path $script:LogPathName -ChildPath $logName -} - - -function Write-Log { - param( - [String] $LogEntry - ) - - $sFullPath = Get-LogName - - $dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss" - if ($null -ne $sFullPath -and $sFullPath -ne "") { - - if (!(Test-Path -Path $sFullPath)) { - Write-Host "Log path: $sFullPath" - $null = New-Item -Path $sFullPath -ItemType File - } - Add-Content -Path $sFullPath -Value "[$dateTime]. $LogEntry" - } - Write-Debug "[$dateTime]. $LogEntry" - -} - -function Open-Log { - $dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss" - Write-Log "***********************************************************************************" - Write-Log "* Start of processing: [$dateTime]" - Write-Log "***********************************************************************************" -} - -function Close-Log { - $dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss" - Write-Log "***********************************************************************************" - Write-Log "* End of processing: [$dateTime]" - Write-Log "***********************************************************************************" -} - -function Get-SoftwareName { - return [String] "MARTIREFERENCE" -} - - - -function New-MartiDefinition -{ - - $oSoftware = [PSCustomObject]@{ - extension = "software" - softwareName = "MartiReference" - author = "Meerkat@merebox.com" - version = "$script:SoftwareVersion" - } - - $publisher = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name - - [System.Collections.ArrayList]$lcustom = @() - $lcustom += $oSoftware - - [System.Collections.ArrayList]$lresource = @() - - $oMarti = [PSCustomObject]@{ - title = "" - uid = (New-Guid).ToString() - resources = $lresource - - description = "" - modified = Get-Date -f "yyyy-MM-ddTHH:mm:ss" - tags = @( "document", "marti") - publisher = $publisher - contactPoint = "" - accessLevel = "Confidential" - rights = "Restricted" - license = "" - state = "active" - - describedBy = "" - landingPage = "" - theme ="" - - custom = $lCustom - } - - return $oMarti -} - - -function Set-MartiAttribute -{ -Param( - [System.Collections.ArrayList] $Attributes, - [String] $ACategory, - [String] $AName, - [String] $AFunction, - [String] $Comparison, - [String] $Value -) - - $matched = $false - - $Attributes | ForEach-Object { - - if ($_.category -eq $ACategory -and $_.name -eq $AName -and $_.function -eq $AFunction) { - $matched = $true - $_.comparison = $comparison - $_.value = $value - } - - } - - if (!($matched)) { - - $oAttribute = [PSCustomObject]@{ - category = $Acategory - name = $AName - function = $Afunction - comparison = $comparison - value = $value - } - - $Attributes += $oAttribute - } - - return $Attributes -} - diff --git a/source/powershell/New-MartiChildItem.ps1 b/source/powershell/New-MartiChildItem.ps1 deleted file mode 100644 index f82fe83..0000000 --- a/source/powershell/New-MartiChildItem.ps1 +++ /dev/null @@ -1,58 +0,0 @@ - -function New-MartiChildItem -{ -Param( - [Parameter(Mandatory)][String] $SourceFolder, - [String] $Filter ="*", - [String] $UrlPath, - [switch] $Recurse, - [switch] $ExtendAttributes, - [switch] $ExcludeHash, - [String] $LogPath - -) - $script:LogPathName = $LogPath - - Write-Debug "Parameter: LogPath Value: $LogPath " - Open-Log - Write-Log "Function 'New-MartiDefinition' parameters follow" - Write-Log "Parameter: SourceFolder Value: $SourceFolder " - Write-Log "Parameter: Filter Value: $Filter " - Write-Log "Parameter: Recurse Value: $Recurse " - Write-Log "Parameter: ExtendAttributes Value: $ExtendAttributes " - Write-Log "Parameter: ExcludeHash Value: $ExcludeHash " - Write-Log "" - - $oMarti = New-MartiDefinition - $lresource = $oMarti.resources - - $SourceFullName = (Get-Item -Path $SourceFolder).FullName - - Get-ChildItem $SourceFolder -Filter $Filter -Recurse:$Recurse -Force| Where-Object {!$_.PSIsContainer} | ForEach-Object { - - $oResource = New-MartiResource -SourcePath $_.FullName -UrlPath $remoteDirectory -LogPath $LogPath -ExtendAttributes:$ExtendAttributes -ExcludeHash:$ExcludeHash - - if ($null -ne $UrlPath -and $UrlPath -ne "") { - $postfixName = $_.FullName.Replace($SourceFullName, "") - if ($postfixName[0] -eq "/" -or $postfixName[0] -eq "`\" ){ - $postfixName = $postfixName.Substring(1, ($postfixName.Length-1)) - } - if ($UrlPath[$UrlPath.Length-1] -eq "/" -or $UrlPath[$UrlPath.Length-1] -eq "`\") { - $oResource.url = $UrlPath.Replace("`\", "/") + $postfixName.Replace("`\", "/") - } else { - $oResource.url = $UrlPath.Replace("`\", "/") + "/" + $postfixName.Replace("`\", "/") - } - } - - $lresource += $oResource - - } - Write-Log "Captured $($lresource.Count) items" - $oMarti.resources = $lresource - - Close-Log - - return $oMarti - -} - diff --git a/source/powershell/New-MartiItem.ps1 b/source/powershell/New-MartiItem.ps1 deleted file mode 100644 index 35ceffc..0000000 --- a/source/powershell/New-MartiItem.ps1 +++ /dev/null @@ -1,112 +0,0 @@ - -function New-MartiResource { -Param( - [Parameter(Mandatory)][String] $SourcePath, - [String] $UrlPath = "", - [switch] $ExcludeHash, - [switch] $ExtendAttributes, - [String] $LogPath - -) - - $Global:MartiErrorId = "" - $script:LogPathName = $LogPath - - Write-Debug "Parameter: LogPath Value: $LogPath " - Open-Log - Write-Log "Function 'New-MartiResource' parameters follow" - Write-Log "Parameter: UrlPath Value: $UrlPath " - Write-Log "Parameter: SourcePath Value: $SourcePath " - Write-Log "Parameter: ExcludeHash Value: $ExcludeHash " - Write-Log "" - - - if (Test-Path -Path $SourcePath -PathType Leaf) { - - $item = Get-Item -Path $SourcePath -Force - - Write-Log "Define file $($item.FullName) " - - if ($ExcludeHash) { - $hash = $null - } else { - $hash = New-MartiHash -Algorithm "SHA256" -FilePath $item.FullName - } - - $lattribute = Set-MartiResourceAttributes -Path $item.FullName -FileType $item.Extension.Substring(1) -ExtendedAttributes:$ExtendAttributes - - $oResource = [PSCustomObject]@{ - title = $item.Name.Replace($item.Extension, "") - uid = (New-Guid).ToString() - documentName = $item.Name - issuedDate = Get-Date -f "yyyy-MM-ddTHH:mm:ss" - modified = $item.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss") - state = "active" - author = "" - length = $item.Length - hash = $hash - - description = "" - url = "" - version = $version - format = $item.Extension.Substring(1) - compression = $null - encryption = $null - - attributes = $lattribute - } - - if ($null -ne $UrlPath -and $UrlPath -ne "") { - if ($UrlPath[$UrlPath.Length-1] -eq "/" -or $UrlPath[$UrlPath.Length-1] -eq "\\") { - $oResource.url = $UrlPath.Replace("\\", "/") + $item.Name - } else { - $oResource.url = $UrlPath.Replace("\\", "/") + "/" + $item.Name - } - } - - } else { - $Global:MartiErrorId = "MRI2001" - $message = "Document '$SourcePath' not found or is a folder" - Write-Log ($message + " " + $Global:MartiErrorId) - Close-Log - throw $message - } - Close-Log - - return $oResource - -} - -function New-MartiHash{ - Param( - [Parameter(Mandatory)][String] $Algorithm, - [String] $FilePath - [String] $Value = "" - ) - - if ($Value -eq "" -and $FilePath -ne "") { - $Value = (Get-FileHash -Path $FilePath -Algorithm $Algorithm).Hash - } - - $oHash = [PSCustomObject]@{ - algo = $Algorithm - value = $Value - } - - return $oHash -} - -function New-Encryption{ -Param( - [Parameter(Mandatory)][String] $Algorithm, - [String] $Value - -) - - $oEncryption = [PSCustomObject]@{ - algo = $Algorithm - value = $Value - } - - return $oEncryption -} diff --git a/source/python/client/martiLQ.py b/source/python/client/martiLQ.py index a39f111..c96ac8f 100644 --- a/source/python/client/martiLQ.py +++ b/source/python/client/martiLQ.py @@ -14,7 +14,7 @@ class martiLQ: gLogPathName = "" gSoftwareVersion = "0.0.1" - gdefault_metaFile = "##marti##.mri" + gdefault_metaFile = "##marti##.mti" gMartiErrorId = "" gLogOpen = False @@ -126,7 +126,7 @@ class martiLQ: "rights": "Restricted", "license": "", "state": "active", - + "batch": 1, "describedBy": "", "landingPage": "", "theme": "", @@ -157,6 +157,7 @@ class martiLQ: item = os.path.basename(SourcePath) self.WriteLog("Define file {}".format(SourcePath)) + HashAlgorithm = "SHA256" try: mtime = os.path.getmtime(SourcePath) @@ -167,7 +168,7 @@ class martiLQ: if ExcludeHash: hash = None else: - hash = self.NewMartiHash(Algorithm="SHA256", FilePath=SourcePath, Value="") + hash = self.NewMartiHash(Algorithm=HashAlgorithm, FilePath=SourcePath, Value="") lattribute = self.SetMartiResourceAttributes(SourcePath, os.path.splitext(SourcePath)[1][1:], ExtendAttributes) @@ -561,7 +562,7 @@ class martiLQ: return errorCount - def TestMartiDefinition(self, oMarti, LQSourcePath, LogPath =""): + def TestMartiDefinition(self, LQSourcePath, oMarti = None, LogPath =""): self.gMartiErrorId = "" self.gLogPathName = LogPath diff --git a/test/powershell/test_Marti.ps1 b/test/powershell/test_Marti.ps1 index 1def922..07dc7e9 100644 --- a/test/powershell/test_Marti.ps1 +++ b/test/powershell/test_Marti.ps1 @@ -1,18 +1,17 @@ -. .\source\powershell\New-Marti.ps1 -. .\source\powershell\Get-Marti.ps1 -. .\source\powershell\Compress-Marti.ps1 +. .\source\powershell\MartiLQ.ps1 +. .\source\powershell\Compress-MartiLQ.ps1 Write-Host "Test case #1" $oMarti = New-MartiChildItem -SourceFolder ".\docs" -Recurse -UrlPath ".\docs" -Filter "*" -LogPath ".\test\powershell\results\Logs" $oMarti.description = "Sample execution" $x = ConvertTo-Json -InputObject $oMarti -Set-Content -Path ".\test\powershell\results\marti_test01.mri.json" -Value $x +Set-Content -Path ".\test\powershell\results\marti_test01.mti" -Value $x Write-Host "Test case #2" $ArchiveFile = ".\test\powershell\results\marti_test02.zip" -Compress-Marti -SourceFolder ".\docs" -Filter "*" -LogPath ".\test\powershell\results\Logs" -ArchiveFile $ArchiveFile +Compress-MartiLQ -SourceFolder ".\docs" -Filter "*" -LogPath ".\test\powershell\results\Logs" -ArchiveFile $ArchiveFile Write-Host "Test case #3" $y = Get-MartiItem -MartiDefintiion $oMarti -Title "ckan" -Format "txt" -LogPath ".\test\powershell\results\Logs" @@ -24,22 +23,22 @@ $oMarti = New-MartiResource -SourcePath ".\docs\ckan.md" -LogPath ".\test\powers $oMarti.description = "Sample execution for ckan" $x = ConvertTo-Json -InputObject $oMarti -Set-Content -Path ".\test\powershell\results\marti_test02.mri.json" -Value $x +Set-Content -Path ".\test\powershell\results\marti_test02json.mti" -Value $x $x = ConvertTo-Csv -InputObject $oMarti -Set-Content -Path ".\test\powershell\results\marti_test02.mri.csv" -Value $x +Set-Content -Path ".\test\powershell\results\marti_test02csv.mti." -Value $x $x = ConvertTo-Xml -As String -InputObject $oMarti -Depth 6 -Set-Content -Path ".\test\powershell\results\marti_test02.mri.xml" -Value $x +Set-Content -Path ".\test\powershell\results\marti_test02xml.mti" -Value $x $x = ConvertTo-Html -InputObject $oMarti -Set-Content -Path ".\test\powershell\results\marti_test02.mri.html" -Value $x +Set-Content -Path ".\test\powershell\results\marti_test02html.mti" -Value $x Write-Host "Test case #5" $oMarti = New-MartiResource -SourcePath ".\docs\eror" -LogPath ".\test\powershell\results\Logs" $oMarti.description = "Sample execution with error" $x = ConvertTo-Json -InputObject $oMarti -Set-Content -Path ".\test\powershell\results\marti_test03.mri.json" -Value $x +Set-Content -Path ".\test\powershell\results\marti_test03.mti" -Value $x diff --git a/test/powershell/test_MartiCkan.ps1 b/test/powershell/test_MartiCkan.ps1 index 7fb4338..9f3d995 100644 --- a/test/powershell/test_MartiCkan.ps1 +++ b/test/powershell/test_MartiCkan.ps1 @@ -7,7 +7,7 @@ $ckan = Get-Content -Path ".\docs\samples\asic_ckan_api.json" -Raw $oMarti = ConvertFrom-Ckan -InputObject $ckan $x = ConvertTo-Json -InputObject $oMarti -Set-Content -Path ".\test\powershell\results\marti_test05.mri.json" -Value $x +Set-Content -Path ".\test\powershell\results\marti_test05.mti" -Value $x $covid_1 = Invoke-WebRequest "https://data.nsw.gov.au/data/api/3/action/package_show?id=793ac07d-a5f4-4851-835c-3f7158c19d15" @@ -18,7 +18,7 @@ $oMarti.tags += "gov" $oMarti.tags += "nsw" $oMarti.publisher = "NSW government (Australia)" $x = ConvertTo-Json -InputObject $oMarti -Set-Content -Path ".\test\powershell\results\marti_test06.mri.json" -Value $x +Set-Content -Path ".\test\powershell\results\marti_test06.mti" -Value $x # cases @@ -32,5 +32,5 @@ $oMarti.tags += "gov" $oMarti.tags += "nsw" $oMarti.publisher = "NSW government (Australia)" $x = ConvertTo-Json -InputObject $oMarti -Set-Content -Path ".\test\powershell\results\marti_test07.mri.json" -Value $x +Set-Content -Path ".\test\powershell\results\marti_test07.mti" -Value $x diff --git a/tools.md b/tools.md index 5f8ed30..0ca1a38 100644 --- a/tools.md +++ b/tools.md @@ -2,9 +2,9 @@ A number of tools are povided that can be incorporated into your projects that are want to use the metadata transfer reconciliation format -(Marti document). +(martiLQ document). You can combine these in different ways such as: -* Use Python extract program to generate the Marti document -* Use a Java program to reconcile the received data document with the Marti file +* Use Python extract program to generate the martiLQ document +* Use a Java program to reconcile the received data document with the martiLQ file