Refactored to martiLQ
parent
f360bdd824
commit
d515a6c4a6
11
README.md
11
README.md
|
|
@ -1,6 +1,6 @@
|
|||
# Marti
|
||||
|
||||
**marti** stands for metadata reconcilation for transfer information.
|
||||
**martiLQ** stands for metadata reconcilation for transfer information, load quality.
|
||||
|
||||
The objective is to provide transfer information for high volume data such as
|
||||
in files. The files can be transferred via HTTPS, SFTP, message queue,
|
||||
|
|
@ -15,9 +15,9 @@ it does not understand the data format.
|
|||
**Note**: The terms file and document are intended to be interchangeable
|
||||
through out this documentation.
|
||||
|
||||
**marti** is intended to provide minimum basic information on the transfer with
|
||||
**martiLQ** is intended to provide minimum basic information on the transfer with
|
||||
ability to include optional information. The metadata reconcilation
|
||||
transfer document being decscribed here wil be referred to as the [marti document](Marti.md)
|
||||
transfer document being decscribed here wil be referred to as the [martiLQ document](MartiLQ.md)
|
||||
throughout this documentation.
|
||||
|
||||
The transfer information is supplied as a separate document which could be another file
|
||||
|
|
@ -39,8 +39,8 @@ information
|
|||
|
||||
## Transfer information
|
||||
|
||||
The information in the **marti** document is summarised below. For more detailed
|
||||
information see [marti definition](marti.md)
|
||||
The information in the **martiLQ** document is summarised below. For more detailed
|
||||
information see [martiLQ definition](martiLQ.md)
|
||||
|
||||
### Mandatory information
|
||||
|
||||
|
|
@ -61,6 +61,7 @@ The optional information is:
|
|||
* Contact point
|
||||
* Acces level
|
||||
* Rights
|
||||
* Batch
|
||||
* License
|
||||
* Spatial (*)
|
||||
* Temporal (*)
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
# Documentation
|
||||
|
||||
**marti** stands for metadata reconcilation for transfer information.
|
||||
**martiLQ** stands for metadata reconcilation for transfer information.
|
||||
|
||||
Before starting with **marti** it is advisable to understand if it is right for
|
||||
Before starting with **martiLQ** it is advisable to understand if it is right for
|
||||
your organisation's needs. Information is available in a number of short
|
||||
documents.
|
||||
|
||||
|
|
@ -13,12 +13,12 @@ can adjust if they resonate with your circumstances,
|
|||
|
||||
## Index
|
||||
|
||||
1. [What is marti](what.md)
|
||||
2. [Who would use marti](who.md)
|
||||
3. [Why you would use marti](why.md)
|
||||
4. [When would you start using marti](when.md)
|
||||
1. [What is martiLQ](what.md)
|
||||
2. [Who would use martiLQ](who.md)
|
||||
3. [Why you would use martiLQ](why.md)
|
||||
4. [When would you start using martiLQ](when.md)
|
||||
5. [Load quality metrics support](quality.md)
|
||||
6. [Comparison of marti definition](comparison.md)
|
||||
6. [Comparison of martiLQ definition](comparison.md)
|
||||
7. [References](references.md)
|
||||
|
||||
[!INCLUDE [marti High Level Definition](../marti.md)]
|
||||
[!INCLUDE [martiLQ High Level Definition](../martiLQ.md)]
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
# Comparison of marti document definition
|
||||
# Comparison of martiLQ document definition
|
||||
|
||||
The use of metadata definitions is not unique and examples
|
||||
exist in many different situations. Some are standard and open
|
||||
|
|
@ -8,20 +8,21 @@ Some open standards are EXIF data for pictures, SQL DDL defintions
|
|||
for databases, the XMP definition and web header responses before the
|
||||
web content.
|
||||
|
||||
The **marti** document definition is intended to cover the situation
|
||||
The **martiLQ** document definition is intended to cover the situation
|
||||
where data files are being transferred and reconciliation is required.
|
||||
|
||||
The **marti** document definition is modelled on the [CKAN API metadata](https://docs.ckan.org/en/2.9/api/index.html)
|
||||
The **martiLQ** document definition is modelled on
|
||||
the [CKAN API metadata](https://docs.ckan.org/en/2.9/api/index.html)
|
||||
which has been adapted to included additional elements relevant to when
|
||||
you are exchanging data files. This includes the reconciliation elements
|
||||
such as number of records and file hash.
|
||||
|
||||
As the definition is based on the CKAN API, there are tools to import
|
||||
a CKAN source into a **marti** document definition and then process the data
|
||||
a CKAN source into a **martiLQ** document definition and then process the data
|
||||
through the pipeline as you would for any other data file that had a
|
||||
**marti** document definition.
|
||||
**martiLQ** document definition.
|
||||
|
||||
## Benefit of CKAN and marti
|
||||
## Benefit of CKAN and martiLQ
|
||||
|
||||
The CKAN is excellent at defining the data source details but it lacks information
|
||||
for load quality. If you have CKAN deployed in your organisation and wish
|
||||
|
|
@ -30,7 +31,7 @@ CKAN and marti.
|
|||
|
||||
Samples exist on CKAN integration.
|
||||
|
||||
## Magda and marti
|
||||
## Magda and martiLQ
|
||||
|
||||
Another source of data is [Magda](https://magda.io/) which has API metadata
|
||||
definitions. Magda is more about data fedaration and as such provides
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ such as end of day.
|
|||
Some files may expand to multiple files if they are
|
||||
compressed with a utility such as WinZIP or 7ZIP. In the situation
|
||||
where a ZIP file expands to multiple documents, then the expectation is
|
||||
that the ZIP file contains a **marti** document describing its contents.
|
||||
that the ZIP file contains a **martiLQ** document describing its contents.
|
||||
|
||||
The elements in the distribution section are:
|
||||
|
||||
|
|
@ -41,15 +41,15 @@ The following are optional in the distribution section.
|
|||
## Compression
|
||||
|
||||
Files can be compressed using a utility. A single compressed file can contain
|
||||
multiple files. The **marti** definition document applies to the compressed file
|
||||
multiple files. The **martiLQ** definition document applies to the compressed file
|
||||
and not to the contents, which could be multiple files.
|
||||
|
||||
In the case of a compressed files, there should be a **marti** definition document in the
|
||||
In the case of a compressed files, there should be a **martiLQ** definition document in the
|
||||
compressed file.
|
||||
|
||||
Compression of files always occur before encryption.
|
||||
|
||||
### Marti definition for Compressed File
|
||||
### martiLQ definition for Compressed File
|
||||
|
||||
For a compressed file that is not encrypted, the distribution definition will be:
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Quality definition
|
||||
|
||||
The **marti** definition allows for the inclusion of a load quality
|
||||
The **martiLQ** definition allows for the inclusion of a load quality
|
||||
definition. This load quality definition is intended to be
|
||||
able to be applied universally with common tools. As such not
|
||||
all needs are covered.
|
||||
|
|
|
|||
|
|
@ -1,10 +1,7 @@
|
|||
|
||||
. ..\..\..\source\powershell\New-Marti.ps1
|
||||
. ..\..\..\source\powershell\New-MartiChildItem.ps1
|
||||
. ..\..\..\source\powershell\New-MartiResource.ps1
|
||||
. ..\..\..\source\powershell\Get-Marti.ps1
|
||||
. ..\..\..\source\powershell\Compress-Marti.ps1
|
||||
. ..\..\..\source\powershell\Get-MartiResourceAttributes.ps1
|
||||
. ..\..\..\source\powershell\MartiLQ.ps1
|
||||
. ..\..\..\source\powershell\MartiLQItem.ps1
|
||||
. ..\..\..\source\powershell\Compress-MartiLQ.ps1
|
||||
|
||||
|
||||
function PullFtpFile {
|
||||
|
|
@ -89,7 +86,7 @@ $fileList = ListFtpDirectory -Username "anonymous" -Password "anon@merebox.com"
|
|||
Write-Host "File list size: $($fileList.count)" -ForegroundColor Gray
|
||||
|
||||
|
||||
Write-Host "Now iterate through the remote files and build remote marti list " -ForeGroundColor Green
|
||||
Write-Host "Now iterate through the remote files and build remote martiLQ list " -ForeGroundColor Green
|
||||
|
||||
$oMarti = New-MartiDefinition
|
||||
$oMarti.title = "Remote_BSB_data"
|
||||
|
|
@ -101,7 +98,7 @@ $oMarti.theme = "payment"
|
|||
ForEach ($item in $fileList) {
|
||||
if ($item -ne "" -and $item.startswith("BSBDirectory")) {
|
||||
PullFtpFile -Username "anonymous" -Password "anon@merebox.com" -RemoteFile ($remoteDirectory + $item) -OutputPath (Join-Path -Path $localDirectory -ChildPath $item)
|
||||
Write-Host "Add BSB $item file to Remote marti metadata sample " -ForeGroundColor Yellow
|
||||
Write-Host "Add BSB $item file to Remote martiLQ metadata sample " -ForeGroundColor Yellow
|
||||
$oResource = New-MartiResource -SourcePath (Join-Path -Path $localDirectory -ChildPath $item) -UrlPath $remoteDirectory -LogPath ".\test\Logs" -ExtendAttributes
|
||||
if ($item.endswith(".txt")) {
|
||||
Set-AttributeValueNumber -Attributes $oResource.attributes -Key "header" -Category "dataset" -Function "count" -Value 1
|
||||
|
|
@ -116,13 +113,12 @@ ForEach ($item in $fileList) {
|
|||
}
|
||||
}
|
||||
|
||||
$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBRemote.mri.json"
|
||||
$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBRemote.mti"
|
||||
$oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson
|
||||
Write-Host "Remote marti definition file is $fileJson " -ForeGroundColor Green
|
||||
Write-Host "Note that file has been given JSON extension for sample purposes" -ForeGroundColor Gray
|
||||
Write-Host "Remote martiLQ definition file is $fileJson " -ForeGroundColor Green
|
||||
|
||||
|
||||
Write-Host "Now iterate through the local files and build marti ZIP " -ForeGroundColor Green
|
||||
Write-Host "Now iterate through the local files and build martiLQ ZIP " -ForeGroundColor Green
|
||||
|
||||
$oMarti = New-MartiDefinition
|
||||
$oMarti.title = "Zip_BSB_data"
|
||||
|
|
@ -139,7 +135,7 @@ if (Test-Path -Path $zipFile) {
|
|||
foreach($file in Get-ChildItem $localDirectory)
|
||||
{
|
||||
if ($file.Name.startswith("BSBDirectory") -and !($file.Name.EndsWith(".zip")) -and !($file.Name.EndsWith(".7z")) ) {
|
||||
Write-Host "Add BSB file $file to ZIP marti metadata sample " -ForeGroundColor Yellow
|
||||
Write-Host "Add BSB file $file to ZIP martiLQ metadata sample " -ForeGroundColor Yellow
|
||||
Compress-Archive -Path $file.FullName -DestinationPath $zipFile -Update
|
||||
$oResource = New-MartiResource -SourcePath $file.FullName -UrlPath $localDirectory -LogPath ".\test\Logs" -ExtendAttributes
|
||||
if ($file.Extension -eq ".txt") {
|
||||
|
|
@ -159,10 +155,9 @@ $oResource = New-MartiResource -SourcePath $zipFile -UrlPath $localDirectory -Lo
|
|||
Set-AttributeValueString -Attributes $oResource.attributes -Key "compression" -Category "format" -Function "algorithm" -Value "WINZIP"
|
||||
$oMarti.resources += $oResource
|
||||
|
||||
$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBZip.mri.json"
|
||||
$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBZip.mti"
|
||||
$oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson
|
||||
Write-Host "ZIP marti definition file is $fileJson " -ForeGroundColor Green
|
||||
Write-Host "Note that file has been given JSON extension for sample purposes" -ForeGroundColor Gray
|
||||
Write-Host "ZIP martiLQ definition file is $fileJson " -ForeGroundColor Green
|
||||
|
||||
|
||||
|
||||
|
|
@ -175,10 +170,9 @@ $oMarti.contactPoint = "meerkat@merebox.com"
|
|||
$oMarti.landingPage = "https://github.com/meerkat-manor/marti/blob/draft_specifications/docs/samples/powershell/Invoke-BSBSample.ps1"
|
||||
$oMarti.theme = "payment"
|
||||
|
||||
$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBLocal.mri.json"
|
||||
$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBLocal.mti"
|
||||
$oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson
|
||||
Write-Host "Local marti definition file is $fileJson " -ForeGroundColor Green
|
||||
Write-Host "Note that file has been given JSON extension for sample purposes" -ForeGroundColor Gray
|
||||
Write-Host "Local martiLQ definition file is $fileJson " -ForeGroundColor Green
|
||||
|
||||
|
||||
Write-Host "Now create an encrypted 7ZIP file with asymmetric password protection" -ForeGroundColor Green
|
||||
|
|
@ -198,7 +192,7 @@ if (Test-Path -Path $zipFile) {
|
|||
foreach($file in Get-ChildItem $localDirectory)
|
||||
{
|
||||
if ($file.Name.startswith("BSBDirectory") -and !($file.Name.EndsWith(".zip")) -and !($file.Name.EndsWith(".7z")) ) {
|
||||
Write-Host "Add BSB file $file to 7ZIP marti metadata sample " -ForeGroundColor Yellow
|
||||
Write-Host "Add BSB file $file to 7ZIP martiLQ metadata sample " -ForeGroundColor Yellow
|
||||
if (Test-Path -Path $zipFile) {
|
||||
Compress-7Zip -Path $file.FullName -ArchiveFileName $zipFile -Format SevenZip -Append
|
||||
} else {
|
||||
|
|
@ -220,7 +214,7 @@ foreach($file in Get-ChildItem $localDirectory)
|
|||
}
|
||||
|
||||
$noticeFile = Join-Path -Path $localDirectory -ChildPath "README.txt"
|
||||
Set-Content -Path $noticeFile -Value "Generated by marti Samples"
|
||||
Set-Content -Path $noticeFile -Value "Generated by martiLQ Samples"
|
||||
$oResource = New-MartiResource -SourcePath $noticeFile -UrlPath $localDirectory -LogPath ".\test\Logs"
|
||||
$oMarti.resources += $oResource
|
||||
|
||||
|
|
@ -233,10 +227,9 @@ $oResource.encryption = New-Encryption -Algorithm "Passphrase" -Value $secret
|
|||
Set-AttributeValueString -Attributes $oResource.attributes -Key "compression" -Category "format" -Function "algorithm" -Value "7ZIP"
|
||||
$oMarti.resources += $oResource
|
||||
|
||||
$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBSecure.mri.json"
|
||||
$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSBSecure.mti"
|
||||
$oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson
|
||||
Write-Host "Secure 7ZIP marti definition file is $fileJson " -ForeGroundColor Green
|
||||
Write-Host "Note that file has been given JSON extension for sample purposes" -ForeGroundColor Gray
|
||||
Write-Host "Secure 7ZIP martiLQ definition file is $fileJson " -ForeGroundColor Green
|
||||
|
||||
|
||||
Write-Host "Sample execution completed" -ForeGroundColor Green
|
||||
|
|
|
|||
|
|
@ -1,10 +1,7 @@
|
|||
|
||||
. ..\..\..\source\powershell\New-Marti.ps1
|
||||
. ..\..\..\source\powershell\New-MartiChildItem.ps1
|
||||
. ..\..\..\source\powershell\New-MartiResource.ps1
|
||||
. ..\..\..\source\powershell\Get-Marti.ps1
|
||||
. ..\..\..\source\powershell\Compress-Marti.ps1
|
||||
. ..\..\..\source\powershell\Get-MartiResourceAttributes.ps1
|
||||
. ..\..\..\source\powershell\MartiLQ.ps1
|
||||
. ..\..\..\source\powershell\MartiLQItem.ps1
|
||||
. ..\..\..\source\powershell\Compress-MartiLQ.ps1
|
||||
|
||||
|
||||
function PullFtpFile {
|
||||
|
|
@ -127,7 +124,7 @@ if (Test-Path -Path $zipFile) {
|
|||
foreach($file in Get-ChildItem $localDirectory)
|
||||
{
|
||||
if ($file.Name.startswith("BSBDirectory") -and $file.Name.EndsWith(".csv") ) {
|
||||
Write-Host "Add BSB file $file to 7ZIP marti metadata sample " -ForeGroundColor Yellow
|
||||
Write-Host "Add BSB file $file to 7ZIP martiLQ metadata sample " -ForeGroundColor Yellow
|
||||
if (Test-Path -Path $zipFile) {
|
||||
Compress-7Zip -Path $file.FullName -ArchiveFileName $zipFile -Format SevenZip -Append
|
||||
} else {
|
||||
|
|
@ -149,7 +146,7 @@ foreach($file in Get-ChildItem $localDirectory)
|
|||
}
|
||||
|
||||
$noticeFile = Join-Path -Path $localDirectory -ChildPath "README.txt"
|
||||
Set-Content -Path $noticeFile -Value "Generated by marti Samples"
|
||||
Set-Content -Path $noticeFile -Value "Generated by martiLQ Samples"
|
||||
$oResource = New-MartiResource -SourcePath $noticeFile -UrlPath $localDirectory -LogPath ".\test\Logs"
|
||||
$oMarti.resources += $oResource
|
||||
|
||||
|
|
@ -167,9 +164,8 @@ Write-Debug "Secret: $secret"
|
|||
Set-AttributeValueString -Attributes $oResource.attributes -Key "compression" -Category "format" -Function "algorithm" -Value "7ZIP"
|
||||
$oMarti.resources += $oResource
|
||||
|
||||
$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSB7ZipPKI.mri.json"
|
||||
$oMarti | ConvertTo-Json -depth 100 | Out-File $fileJson
|
||||
Write-Host "7ZIP marti definition file is $fileJson " -ForeGroundColor Green
|
||||
Write-Host "Note that file has been given JSON extension for sample purposes" -ForeGroundColor Gray
|
||||
$fileJson = Join-Path -Path $localDirectory -ChildPath "MartiBSB7ZipPKI.mti"
|
||||
$oMarti | ConvertTo-Json -depth 50 | Out-File $fileJson
|
||||
Write-Host "7ZIP martiLQ definition file is $fileJson " -ForeGroundColor Green
|
||||
|
||||
Write-Host "Sample secure execution completed" -ForeGroundColor Green
|
||||
|
|
|
|||
|
|
@ -59,9 +59,9 @@ for file_name in files:
|
|||
if file_name.endswith(".csv") | file_name.endswith(".txt"):
|
||||
file_remote = remote_dir + file_name
|
||||
file_local = "./test/" + file_name
|
||||
# ftpPull(remote_host, file_remote, file_local)
|
||||
ftpPull(remote_host, file_remote, file_local)
|
||||
|
||||
print("Creating marti definition")
|
||||
print("Creating martiLQ definition")
|
||||
mlq = martiLQ()
|
||||
oMarti = mlq.NewMartiDefinition()
|
||||
|
||||
|
|
@ -72,15 +72,15 @@ for file_name in files:
|
|||
oMarti["resources"].append(oResource)
|
||||
|
||||
mlq.CloseLog()
|
||||
print("Save marti definition")
|
||||
print("Save martiLQ definition")
|
||||
jd = json.dumps(oMarti, indent=5)
|
||||
|
||||
jsonFile = open("./test/BSBDirectoryPlain.mri.json", "w")
|
||||
jsonFile = open("./test/BSBDirectoryPlain.mti", "w")
|
||||
jsonFile.write(jd)
|
||||
jsonFile.close()
|
||||
print("Sample completed: SampleGenerateBsb.py")
|
||||
|
||||
lqresults, testError = mlq.TestMartiDefinition(oMarti, "./test/BSBDirectoryPlain.mri.json")
|
||||
lqresults, testError = mlq.TestMartiDefinition(oMarti, "./test/BSBDirectoryPlain.mti")
|
||||
|
||||
testfile = open("./test/LoadQualityTest01.csv", "w+", newline ="")
|
||||
with testfile:
|
||||
|
|
|
|||
|
|
@ -1,13 +1,13 @@
|
|||
# What is marti
|
||||
|
||||
The foundation pillar for the **marti** framework is the [marti document](../marti.md)
|
||||
The foundation pillar for the **martiLQ** framework is the [martiLQ document](../martiLQ.md)
|
||||
that defines the reconciliation and other metadata of the document / file being transferred.
|
||||
|
||||
A definition, while fundamental, benefits from having tools that can create, read and
|
||||
interpret the definition. The **marti** framework is about providing those tools
|
||||
interpret the definition. The **martiLQ** framework is about providing those tools
|
||||
and a ecosystem that can be added to.
|
||||
|
||||
The majority of the effort in creating **marti** is in the tools. There are tools
|
||||
The majority of the effort in creating **martiLQ** is in the tools. There are tools
|
||||
for various programming languages and situations. As many programming languages
|
||||
generate portable programs that can execute on multiple operating systems, the
|
||||
likelihood is that a tools exists for you.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# When would you use marti
|
||||
# When would you use martiLQ
|
||||
|
||||
You are likely to start using the **marti** framework when:
|
||||
You are likely to start using the **martiLQ** framework when:
|
||||
1. you have no existing standard or framework or;
|
||||
2. your existing standards no longer meets you needs or;
|
||||
3. you are starting document exchange with another business or division that uses the framework or;
|
||||
|
|
@ -12,7 +12,7 @@ of the framework are unlikely to weigh in your framework
|
|||
|
||||
## Read the material
|
||||
|
||||
Please read the material before jumping in and make sure the **marti** framework will
|
||||
Please read the material before jumping in and make sure the **martiLQ** framework will
|
||||
provide you with the expected benefits. Implementing the framework and then changing
|
||||
your direction will more than likely add to your support costs.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Who is likely to use marti
|
||||
|
||||
You are likely to find the **marti** framework relevant if you:
|
||||
You are likely to find the **martiLQ** framework relevant if you:
|
||||
|
||||
1. Have many document exchanges
|
||||
2. Need to verify or reconcile the documents
|
||||
|
|
@ -24,7 +24,7 @@ Simple framework such as the above have limitations, such as:
|
|||
|
||||
## Framework Sidecar files
|
||||
|
||||
The **marti** framework addresses the issues and limitations by using sidecar
|
||||
The **martiLQ** framework addresses the issues and limitations by using sidecar
|
||||
or shadow files. The [concept of sidecar files](https://en.wikipedia.org/wiki/Sidecar_file) is
|
||||
not new and are commonly found associated to media file processing.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Why use marti
|
||||
|
||||
**marti** is a framework for providing a degree of auditability and reconciliation of
|
||||
**martiLQ** is a framework for providing a degree of auditability and reconciliation of
|
||||
documents transferred between systems in an organisation and externally. It does not intend
|
||||
define the format or content of the document. It defines controls that can be used to:
|
||||
|
||||
|
|
@ -11,7 +11,7 @@ define the format or content of the document. It defines controls that can be u
|
|||
5. describe metadata including format, extract time
|
||||
6. link to further information
|
||||
|
||||
You would use **marti** if any of the controls are a requirement for you.
|
||||
You would use **martiLQ** if any of the controls are a requirement for you.
|
||||
|
||||
## Documents
|
||||
|
||||
|
|
@ -19,7 +19,7 @@ Documents in this context are digital storage objects such as operating system f
|
|||
cloud storage objects or blobs. The document content has structure and contains multiple
|
||||
records.
|
||||
|
||||
The **marti** framework is not intended to be used for single record transfers such as
|
||||
The **martiLQ** framework is not intended to be used for single record transfers such as
|
||||
in single web transactions. It is for providing controls when moving large amounts of
|
||||
data as one event. This data are commonly referred to as batch extracts and performed
|
||||
at scheduled times such as end of day.
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
# Marti document
|
||||
# MartiLQ document
|
||||
|
||||
The metadata reconciliation transfer information is referred
|
||||
to as the **marti** document throughout this documentation.
|
||||
to as the **martiLQ** document throughout this documentation.
|
||||
|
||||
The **marti** document can be part of a message or a document
|
||||
The **martiLQ** document can be part of a message or a document
|
||||
in its own right. If the document is a file then the recommended
|
||||
name for the document is the same name as the data file,
|
||||
including extension, with the added extension of ``.mti``
|
||||
|
|
@ -1,5 +1,5 @@
|
|||
# Contents
|
||||
|
||||
This is the parent directory for **marti** tools written in various languages.
|
||||
This is the parent directory for **martiLQ** tools written in various languages.
|
||||
|
||||
Please browse the folder for the programming language of interest.
|
||||
|
|
|
|||
|
|
@ -16,7 +16,7 @@ function Compare-MartiResource {
|
|||
|
||||
if ($null -eq $Resource) {
|
||||
$Global:MartiErrorId = "MRI2201"
|
||||
$message = "No Marti resource definition supplied"
|
||||
$message = "No resource definition supplied"
|
||||
Write-Log ($message + " " + $Global:MartiErrorId)
|
||||
Close-Log
|
||||
throw $message
|
||||
|
|
|
|||
|
|
@ -1,68 +1,6 @@
|
|||
|
||||
$script:LogPathName = ""
|
||||
$script:SoftwareVersion = "0.0.1"
|
||||
|
||||
$global:default_metaFile = "##marti##.mri"
|
||||
|
||||
function Get-LogName {
|
||||
|
||||
$date = Get-Date -f "yyyy-MM-dd"
|
||||
|
||||
if (($null -eq $script:LogPathName) -or ($script:LogPathName -eq ""))
|
||||
{
|
||||
return $null
|
||||
}
|
||||
|
||||
if (!(Test-Path -Path $script:LogPathName)) {
|
||||
$null = New-Item -Path $script:LogPathName -ItemType Directory
|
||||
}
|
||||
|
||||
$logName = $(Get-SoftwareName) + "_$date.log"
|
||||
|
||||
return Join-Path -Path $script:LogPathName -ChildPath $logName
|
||||
}
|
||||
|
||||
|
||||
function Write-Log {
|
||||
param(
|
||||
[String] $LogEntry
|
||||
)
|
||||
|
||||
$sFullPath = Get-LogName
|
||||
|
||||
$dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss"
|
||||
if ($null -ne $sFullPath -and $sFullPath -ne "") {
|
||||
|
||||
if (!(Test-Path -Path $sFullPath)) {
|
||||
Write-Host "Log path: $sFullPath"
|
||||
$null = New-Item -Path $sFullPath -ItemType File
|
||||
}
|
||||
Add-Content -Path $sFullPath -Value "[$dateTime]. $LogEntry"
|
||||
}
|
||||
Write-Debug "[$dateTime]. $LogEntry"
|
||||
|
||||
}
|
||||
|
||||
function Open-Log {
|
||||
$dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss"
|
||||
Write-Log "***********************************************************************************"
|
||||
Write-Log "* Start of processing: [$dateTime]"
|
||||
Write-Log "***********************************************************************************"
|
||||
}
|
||||
|
||||
function Close-Log {
|
||||
$dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss"
|
||||
Write-Log "***********************************************************************************"
|
||||
Write-Log "* End of processing: [$dateTime]"
|
||||
Write-Log "***********************************************************************************"
|
||||
}
|
||||
|
||||
function Get-SoftwareName {
|
||||
return [String] "MARTIREFERENCE"
|
||||
}
|
||||
|
||||
|
||||
function Compress-Marti
|
||||
function Compress-MartiLQ
|
||||
{
|
||||
Param(
|
||||
[Parameter(Mandatory)][String] $SourceFolder,
|
||||
|
|
@ -76,7 +14,7 @@ Param(
|
|||
|
||||
Write-Debug "Parameter: LogPath Value: $LogPath "
|
||||
Open-Log
|
||||
Write-Log "Function 'Compress-Marti' parameters follow"
|
||||
Write-Log "Function 'Compress-MartiLQ' parameters follow"
|
||||
Write-Log "Parameter: SourceFolder Value: $SourceFolder "
|
||||
Write-Log "Parameter: ArchiveFile Value: $ArchiveFile "
|
||||
Write-Log "Parameter: Filter Value: $Filter "
|
||||
|
|
@ -2,7 +2,7 @@
|
|||
$script:LogPathName = ""
|
||||
$script:SoftwareVersion = "0.0.1"
|
||||
|
||||
$global:default_metaFile = "##marti##.mri"
|
||||
$global:default_metaFile = "##marti##.mti"
|
||||
|
||||
function Get-LogName {
|
||||
|
||||
|
|
@ -63,6 +63,89 @@ function Get-SoftwareName {
|
|||
|
||||
|
||||
|
||||
function New-MartiDefinition
|
||||
{
|
||||
|
||||
$oSoftware = [PSCustomObject]@{
|
||||
extension = "software"
|
||||
softwareName = "MartiReference"
|
||||
author = "Meerkat@merebox.com"
|
||||
version = "$script:SoftwareVersion"
|
||||
}
|
||||
|
||||
$publisher = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name
|
||||
|
||||
[System.Collections.ArrayList]$lcustom = @()
|
||||
$lcustom += $oSoftware
|
||||
|
||||
[System.Collections.ArrayList]$lresource = @()
|
||||
|
||||
$oMarti = [PSCustomObject]@{
|
||||
title = ""
|
||||
uid = (New-Guid).ToString()
|
||||
resources = $lresource
|
||||
|
||||
description = ""
|
||||
modified = Get-Date -f "yyyy-MM-ddTHH:mm:ss"
|
||||
tags = @( "document", "marti")
|
||||
publisher = $publisher
|
||||
contactPoint = ""
|
||||
accessLevel = "Confidential"
|
||||
rights = "Restricted"
|
||||
license = ""
|
||||
state = "active"
|
||||
batch = 1
|
||||
describedBy = ""
|
||||
landingPage = ""
|
||||
theme =""
|
||||
|
||||
custom = $lCustom
|
||||
}
|
||||
|
||||
return $oMarti
|
||||
}
|
||||
|
||||
|
||||
function Set-MartiAttribute
|
||||
{
|
||||
Param(
|
||||
[System.Collections.ArrayList] $Attributes,
|
||||
[String] $ACategory,
|
||||
[String] $AName,
|
||||
[String] $AFunction,
|
||||
[String] $Comparison,
|
||||
[String] $Value
|
||||
)
|
||||
|
||||
$matched = $false
|
||||
|
||||
$Attributes | ForEach-Object {
|
||||
|
||||
if ($_.category -eq $ACategory -and $_.name -eq $AName -and $_.function -eq $AFunction) {
|
||||
$matched = $true
|
||||
$_.comparison = $comparison
|
||||
$_.value = $value
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
if (!($matched)) {
|
||||
|
||||
$oAttribute = [PSCustomObject]@{
|
||||
category = $Acategory
|
||||
name = $AName
|
||||
function = $Afunction
|
||||
comparison = $comparison
|
||||
value = $value
|
||||
}
|
||||
|
||||
$Attributes += $oAttribute
|
||||
}
|
||||
|
||||
return $Attributes
|
||||
}
|
||||
|
||||
|
||||
|
||||
function Get-MartChildResource
|
||||
{
|
||||
|
|
@ -87,7 +170,7 @@ function Get-MartChildResource
|
|||
|
||||
if ($null -eq $Marti) {
|
||||
$Global:MartiErrorId = "MRI2101"
|
||||
$message = "No Marti definition supplied"
|
||||
$message = "No definition supplied"
|
||||
Write-Log ($message + " " + $Global:MartiErrorId)
|
||||
Close-Log
|
||||
throw $message
|
||||
|
|
@ -1,4 +1,176 @@
|
|||
|
||||
function New-MartiResource {
|
||||
Param(
|
||||
[Parameter(Mandatory)][String] $SourcePath,
|
||||
[String] $UrlPath = "",
|
||||
[switch] $ExcludeHash,
|
||||
[switch] $ExtendAttributes,
|
||||
[String] $LogPath
|
||||
|
||||
)
|
||||
|
||||
$Global:MartiErrorId = ""
|
||||
$script:LogPathName = $LogPath
|
||||
|
||||
Write-Debug "Parameter: LogPath Value: $LogPath "
|
||||
Open-Log
|
||||
Write-Log "Function 'New-MartiResource' parameters follow"
|
||||
Write-Log "Parameter: UrlPath Value: $UrlPath "
|
||||
Write-Log "Parameter: SourcePath Value: $SourcePath "
|
||||
Write-Log "Parameter: ExcludeHash Value: $ExcludeHash "
|
||||
Write-Log ""
|
||||
|
||||
|
||||
if (Test-Path -Path $SourcePath -PathType Leaf) {
|
||||
|
||||
$item = Get-Item -Path $SourcePath -Force
|
||||
|
||||
Write-Log "Define file $($item.FullName) "
|
||||
|
||||
if ($ExcludeHash) {
|
||||
$hash = $null
|
||||
} else {
|
||||
$hash = New-MartiHash -Algorithm "SHA256" -FilePath $item.FullName
|
||||
}
|
||||
|
||||
$lattribute = Set-MartiResourceAttributes -Path $item.FullName -FileType $item.Extension.Substring(1) -ExtendedAttributes:$ExtendAttributes
|
||||
|
||||
$oResource = [PSCustomObject]@{
|
||||
title = $item.Name.Replace($item.Extension, "")
|
||||
uid = (New-Guid).ToString()
|
||||
documentName = $item.Name
|
||||
issuedDate = Get-Date -f "yyyy-MM-ddTHH:mm:ss"
|
||||
modified = $item.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss")
|
||||
state = "active"
|
||||
author = ""
|
||||
length = $item.Length
|
||||
hash = $hash
|
||||
|
||||
description = ""
|
||||
url = ""
|
||||
version = $version
|
||||
format = $item.Extension.Substring(1)
|
||||
compression = $null
|
||||
encryption = $null
|
||||
|
||||
attributes = $lattribute
|
||||
}
|
||||
|
||||
if ($null -ne $UrlPath -and $UrlPath -ne "") {
|
||||
if ($UrlPath[$UrlPath.Length-1] -eq "/" -or $UrlPath[$UrlPath.Length-1] -eq "\\") {
|
||||
$oResource.url = $UrlPath.Replace("\\", "/") + $item.Name
|
||||
} else {
|
||||
$oResource.url = $UrlPath.Replace("\\", "/") + "/" + $item.Name
|
||||
}
|
||||
}
|
||||
|
||||
} else {
|
||||
$Global:MartiErrorId = "MRI2001"
|
||||
$message = "Document '$SourcePath' not found or is a folder"
|
||||
Write-Log ($message + " " + $Global:MartiErrorId)
|
||||
Close-Log
|
||||
throw $message
|
||||
}
|
||||
Close-Log
|
||||
|
||||
return $oResource
|
||||
|
||||
}
|
||||
|
||||
function New-MartiHash{
|
||||
Param(
|
||||
[Parameter(Mandatory)][String] $Algorithm,
|
||||
[String] $FilePath,
|
||||
[String] $Value = ""
|
||||
)
|
||||
|
||||
if ($Value -eq "" -and $FilePath -ne "") {
|
||||
$Value = (Get-FileHash -Path $FilePath -Algorithm $Algorithm).Hash
|
||||
}
|
||||
|
||||
$oHash = [PSCustomObject]@{
|
||||
algo = $Algorithm
|
||||
value = $Value
|
||||
}
|
||||
|
||||
return $oHash
|
||||
}
|
||||
|
||||
function New-Encryption{
|
||||
Param(
|
||||
[Parameter(Mandatory)][String] $Algorithm,
|
||||
[String] $Value
|
||||
|
||||
)
|
||||
|
||||
$oEncryption = [PSCustomObject]@{
|
||||
algo = $Algorithm
|
||||
value = $Value
|
||||
}
|
||||
|
||||
return $oEncryption
|
||||
}
|
||||
|
||||
|
||||
|
||||
function New-MartiChildItem
|
||||
{
|
||||
Param(
|
||||
[Parameter(Mandatory)][String] $SourceFolder,
|
||||
[String] $Filter ="*",
|
||||
[String] $UrlPath,
|
||||
[switch] $Recurse,
|
||||
[switch] $ExtendAttributes,
|
||||
[switch] $ExcludeHash,
|
||||
[String] $LogPath
|
||||
|
||||
)
|
||||
$script:LogPathName = $LogPath
|
||||
|
||||
Write-Debug "Parameter: LogPath Value: $LogPath "
|
||||
Open-Log
|
||||
Write-Log "Function 'New-MartiDefinition' parameters follow"
|
||||
Write-Log "Parameter: SourceFolder Value: $SourceFolder "
|
||||
Write-Log "Parameter: Filter Value: $Filter "
|
||||
Write-Log "Parameter: Recurse Value: $Recurse "
|
||||
Write-Log "Parameter: ExtendAttributes Value: $ExtendAttributes "
|
||||
Write-Log "Parameter: ExcludeHash Value: $ExcludeHash "
|
||||
Write-Log ""
|
||||
|
||||
$oMarti = New-MartiDefinition
|
||||
$lresource = $oMarti.resources
|
||||
|
||||
$SourceFullName = (Get-Item -Path $SourceFolder).FullName
|
||||
|
||||
Get-ChildItem $SourceFolder -Filter $Filter -Recurse:$Recurse -Force| Where-Object {!$_.PSIsContainer} | ForEach-Object {
|
||||
|
||||
$oResource = New-MartiResource -SourcePath $_.FullName -UrlPath $remoteDirectory -LogPath $LogPath -ExtendAttributes:$ExtendAttributes -ExcludeHash:$ExcludeHash
|
||||
|
||||
if ($null -ne $UrlPath -and $UrlPath -ne "") {
|
||||
$postfixName = $_.FullName.Replace($SourceFullName, "")
|
||||
if ($postfixName[0] -eq "/" -or $postfixName[0] -eq "`\" ){
|
||||
$postfixName = $postfixName.Substring(1, ($postfixName.Length-1))
|
||||
}
|
||||
if ($UrlPath[$UrlPath.Length-1] -eq "/" -or $UrlPath[$UrlPath.Length-1] -eq "`\") {
|
||||
$oResource.url = $UrlPath.Replace("`\", "/") + $postfixName.Replace("`\", "/")
|
||||
} else {
|
||||
$oResource.url = $UrlPath.Replace("`\", "/") + "/" + $postfixName.Replace("`\", "/")
|
||||
}
|
||||
}
|
||||
|
||||
$lresource += $oResource
|
||||
|
||||
}
|
||||
Write-Log "Captured $($lresource.Count) items"
|
||||
$oMarti.resources = $lresource
|
||||
|
||||
Close-Log
|
||||
|
||||
return $oMarti
|
||||
|
||||
}
|
||||
|
||||
|
||||
|
||||
function New-DefaultCsvAttributes {
|
||||
|
||||
|
|
@ -306,3 +478,4 @@ function Set-MartiResourceAttributes {
|
|||
}
|
||||
|
||||
|
||||
|
||||
|
|
@ -1,147 +0,0 @@
|
|||
|
||||
$script:LogPathName = ""
|
||||
$script:SoftwareVersion = "0.0.1"
|
||||
|
||||
$global:default_metaFile = "##marti##.mri"
|
||||
|
||||
function Get-LogName {
|
||||
|
||||
$date = Get-Date -f "yyyy-MM-dd"
|
||||
|
||||
if (($null -eq $script:LogPathName) -or ($script:LogPathName -eq ""))
|
||||
{
|
||||
return $null
|
||||
}
|
||||
|
||||
if (!(Test-Path -Path $script:LogPathName)) {
|
||||
$null = New-Item -Path $script:LogPathName -ItemType Directory
|
||||
}
|
||||
|
||||
$logName = $(Get-SoftwareName) + "_$date.log"
|
||||
|
||||
return Join-Path -Path $script:LogPathName -ChildPath $logName
|
||||
}
|
||||
|
||||
|
||||
function Write-Log {
|
||||
param(
|
||||
[String] $LogEntry
|
||||
)
|
||||
|
||||
$sFullPath = Get-LogName
|
||||
|
||||
$dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss"
|
||||
if ($null -ne $sFullPath -and $sFullPath -ne "") {
|
||||
|
||||
if (!(Test-Path -Path $sFullPath)) {
|
||||
Write-Host "Log path: $sFullPath"
|
||||
$null = New-Item -Path $sFullPath -ItemType File
|
||||
}
|
||||
Add-Content -Path $sFullPath -Value "[$dateTime]. $LogEntry"
|
||||
}
|
||||
Write-Debug "[$dateTime]. $LogEntry"
|
||||
|
||||
}
|
||||
|
||||
function Open-Log {
|
||||
$dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss"
|
||||
Write-Log "***********************************************************************************"
|
||||
Write-Log "* Start of processing: [$dateTime]"
|
||||
Write-Log "***********************************************************************************"
|
||||
}
|
||||
|
||||
function Close-Log {
|
||||
$dateTime = Get-Date -f "yyyy-MM-dd HH:mm:ss"
|
||||
Write-Log "***********************************************************************************"
|
||||
Write-Log "* End of processing: [$dateTime]"
|
||||
Write-Log "***********************************************************************************"
|
||||
}
|
||||
|
||||
function Get-SoftwareName {
|
||||
return [String] "MARTIREFERENCE"
|
||||
}
|
||||
|
||||
|
||||
|
||||
function New-MartiDefinition
|
||||
{
|
||||
|
||||
$oSoftware = [PSCustomObject]@{
|
||||
extension = "software"
|
||||
softwareName = "MartiReference"
|
||||
author = "Meerkat@merebox.com"
|
||||
version = "$script:SoftwareVersion"
|
||||
}
|
||||
|
||||
$publisher = [System.Security.Principal.WindowsIdentity]::GetCurrent().Name
|
||||
|
||||
[System.Collections.ArrayList]$lcustom = @()
|
||||
$lcustom += $oSoftware
|
||||
|
||||
[System.Collections.ArrayList]$lresource = @()
|
||||
|
||||
$oMarti = [PSCustomObject]@{
|
||||
title = ""
|
||||
uid = (New-Guid).ToString()
|
||||
resources = $lresource
|
||||
|
||||
description = ""
|
||||
modified = Get-Date -f "yyyy-MM-ddTHH:mm:ss"
|
||||
tags = @( "document", "marti")
|
||||
publisher = $publisher
|
||||
contactPoint = ""
|
||||
accessLevel = "Confidential"
|
||||
rights = "Restricted"
|
||||
license = ""
|
||||
state = "active"
|
||||
|
||||
describedBy = ""
|
||||
landingPage = ""
|
||||
theme =""
|
||||
|
||||
custom = $lCustom
|
||||
}
|
||||
|
||||
return $oMarti
|
||||
}
|
||||
|
||||
|
||||
function Set-MartiAttribute
|
||||
{
|
||||
Param(
|
||||
[System.Collections.ArrayList] $Attributes,
|
||||
[String] $ACategory,
|
||||
[String] $AName,
|
||||
[String] $AFunction,
|
||||
[String] $Comparison,
|
||||
[String] $Value
|
||||
)
|
||||
|
||||
$matched = $false
|
||||
|
||||
$Attributes | ForEach-Object {
|
||||
|
||||
if ($_.category -eq $ACategory -and $_.name -eq $AName -and $_.function -eq $AFunction) {
|
||||
$matched = $true
|
||||
$_.comparison = $comparison
|
||||
$_.value = $value
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
if (!($matched)) {
|
||||
|
||||
$oAttribute = [PSCustomObject]@{
|
||||
category = $Acategory
|
||||
name = $AName
|
||||
function = $Afunction
|
||||
comparison = $comparison
|
||||
value = $value
|
||||
}
|
||||
|
||||
$Attributes += $oAttribute
|
||||
}
|
||||
|
||||
return $Attributes
|
||||
}
|
||||
|
||||
|
|
@ -1,58 +0,0 @@
|
|||
|
||||
function New-MartiChildItem
|
||||
{
|
||||
Param(
|
||||
[Parameter(Mandatory)][String] $SourceFolder,
|
||||
[String] $Filter ="*",
|
||||
[String] $UrlPath,
|
||||
[switch] $Recurse,
|
||||
[switch] $ExtendAttributes,
|
||||
[switch] $ExcludeHash,
|
||||
[String] $LogPath
|
||||
|
||||
)
|
||||
$script:LogPathName = $LogPath
|
||||
|
||||
Write-Debug "Parameter: LogPath Value: $LogPath "
|
||||
Open-Log
|
||||
Write-Log "Function 'New-MartiDefinition' parameters follow"
|
||||
Write-Log "Parameter: SourceFolder Value: $SourceFolder "
|
||||
Write-Log "Parameter: Filter Value: $Filter "
|
||||
Write-Log "Parameter: Recurse Value: $Recurse "
|
||||
Write-Log "Parameter: ExtendAttributes Value: $ExtendAttributes "
|
||||
Write-Log "Parameter: ExcludeHash Value: $ExcludeHash "
|
||||
Write-Log ""
|
||||
|
||||
$oMarti = New-MartiDefinition
|
||||
$lresource = $oMarti.resources
|
||||
|
||||
$SourceFullName = (Get-Item -Path $SourceFolder).FullName
|
||||
|
||||
Get-ChildItem $SourceFolder -Filter $Filter -Recurse:$Recurse -Force| Where-Object {!$_.PSIsContainer} | ForEach-Object {
|
||||
|
||||
$oResource = New-MartiResource -SourcePath $_.FullName -UrlPath $remoteDirectory -LogPath $LogPath -ExtendAttributes:$ExtendAttributes -ExcludeHash:$ExcludeHash
|
||||
|
||||
if ($null -ne $UrlPath -and $UrlPath -ne "") {
|
||||
$postfixName = $_.FullName.Replace($SourceFullName, "")
|
||||
if ($postfixName[0] -eq "/" -or $postfixName[0] -eq "`\" ){
|
||||
$postfixName = $postfixName.Substring(1, ($postfixName.Length-1))
|
||||
}
|
||||
if ($UrlPath[$UrlPath.Length-1] -eq "/" -or $UrlPath[$UrlPath.Length-1] -eq "`\") {
|
||||
$oResource.url = $UrlPath.Replace("`\", "/") + $postfixName.Replace("`\", "/")
|
||||
} else {
|
||||
$oResource.url = $UrlPath.Replace("`\", "/") + "/" + $postfixName.Replace("`\", "/")
|
||||
}
|
||||
}
|
||||
|
||||
$lresource += $oResource
|
||||
|
||||
}
|
||||
Write-Log "Captured $($lresource.Count) items"
|
||||
$oMarti.resources = $lresource
|
||||
|
||||
Close-Log
|
||||
|
||||
return $oMarti
|
||||
|
||||
}
|
||||
|
||||
|
|
@ -1,112 +0,0 @@
|
|||
|
||||
function New-MartiResource {
|
||||
Param(
|
||||
[Parameter(Mandatory)][String] $SourcePath,
|
||||
[String] $UrlPath = "",
|
||||
[switch] $ExcludeHash,
|
||||
[switch] $ExtendAttributes,
|
||||
[String] $LogPath
|
||||
|
||||
)
|
||||
|
||||
$Global:MartiErrorId = ""
|
||||
$script:LogPathName = $LogPath
|
||||
|
||||
Write-Debug "Parameter: LogPath Value: $LogPath "
|
||||
Open-Log
|
||||
Write-Log "Function 'New-MartiResource' parameters follow"
|
||||
Write-Log "Parameter: UrlPath Value: $UrlPath "
|
||||
Write-Log "Parameter: SourcePath Value: $SourcePath "
|
||||
Write-Log "Parameter: ExcludeHash Value: $ExcludeHash "
|
||||
Write-Log ""
|
||||
|
||||
|
||||
if (Test-Path -Path $SourcePath -PathType Leaf) {
|
||||
|
||||
$item = Get-Item -Path $SourcePath -Force
|
||||
|
||||
Write-Log "Define file $($item.FullName) "
|
||||
|
||||
if ($ExcludeHash) {
|
||||
$hash = $null
|
||||
} else {
|
||||
$hash = New-MartiHash -Algorithm "SHA256" -FilePath $item.FullName
|
||||
}
|
||||
|
||||
$lattribute = Set-MartiResourceAttributes -Path $item.FullName -FileType $item.Extension.Substring(1) -ExtendedAttributes:$ExtendAttributes
|
||||
|
||||
$oResource = [PSCustomObject]@{
|
||||
title = $item.Name.Replace($item.Extension, "")
|
||||
uid = (New-Guid).ToString()
|
||||
documentName = $item.Name
|
||||
issuedDate = Get-Date -f "yyyy-MM-ddTHH:mm:ss"
|
||||
modified = $item.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss")
|
||||
state = "active"
|
||||
author = ""
|
||||
length = $item.Length
|
||||
hash = $hash
|
||||
|
||||
description = ""
|
||||
url = ""
|
||||
version = $version
|
||||
format = $item.Extension.Substring(1)
|
||||
compression = $null
|
||||
encryption = $null
|
||||
|
||||
attributes = $lattribute
|
||||
}
|
||||
|
||||
if ($null -ne $UrlPath -and $UrlPath -ne "") {
|
||||
if ($UrlPath[$UrlPath.Length-1] -eq "/" -or $UrlPath[$UrlPath.Length-1] -eq "\\") {
|
||||
$oResource.url = $UrlPath.Replace("\\", "/") + $item.Name
|
||||
} else {
|
||||
$oResource.url = $UrlPath.Replace("\\", "/") + "/" + $item.Name
|
||||
}
|
||||
}
|
||||
|
||||
} else {
|
||||
$Global:MartiErrorId = "MRI2001"
|
||||
$message = "Document '$SourcePath' not found or is a folder"
|
||||
Write-Log ($message + " " + $Global:MartiErrorId)
|
||||
Close-Log
|
||||
throw $message
|
||||
}
|
||||
Close-Log
|
||||
|
||||
return $oResource
|
||||
|
||||
}
|
||||
|
||||
function New-MartiHash{
|
||||
Param(
|
||||
[Parameter(Mandatory)][String] $Algorithm,
|
||||
[String] $FilePath
|
||||
[String] $Value = ""
|
||||
)
|
||||
|
||||
if ($Value -eq "" -and $FilePath -ne "") {
|
||||
$Value = (Get-FileHash -Path $FilePath -Algorithm $Algorithm).Hash
|
||||
}
|
||||
|
||||
$oHash = [PSCustomObject]@{
|
||||
algo = $Algorithm
|
||||
value = $Value
|
||||
}
|
||||
|
||||
return $oHash
|
||||
}
|
||||
|
||||
function New-Encryption{
|
||||
Param(
|
||||
[Parameter(Mandatory)][String] $Algorithm,
|
||||
[String] $Value
|
||||
|
||||
)
|
||||
|
||||
$oEncryption = [PSCustomObject]@{
|
||||
algo = $Algorithm
|
||||
value = $Value
|
||||
}
|
||||
|
||||
return $oEncryption
|
||||
}
|
||||
|
|
@ -14,7 +14,7 @@ class martiLQ:
|
|||
|
||||
gLogPathName = ""
|
||||
gSoftwareVersion = "0.0.1"
|
||||
gdefault_metaFile = "##marti##.mri"
|
||||
gdefault_metaFile = "##marti##.mti"
|
||||
|
||||
gMartiErrorId = ""
|
||||
gLogOpen = False
|
||||
|
|
@ -126,7 +126,7 @@ class martiLQ:
|
|||
"rights": "Restricted",
|
||||
"license": "",
|
||||
"state": "active",
|
||||
|
||||
"batch": 1,
|
||||
"describedBy": "",
|
||||
"landingPage": "",
|
||||
"theme": "",
|
||||
|
|
@ -157,6 +157,7 @@ class martiLQ:
|
|||
|
||||
item = os.path.basename(SourcePath)
|
||||
self.WriteLog("Define file {}".format(SourcePath))
|
||||
HashAlgorithm = "SHA256"
|
||||
|
||||
try:
|
||||
mtime = os.path.getmtime(SourcePath)
|
||||
|
|
@ -167,7 +168,7 @@ class martiLQ:
|
|||
if ExcludeHash:
|
||||
hash = None
|
||||
else:
|
||||
hash = self.NewMartiHash(Algorithm="SHA256", FilePath=SourcePath, Value="")
|
||||
hash = self.NewMartiHash(Algorithm=HashAlgorithm, FilePath=SourcePath, Value="")
|
||||
|
||||
lattribute = self.SetMartiResourceAttributes(SourcePath, os.path.splitext(SourcePath)[1][1:], ExtendAttributes)
|
||||
|
||||
|
|
@ -561,7 +562,7 @@ class martiLQ:
|
|||
|
||||
return errorCount
|
||||
|
||||
def TestMartiDefinition(self, oMarti, LQSourcePath, LogPath =""):
|
||||
def TestMartiDefinition(self, LQSourcePath, oMarti = None, LogPath =""):
|
||||
|
||||
self.gMartiErrorId = ""
|
||||
self.gLogPathName = LogPath
|
||||
|
|
|
|||
|
|
@ -1,18 +1,17 @@
|
|||
|
||||
. .\source\powershell\New-Marti.ps1
|
||||
. .\source\powershell\Get-Marti.ps1
|
||||
. .\source\powershell\Compress-Marti.ps1
|
||||
. .\source\powershell\MartiLQ.ps1
|
||||
. .\source\powershell\Compress-MartiLQ.ps1
|
||||
|
||||
Write-Host "Test case #1"
|
||||
$oMarti = New-MartiChildItem -SourceFolder ".\docs" -Recurse -UrlPath ".\docs" -Filter "*" -LogPath ".\test\powershell\results\Logs"
|
||||
$oMarti.description = "Sample execution"
|
||||
|
||||
$x = ConvertTo-Json -InputObject $oMarti
|
||||
Set-Content -Path ".\test\powershell\results\marti_test01.mri.json" -Value $x
|
||||
Set-Content -Path ".\test\powershell\results\marti_test01.mti" -Value $x
|
||||
|
||||
Write-Host "Test case #2"
|
||||
$ArchiveFile = ".\test\powershell\results\marti_test02.zip"
|
||||
Compress-Marti -SourceFolder ".\docs" -Filter "*" -LogPath ".\test\powershell\results\Logs" -ArchiveFile $ArchiveFile
|
||||
Compress-MartiLQ -SourceFolder ".\docs" -Filter "*" -LogPath ".\test\powershell\results\Logs" -ArchiveFile $ArchiveFile
|
||||
|
||||
Write-Host "Test case #3"
|
||||
$y = Get-MartiItem -MartiDefintiion $oMarti -Title "ckan" -Format "txt" -LogPath ".\test\powershell\results\Logs"
|
||||
|
|
@ -24,22 +23,22 @@ $oMarti = New-MartiResource -SourcePath ".\docs\ckan.md" -LogPath ".\test\powers
|
|||
$oMarti.description = "Sample execution for ckan"
|
||||
|
||||
$x = ConvertTo-Json -InputObject $oMarti
|
||||
Set-Content -Path ".\test\powershell\results\marti_test02.mri.json" -Value $x
|
||||
Set-Content -Path ".\test\powershell\results\marti_test02json.mti" -Value $x
|
||||
|
||||
$x = ConvertTo-Csv -InputObject $oMarti
|
||||
Set-Content -Path ".\test\powershell\results\marti_test02.mri.csv" -Value $x
|
||||
Set-Content -Path ".\test\powershell\results\marti_test02csv.mti." -Value $x
|
||||
|
||||
$x = ConvertTo-Xml -As String -InputObject $oMarti -Depth 6
|
||||
Set-Content -Path ".\test\powershell\results\marti_test02.mri.xml" -Value $x
|
||||
Set-Content -Path ".\test\powershell\results\marti_test02xml.mti" -Value $x
|
||||
|
||||
$x = ConvertTo-Html -InputObject $oMarti
|
||||
Set-Content -Path ".\test\powershell\results\marti_test02.mri.html" -Value $x
|
||||
Set-Content -Path ".\test\powershell\results\marti_test02html.mti" -Value $x
|
||||
|
||||
Write-Host "Test case #5"
|
||||
$oMarti = New-MartiResource -SourcePath ".\docs\eror" -LogPath ".\test\powershell\results\Logs"
|
||||
$oMarti.description = "Sample execution with error"
|
||||
|
||||
$x = ConvertTo-Json -InputObject $oMarti
|
||||
Set-Content -Path ".\test\powershell\results\marti_test03.mri.json" -Value $x
|
||||
Set-Content -Path ".\test\powershell\results\marti_test03.mti" -Value $x
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@
|
|||
$ckan = Get-Content -Path ".\docs\samples\asic_ckan_api.json" -Raw
|
||||
$oMarti = ConvertFrom-Ckan -InputObject $ckan
|
||||
$x = ConvertTo-Json -InputObject $oMarti
|
||||
Set-Content -Path ".\test\powershell\results\marti_test05.mri.json" -Value $x
|
||||
Set-Content -Path ".\test\powershell\results\marti_test05.mti" -Value $x
|
||||
|
||||
|
||||
$covid_1 = Invoke-WebRequest "https://data.nsw.gov.au/data/api/3/action/package_show?id=793ac07d-a5f4-4851-835c-3f7158c19d15"
|
||||
|
|
@ -18,7 +18,7 @@ $oMarti.tags += "gov"
|
|||
$oMarti.tags += "nsw"
|
||||
$oMarti.publisher = "NSW government (Australia)"
|
||||
$x = ConvertTo-Json -InputObject $oMarti
|
||||
Set-Content -Path ".\test\powershell\results\marti_test06.mri.json" -Value $x
|
||||
Set-Content -Path ".\test\powershell\results\marti_test06.mti" -Value $x
|
||||
|
||||
|
||||
# cases
|
||||
|
|
@ -32,5 +32,5 @@ $oMarti.tags += "gov"
|
|||
$oMarti.tags += "nsw"
|
||||
$oMarti.publisher = "NSW government (Australia)"
|
||||
$x = ConvertTo-Json -InputObject $oMarti
|
||||
Set-Content -Path ".\test\powershell\results\marti_test07.mri.json" -Value $x
|
||||
Set-Content -Path ".\test\powershell\results\marti_test07.mti" -Value $x
|
||||
|
||||
|
|
|
|||
6
tools.md
6
tools.md
|
|
@ -2,9 +2,9 @@
|
|||
|
||||
A number of tools are povided that can be incorporated into your
|
||||
projects that are want to use the metadata transfer reconciliation format
|
||||
(Marti document).
|
||||
(martiLQ document).
|
||||
|
||||
You can combine these in different ways such as:
|
||||
|
||||
* Use Python extract program to generate the Marti document
|
||||
* Use a Java program to reconcile the received data document with the Marti file
|
||||
* Use Python extract program to generate the martiLQ document
|
||||
* Use a Java program to reconcile the received data document with the martiLQ file
|
||||
|
|
|
|||
Loading…
Reference in New Issue