Wardrobe
/
PeterDocs
mirror of https://github.com/meerkat-manor/PeterDocs.git
3
0
Fork 0
Code Issues Projects Releases Wiki Activity

Enabled processing and compare of hidden files

pull/1/head
Tom Peltonen 2021-08-14 12:32:31 +10:00
parent 832dde3e2a
commit ae5def5aa7
5 changed files with 155 additions and 40 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

29
Docs/Advanced.md
View File

@ -4,22 +4,22 @@ There are various options for using PeterDocs. The following sections will cove
## File Filter ## File Filter
The ``-FileFilter`` parameter allows selection of files tha are to be included into the archive file. The ``-FileFilter`` parameter allows selection of files that are to be included into the archive file.
The parameter only applies to the compress function or buidling the reconciliation file. The parameter only applies to the compress function or buidling the reconciliation file.
For example: For example:
```powershell ```powershell
Compress-Peter -SourceFolder "~/Pictures/" -Secret "c0mpleX%S3cret" -FileFiletr "*.jpg" Compress-Peter -SourceFolder "~/Pictures/" -Secret "c0mpleX%S3cret" -FileFilter "*.jpg"
``` ```
will only include files with the extension ".jpg" will only include files with the extension ".jpg"
```powershell ```powershell
Compress-Peter -SourceFolder "~/Pictures/" -Secret "c0mpleX%S3cret" -FileFiletr "IMG9*.jpg" Compress-Peter -SourceFolder "~/Pictures/" -Secret "c0mpleX%S3cret" -FileFilter "IMG9*.jpg"
``` ```
will only include files with the extension ".jpg" and starting with the characters "IMG90" will only include files with the extension ".jpg" and starting with the characters "IMG90"
## ReconcileFile ## ReconcileFile
@ -61,4 +61,21 @@ Expand-Peter -RestoreFolder "c:\backup\pictures" -RecipientKey "meerkat@merebox
## LogPath ## LogPath
The ``-LogPath`` parameter allows definition of the folder that will contain the The ``-LogPath`` parameter allows definition of the folder that will contain the
execution log. execution log. The name of the log file is automatically generated for you and
includes the date.
## Compression Level
By setting the Compression level to a value recognized by the 7Zip4Powershell module you can gain more control
of the compresison. The main use case here is for documents that are already compressed and would
not benefit from future compression. To use this feature you need to set the environment variable.
If all documents are JPEG pictures then setting this value can speed up the compress process
and potentially save a few kilobytes of 7Zip archive size.
An example for archiving a source folder with already compressed files is:
```powershell
$env:PETERDOCS_7ZIPLEVEL="None"
Compress-Peter -SourceFolder "./zip_files" -RecipientKey "meerkat@merebox.com" -ArchiveFile .\myarchive.7z -ExcludeHash
```

59
Docs/AlternateUses.md 100644
View File

@ -0,0 +1,59 @@
# Alternative Use Cases
While ``PeterDocs`` has been built with the objective to transfer documents from
one computer to another where the computers are on isolated networks, there are
alternatives uses.
## Documents on the same network
You can use ``PeterDocs`` to reconcile files transferred using the Windows
``Robocopy`` command. Robocopy is installed by default on your Windows
system.
Robocopy does require your source and target folders to be accessible from
the coputer that is executing the command.
To use ``PeterDocs`` and ``Robocopy`` install PeterDocs from the PowerShell Gallery
and execute the below commands in a PowerShell terminal, changing the values to suit.
```powershell
New-PeterReconcile -ReconcileFile .\myrobocopy.csv -SourceFolder <Source> -ExcludeHash
robocopy <Source> <Destination> /mt /e /z /j /copy:DAT /dcopy:DAT /r:100 /eta /log+:robocopy_run.log /tee
Compare-Peter -ReconcileFile .\myrobocopy.csv -RestoreFolder <Destination> -ExcludeHash
```
The source and destination folders can be network paths i.e. start with \\\\
The above robocopy command retries 100 times failed copies. The default is a million with a 30 second
wait time between retries. Probably no a realistic time before failing.
If you want to verify the HASH for each file copied, then remove the ``-ExcludeHash`` directive. Be
warned that generating a hash on both source and destination will take some time if you
have many files.
Further information on Robocopy can be found on the internet such as:
* [https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy](https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy)
* [https://pureinfotech.com/robocopy-transfer-files-fast-network-windows-10/](https://pureinfotech.com/robocopy-transfer-files-fast-network-windows-10/)
* [https://www.techrepublic.com/article/how-to-quickly-back-up-just-your-data-in-windows-10-with-robocopys-multi-threaded-feature/](https://www.techrepublic.com/article/how-to-quickly-back-up-just-your-data-in-windows-10-with-robocopys-multi-threaded-feature/)
* [https://www.youtube.com/watch?v=gTzTeHmKMKw](https://www.youtube.com/watch?v=gTzTeHmKMKw)
## Picture EXIF data
You can use ``PeterDocs`` to extract EXIF data from your picture files. To do
this just install PeterDocs from the PowerShell Gallery and execute the
below command in a PowerShell terminal, changing the values to suit.
```powershell
New-PeterReconcile -ReconcileFile .\mypictures_metadata.csv -SourceFolder <Source> -ExcludeHash -IncludeExif
```
At the conclusion of the exceution, you will have a file named ``##peter_exif##.csv`` that
contains your pictures metadata. You will also have a CSV file with picture file
general metadata named ``mypictures_metadata.csv`` such as creation time and size.
Further information on EXIF can be found on the internet such as:
* [https://en.wikipedia.org/wiki/Exif](https://en.wikipedia.org/wiki/Exif)
* [https://photographylife.com/what-is-exif-data](https://photographylife.com/what-is-exif-data)
* [https://exiftool.org/](https://exiftool.org/)

25
Docs/Install.md
View File

@ -33,7 +33,10 @@ Please read next the documentation on [creating an archive file](Compress.md)
If the computer you wish to use PeterDocs module on does not have Internet access, If the computer you wish to use PeterDocs module on does not have Internet access,
a common situation for secured servers, then you will need to install the a common situation for secured servers, then you will need to install the
PeterDocs module manually by following the instructions below: PeterDocs module manually by following the instructions below.
Please familiarize yourself with all the instructions before commencing so that you are aware of
all steps to be followed.
On a **computer with Internet (PowerShell Gallery) access** do the following steps On a **computer with Internet (PowerShell Gallery) access** do the following steps
@ -43,13 +46,13 @@ On a **computer with Internet (PowerShell Gallery) access** do the following ste
$PSVersionTabe.PSVersion $PSVersionTabe.PSVersion
``` ```
2. Download the module 2. Download the PeterDocs module
```powershell ```powershell
Save-Module -Name PeterDocs -Path C:\Temp Save-Module -Name PeterDocs -Path C:\Temp
``` ```
A few directories will be created in C:\Temp. The names of the folders are A few folders will be created in C:\Temp. The names of the folders are
7Zip4PowerShell, AWS.Tools.Common, AWS.Tools.S3, and PeterDocs 7Zip4PowerShell, AWS.Tools.Common, AWS.Tools.S3, and PeterDocs
3. Compress the new 7Zip4PowerShell, AWS.Tools.Common, AWS.Tools.S3, PeterDocs folders into a single ZIP file 3. Compress the new 7Zip4PowerShell, AWS.Tools.Common, AWS.Tools.S3, PeterDocs folders into a single ZIP file
@ -57,21 +60,29 @@ A few directories will be created in C:\Temp. The names of the folders are
On the **computer lacking Internet (PowerShell Gallery) access** do the following steps On the **computer lacking Internet (PowerShell Gallery) access** do the following steps
1. Run the following command to determine where the ZIP needs to be unpacked 1. First check you have PowerShell version 5 or later
```powershell
$PSVersionTabe.PSVersion
```
2. Run the following command to determine where the ZIP needs to be unpacked
```powershell ```powershell
$env:PSModulePath -Split ";" $env:PSModulePath -Split ";"
``` ```
2. Take a note of the name of the Windows PowerShell Modules folder that is linked to your account 3. Take a note of the name of the Windows PowerShell Modules folder that is linked to your account.
We will install the module linking to you account as you may not be authorised to the global location. We will install the module linking to you account as you may not be authorised to the global location.
3. Unpack the ZIP contents into the folder name noted above. This should restore the 4. Unpack the ZIP contents into the folder name noted above. This should restore the
7Zip4PowerShell, AWS.Tools.Common, AWS.Tools.S3, PeterDocs folders 7Zip4PowerShell, AWS.Tools.Common, AWS.Tools.S3, PeterDocs folders
as a child of the Windows PowerShell Module folder as a child of the Windows PowerShell Module folder
4. To check the module is installed, run the following command 5. To check the module is installed, run the following command
```powershell ```powershell
Import-Module PeterDocs Import-Module PeterDocs
``` ```
The summary details on PeterDocs should be displayed.

20
Docs/Publish.md 100644
View File

@ -0,0 +1,20 @@
# Publishing PeterDocs
To publish the PeterDocs module to the PowerShell Gallery, follow these instructions.
**Note** Only the author or delegate for PeterDocs will be authorized to perform this action.
1. Ensure you have incremented the version number in ``PeterDocs.psm1`` and ``PeterDocs.psd1``
2. Open PowerShell terminal
3. Retrieve the PowerShell Gallery API key and set it
4. Do a Whatif check on the module before publishing
5. Publish the module
```powershell
$apiKey = ""
Publish-Module -Path .\PeterDocs\ -NuGetApiKey $apiKey -WhatIf -Verbose
Publish-Module -Name .\PeterDocs\PeterDocs.psd1 -NuGetApiKey $apiKey
```

62
PeterDocs/PeterDocs.psm1
View File

@ -29,7 +29,6 @@
#> #>
$global:default_reconcileFile = "##peter_files##.csv" $global:default_reconcileFile = "##peter_files##.csv"
$global:default_exifFile = "##peter_exif##.csv"
$global:default_metaFile = "##peter##.json" $global:default_metaFile = "##peter##.json"
$global:default_errorListFile = Join-Path -Path ".\" -ChildPath "##peter_error_list##.txt" $global:default_errorListFile = Join-Path -Path ".\" -ChildPath "##peter_error_list##.txt"
$global:LogPathName = "" $global:LogPathName = ""
@ -139,7 +138,7 @@ function Test-FilesExist
[String] $FileFilter [String] $FileFilter
) )
Get-ChildItem $folderName -Recurse | Where-Object {!$_.PSIsContainer} | ForEach-Object { Get-ChildItem $folderName -Recurse -Force | Where-Object {!$_.PSIsContainer} | ForEach-Object {
return $true return $true
} }
@ -366,7 +365,8 @@ Param(
if (!(Test-Path -Path $dirpath )) { if (!(Test-Path -Path $dirpath )) {
$null = New-Item -Path $dirpath -ItemType Directory $null = New-Item -Path $dirpath -ItemType Directory
} }
$ExifFile = Join-Path -Path $dirpath -ChildPath $global:default_exifFile $fname = [System.IO.Path]::GetFileNameWithoutExtension($ReconcileFile)
$ExifFile = Join-Path -Path $dirpath -ChildPath $($fname+"_exif.csv")
Write-Log "Generating Exif file '$ExifFile'" Write-Log "Generating Exif file '$ExifFile'"
Set-Content -Encoding utf8 -Path $ExifFile -Value $(Set-ExifCsvHeader) Set-Content -Encoding utf8 -Path $ExifFile -Value $(Set-ExifCsvHeader)
} }
@ -720,14 +720,18 @@ Param(
} }
} }
if ($null -ne $env:PETERDOCS_7ZIPLEVEL -and $env:PETERDOCS_7ZIPLEVEL -ne "") { $getEnvName = $(Get-SoftwareName) + "_7ZIPLEVEL"
$7zipLevel = $env:PETERDOCS_7ZIPLEVEL if ([System.Environment]::GetEnvironmentVariable($getEnvName) -ne "" -and $null -ne [System.Environment]::GetEnvironmentVariable($getEnvName)) {
$7zipLevel = [System.Environment]::GetEnvironmentVariable($getEnvName)
Write-Log "Compression level set to '$7zipLevel'"
} else { } else {
$7zipLevel = "Normal" $7zipLevel = "Normal"
} }
if ($null -ne $env:PETERDOCS_ZIPFORMAT -and $env:PETERDOCS_ZIPFORMAT -ne "") { $getEnvName = $(Get-SoftwareName) + "_ZIPFORMAT"
$7zipFormat = $env:PETERDOCS_ZIPFORMAT if ([System.Environment]::GetEnvironmentVariable($getEnvName) -ne "" -and $null -ne [System.Environment]::GetEnvironmentVariable($getEnvName)) {
$7zipFormat = [System.Environment]::GetEnvironmentVariable($getEnvName)
Write-Log "Compression format set to '$7zipFormat'"
} else { } else {
$7zipFormat= "SevenZip" $7zipFormat= "SevenZip"
} }
@ -808,7 +812,7 @@ Param(
{ {
Write-Log "Archive primary folder is '$SourceFolder'" Write-Log "Archive primary folder is '$SourceFolder'"
$firstCompress = $true $firstCompress = $true
Get-ChildItem $SourceFolder| ForEach-Object { Get-ChildItem $SourceFolder -Force | ForEach-Object {
$firstCompress = Invoke-SinglePack -ArchiveFolder $_.FullName -ArchiveFile $ArchiveFile -FileFilter $FileFilter -ZipFormat $7zipFormat -FirstCompress $firstCompress -CompressionLevel $7zipLevel $firstCompress = Invoke-SinglePack -ArchiveFolder $_.FullName -ArchiveFile $ArchiveFile -FileFilter $FileFilter -ZipFormat $7zipFormat -FirstCompress $firstCompress -CompressionLevel $7zipLevel
} }
} else { } else {
@ -821,7 +825,7 @@ Param(
if ($_ -ne "") { if ($_ -ne "") {
if ($_.EndsWith("*")) { if ($_.EndsWith("*")) {
Get-ChildItem $_ | ForEach-Object { Get-ChildItem $_ -Force | ForEach-Object {
$firstCompress = Invoke-SinglePack -ArchiveFolder $_.FullName -ArchiveFile $ArchiveFile -FileFilter $FileFilter -ZipFormat $7zipFormat -FirstCompress $firstCompress -CompressionLevel $7zipLevel $firstCompress = Invoke-SinglePack -ArchiveFolder $_.FullName -ArchiveFile $ArchiveFile -FileFilter $FileFilter -ZipFormat $7zipFormat -FirstCompress $firstCompress -CompressionLevel $7zipLevel
} }
} else { } else {
@ -884,19 +888,20 @@ Param(
$jsonData.Add("Software",$dataItem) $jsonData.Add("Software",$dataItem)
$items = New-Object System.Collections.ArrayList $items = New-Object System.Collections.ArrayList
$dataItem = @{"Reconcile"="$ReconcileFile";"Caption"="File listing of archive for reconciliation";} $dataItem = @{"Topic"="Reconcile";"Reconcile"="$ReconcileFile";"Caption"="File listing of archive for reconciliation";}
$null = $items.Add($dataItem) $null = $items.Add($dataItem)
if ($IncludeExif) { if ($IncludeExif) {
$ExifFile = Join-Path -Path $global:MetadataPathName -ChildPath $global:default_exifFile $fname = [System.IO.Path]::GetFileNameWithoutExtension($ReconcileFile)
$dataItem = @{"Exif"="$ExifFile";"Caption"="Exif information";} $ExifFile = Join-Path -Path $global:MetadataPathName -ChildPath $($fname+"_exif.csv")
$dataItem = @{"Topic"="Exif";"Exif"="$ExifFile";"Caption"="Exif information";}
$null = $items.Add($dataItem) $null = $items.Add($dataItem)
} }
$dataItem = @{"SecretFile"="$SecretFile";"Caption"="File used for complex password storage with asymmetric key";} $dataItem = @{"Topic"="SecretFile";"SecretFile"="$SecretFile";"Caption"="File used for complex password storage with asymmetric key";}
$null = $items.Add($dataItem) $null = $items.Add($dataItem)
$dataItem = @{"FileFilter"="$FileFilter";"Caption"="File filter used with Compress";} $dataItem = @{"Topic"="FileFilter";"FileFilter"="$FileFilter";"Caption"="File filter used with Compress";}
$null = $items.Add($dataItem) $null = $items.Add($dataItem)
$jsonData.Add("Links",$items) $jsonData.Add("Links",$items)
@ -1774,6 +1779,9 @@ Param(
$restoreFileName = $(Join-Path -Path $RestoreFolder -ChildPath $_.FullName) $restoreFileName = $(Join-Path -Path $RestoreFolder -ChildPath $_.FullName)
} }
If (Test-Path -Path $restoreFileName ) { If (Test-Path -Path $restoreFileName ) {
$fileItem = Get-Item -Path $restoreFileName -Force
if (!($ExcludeHash)) { if (!($ExcludeHash)) {
if ($_.Hash -ne "") { if ($_.Hash -ne "") {
$targetHash= (Get-FileHash -Path $restoreFileName).Hash $targetHash= (Get-FileHash -Path $restoreFileName).Hash
@ -1795,12 +1803,12 @@ Param(
} }
} }
if ((Get-Item -Path $restoreFileName).LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastWriteTime) { if ($fileItem.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastWriteTime) {
Write-Log "LastWrite mismatch for file '$restoreFileName' with target value $((Get-Item -Path $restoreFileName).LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss")) expected $($_.LastWriteTime)" Write-Log "LastWrite mismatch for file '$restoreFileName' with target value $($fileItem.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss")) expected $($_.LastWriteTime)"
$errorCreateCount = $errorCreateCount + 1 $errorCreateCount = $errorCreateCount + 1
$dateTimeValue = [Datetime]::ParseExact($_.LastWriteTime, 'yyyy-MM-ddTHH:mm:ss', $null) $dateTimeValue = [Datetime]::ParseExact($_.LastWriteTime, 'yyyy-MM-ddTHH:mm:ss', $null)
$fileValue = (Get-Item -Path $restoreFileName).LastWriteTime $fileValue = $fileItem.LastWriteTime
$diff = ($dateTimeValue - $fileValue).Seconds $diff = ($dateTimeValue - $fileValue).Seconds
# Allow +/- 2 second discrepancy # Allow +/- 2 second discrepancy
if (($diff.Seconds -lt -2) -or ($diff.Seconds -gt 2)) { if (($diff.Seconds -lt -2) -or ($diff.Seconds -gt 2)) {
@ -1815,9 +1823,9 @@ Param(
$errorFileLogged = $true $errorFileLogged = $true
} }
} }
if ((Get-Item -Path $restoreFileName).Length -ne $_.Length) { if ($fileItem.Length -ne $_.Length) {
$errorCount = $errorCount + 1 $errorCount = $errorCount + 1
Write-Log "Length mismatch for file '$restoreFileName' with target value $(Get-Item -Path $restoreFileName).Length) expected $($_.Length)" Write-Log "Length mismatch for file '$restoreFileName' with target value $($fileItem.Length) expected $($_.Length)"
if (!$errorFileLogged) { if (!$errorFileLogged) {
if (!(Test-Path -Path $global:default_errorListFile)) { if (!(Test-Path -Path $global:default_errorListFile)) {
@ -1832,12 +1840,12 @@ Param(
# Note that last / write access time is not checked by default as it will commonly be changed after restore # Note that last / write access time is not checked by default as it will commonly be changed after restore
if ($extendedCheck) { if ($extendedCheck) {
if ((Get-Item -Path $restoreFileName).CreationTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.CreationTime) { if ($fileItem.CreationTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.CreationTime) {
Write-Log "Creation mismatch for file '$restoreFileName' with target value $((Get-Item -Path $restoreFileName).CreationTime.ToString("yyyy-MM-ddTHH:mm:ss")) expected $($_.CreationTime)" Write-Log "Creation mismatch for file '$restoreFileName' with target value $($fileItem.CreationTime.ToString("yyyy-MM-ddTHH:mm:ss")) expected $($_.CreationTime)"
$errorCreateCount = $errorCreateCount + 1 $errorCreateCount = $errorCreateCount + 1
$dateTimeValue = [Datetime]::ParseExact($_.CreationTime, 'yyyy-MM-ddTHH:mm:ss', $null) $dateTimeValue = [Datetime]::ParseExact($_.CreationTime, 'yyyy-MM-ddTHH:mm:ss', $null)
$fileValue = (Get-Item -Path $restoreFileName).CreationTime $fileValue = $fileItem.CreationTime
$diff = ($dateTimeValue - $fileValue).Seconds $diff = ($dateTimeValue - $fileValue).Seconds
# Allow +/- 2 second discrepancy # Allow +/- 2 second discrepancy
if (($diff.Seconds -lt -2) -or ($diff.Seconds -gt 2)) { if (($diff.Seconds -lt -2) -or ($diff.Seconds -gt 2)) {
@ -1854,9 +1862,9 @@ Param(
} }
if ((Get-Item -Path $restoreFileName).LastAccessTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastAccessTime) { if ($fileItem.LastAccessTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastAccessTime) {
$errorCount = $errorCount + 1 $errorCount = $errorCount + 1
Write-Log "Last access mismatch for file '$restoreFileName' with target value $((Get-Item -Path $restoreFileName).LastAccessTime.ToString("yyyy-MM-ddTHH:mm:ss"))" Write-Log "Last access mismatch for file '$restoreFileName' with target value $($fileItem.LastAccessTime.ToString("yyyy-MM-ddTHH:mm:ss"))"
if (!$errorFileLogged) { if (!$errorFileLogged) {
if (!(Test-Path -Path $global:default_errorListFile)) { if (!(Test-Path -Path $global:default_errorListFile)) {
@ -1867,9 +1875,9 @@ Param(
} }
} }
if ((Get-Item -Path $restoreFileName).LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastWriteTime) { if ($fileItem.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastWriteTime) {
$errorCount = $errorCount + 1 $errorCount = $errorCount + 1
Write-Log "Last write mismatch for file '$restoreFileName' with target value $((Get-Item -Path $restoreFileName).LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss"))" Write-Log "Last write mismatch for file '$restoreFileName' with target value $($fileItem.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss"))"
if (!$errorFileLogged) { if (!$errorFileLogged) {
if (!(Test-Path -Path $global:default_errorListFile)) { if (!(Test-Path -Path $global:default_errorListFile)) {
@ -1882,7 +1890,7 @@ Param(
} }
} }
$totalFileSize = $totalFileSize + (Get-Item -Path $restoreFileName).Length $totalFileSize = $totalFileSize + $fileItem.Length
} else { } else {
$missingFileCount = $missingFileCount + 1 $missingFileCount = $missingFileCount + 1
$errorCount = $errorCount + 1 $errorCount = $errorCount + 1