diff --git a/Docs/Advanced.md b/Docs/Advanced.md index 8f6271e..c8021ae 100644 --- a/Docs/Advanced.md +++ b/Docs/Advanced.md @@ -4,22 +4,22 @@ There are various options for using PeterDocs. The following sections will cove ## File Filter -The ``-FileFilter`` parameter allows selection of files tha are to be included into the archive file. +The ``-FileFilter`` parameter allows selection of files that are to be included into the archive file. The parameter only applies to the compress function or buidling the reconciliation file. For example: ```powershell -Compress-Peter -SourceFolder "~/Pictures/" -Secret "c0mpleX%S3cret" -FileFiletr "*.jpg" +Compress-Peter -SourceFolder "~/Pictures/" -Secret "c0mpleX%S3cret" -FileFilter "*.jpg" ``` -will only include files with the extension ".jpg" +will only include files with the extension ".jpg" ```powershell -Compress-Peter -SourceFolder "~/Pictures/" -Secret "c0mpleX%S3cret" -FileFiletr "IMG9*.jpg" +Compress-Peter -SourceFolder "~/Pictures/" -Secret "c0mpleX%S3cret" -FileFilter "IMG9*.jpg" ``` -will only include files with the extension ".jpg" and starting with the characters "IMG90" +will only include files with the extension ".jpg" and starting with the characters "IMG90" ## ReconcileFile @@ -61,4 +61,21 @@ Expand-Peter -RestoreFolder "c:\backup\pictures" -RecipientKey "meerkat@merebox ## LogPath The ``-LogPath`` parameter allows definition of the folder that will contain the -execution log. +execution log. The name of the log file is automatically generated for you and +includes the date. + +## Compression Level + +By setting the Compression level to a value recognized by the 7Zip4Powershell module you can gain more control +of the compresison. The main use case here is for documents that are already compressed and would +not benefit from future compression. To use this feature you need to set the environment variable. + +If all documents are JPEG pictures then setting this value can speed up the compress process +and potentially save a few kilobytes of 7Zip archive size. + +An example for archiving a source folder with already compressed files is: + +```powershell +$env:PETERDOCS_7ZIPLEVEL="None" +Compress-Peter -SourceFolder "./zip_files" -RecipientKey "meerkat@merebox.com" -ArchiveFile .\myarchive.7z -ExcludeHash +``` diff --git a/Docs/AlternateUses.md b/Docs/AlternateUses.md new file mode 100644 index 0000000..ba47086 --- /dev/null +++ b/Docs/AlternateUses.md @@ -0,0 +1,59 @@ +# Alternative Use Cases + +While ``PeterDocs`` has been built with the objective to transfer documents from +one computer to another where the computers are on isolated networks, there are +alternatives uses. + +## Documents on the same network + +You can use ``PeterDocs`` to reconcile files transferred using the Windows +``Robocopy`` command. Robocopy is installed by default on your Windows +system. + +Robocopy does require your source and target folders to be accessible from +the coputer that is executing the command. + +To use ``PeterDocs`` and ``Robocopy`` install PeterDocs from the PowerShell Gallery +and execute the below commands in a PowerShell terminal, changing the values to suit. + +```powershell +New-PeterReconcile -ReconcileFile .\myrobocopy.csv -SourceFolder -ExcludeHash +robocopy /mt /e /z /j /copy:DAT /dcopy:DAT /r:100 /eta /log+:robocopy_run.log /tee +Compare-Peter -ReconcileFile .\myrobocopy.csv -RestoreFolder -ExcludeHash +``` + +The source and destination folders can be network paths i.e. start with \\\\ + +The above robocopy command retries 100 times failed copies. The default is a million with a 30 second +wait time between retries. Probably no a realistic time before failing. + +If you want to verify the HASH for each file copied, then remove the ``-ExcludeHash`` directive. Be +warned that generating a hash on both source and destination will take some time if you +have many files. + +Further information on Robocopy can be found on the internet such as: + +* [https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy](https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy) +* [https://pureinfotech.com/robocopy-transfer-files-fast-network-windows-10/](https://pureinfotech.com/robocopy-transfer-files-fast-network-windows-10/) +* [https://www.techrepublic.com/article/how-to-quickly-back-up-just-your-data-in-windows-10-with-robocopys-multi-threaded-feature/](https://www.techrepublic.com/article/how-to-quickly-back-up-just-your-data-in-windows-10-with-robocopys-multi-threaded-feature/) +* [https://www.youtube.com/watch?v=gTzTeHmKMKw](https://www.youtube.com/watch?v=gTzTeHmKMKw) + +## Picture EXIF data + +You can use ``PeterDocs`` to extract EXIF data from your picture files. To do +this just install PeterDocs from the PowerShell Gallery and execute the +below command in a PowerShell terminal, changing the values to suit. + +```powershell +New-PeterReconcile -ReconcileFile .\mypictures_metadata.csv -SourceFolder -ExcludeHash -IncludeExif +``` + +At the conclusion of the exceution, you will have a file named ``##peter_exif##.csv`` that +contains your pictures metadata. You will also have a CSV file with picture file +general metadata named ``mypictures_metadata.csv`` such as creation time and size. + +Further information on EXIF can be found on the internet such as: + +* [https://en.wikipedia.org/wiki/Exif](https://en.wikipedia.org/wiki/Exif) +* [https://photographylife.com/what-is-exif-data](https://photographylife.com/what-is-exif-data) +* [https://exiftool.org/](https://exiftool.org/) diff --git a/Docs/Install.md b/Docs/Install.md index 9244163..5fbc4ee 100644 --- a/Docs/Install.md +++ b/Docs/Install.md @@ -33,7 +33,10 @@ Please read next the documentation on [creating an archive file](Compress.md) If the computer you wish to use PeterDocs module on does not have Internet access, a common situation for secured servers, then you will need to install the -PeterDocs module manually by following the instructions below: +PeterDocs module manually by following the instructions below. + +Please familiarize yourself with all the instructions before commencing so that you are aware of +all steps to be followed. On a **computer with Internet (PowerShell Gallery) access** do the following steps @@ -43,13 +46,13 @@ On a **computer with Internet (PowerShell Gallery) access** do the following ste $PSVersionTabe.PSVersion ``` -2. Download the module +2. Download the PeterDocs module ```powershell Save-Module -Name PeterDocs -Path C:\Temp ``` -A few directories will be created in C:\Temp. The names of the folders are +A few folders will be created in C:\Temp. The names of the folders are 7Zip4PowerShell, AWS.Tools.Common, AWS.Tools.S3, and PeterDocs 3. Compress the new 7Zip4PowerShell, AWS.Tools.Common, AWS.Tools.S3, PeterDocs folders into a single ZIP file @@ -57,21 +60,29 @@ A few directories will be created in C:\Temp. The names of the folders are On the **computer lacking Internet (PowerShell Gallery) access** do the following steps -1. Run the following command to determine where the ZIP needs to be unpacked +1. First check you have PowerShell version 5 or later + +```powershell +$PSVersionTabe.PSVersion +``` + +2. Run the following command to determine where the ZIP needs to be unpacked ```powershell $env:PSModulePath -Split ";" ``` -2. Take a note of the name of the Windows PowerShell Modules folder that is linked to your account +3. Take a note of the name of the Windows PowerShell Modules folder that is linked to your account. We will install the module linking to you account as you may not be authorised to the global location. -3. Unpack the ZIP contents into the folder name noted above. This should restore the +4. Unpack the ZIP contents into the folder name noted above. This should restore the 7Zip4PowerShell, AWS.Tools.Common, AWS.Tools.S3, PeterDocs folders as a child of the Windows PowerShell Module folder -4. To check the module is installed, run the following command +5. To check the module is installed, run the following command ```powershell Import-Module PeterDocs ``` + +The summary details on PeterDocs should be displayed. diff --git a/Docs/Publish.md b/Docs/Publish.md new file mode 100644 index 0000000..b5bb5b8 --- /dev/null +++ b/Docs/Publish.md @@ -0,0 +1,20 @@ +# Publishing PeterDocs + +To publish the PeterDocs module to the PowerShell Gallery, follow these instructions. + +**Note** Only the author or delegate for PeterDocs will be authorized to perform this action. + +1. Ensure you have incremented the version number in ``PeterDocs.psm1`` and ``PeterDocs.psd1`` +2. Open PowerShell terminal +3. Retrieve the PowerShell Gallery API key and set it +4. Do a Whatif check on the module before publishing +5. Publish the module + +```powershell +$apiKey = "" + +Publish-Module -Path .\PeterDocs\ -NuGetApiKey $apiKey -WhatIf -Verbose + +Publish-Module -Name .\PeterDocs\PeterDocs.psd1 -NuGetApiKey $apiKey + +``` diff --git a/PeterDocs/PeterDocs.psm1 b/PeterDocs/PeterDocs.psm1 index db8d040..f086cfe 100644 --- a/PeterDocs/PeterDocs.psm1 +++ b/PeterDocs/PeterDocs.psm1 @@ -29,7 +29,6 @@ #> $global:default_reconcileFile = "##peter_files##.csv" -$global:default_exifFile = "##peter_exif##.csv" $global:default_metaFile = "##peter##.json" $global:default_errorListFile = Join-Path -Path ".\" -ChildPath "##peter_error_list##.txt" $global:LogPathName = "" @@ -139,7 +138,7 @@ function Test-FilesExist [String] $FileFilter ) - Get-ChildItem $folderName -Recurse | Where-Object {!$_.PSIsContainer} | ForEach-Object { + Get-ChildItem $folderName -Recurse -Force | Where-Object {!$_.PSIsContainer} | ForEach-Object { return $true } @@ -366,7 +365,8 @@ Param( if (!(Test-Path -Path $dirpath )) { $null = New-Item -Path $dirpath -ItemType Directory } - $ExifFile = Join-Path -Path $dirpath -ChildPath $global:default_exifFile + $fname = [System.IO.Path]::GetFileNameWithoutExtension($ReconcileFile) + $ExifFile = Join-Path -Path $dirpath -ChildPath $($fname+"_exif.csv") Write-Log "Generating Exif file '$ExifFile'" Set-Content -Encoding utf8 -Path $ExifFile -Value $(Set-ExifCsvHeader) } @@ -720,14 +720,18 @@ Param( } } - if ($null -ne $env:PETERDOCS_7ZIPLEVEL -and $env:PETERDOCS_7ZIPLEVEL -ne "") { - $7zipLevel = $env:PETERDOCS_7ZIPLEVEL + $getEnvName = $(Get-SoftwareName) + "_7ZIPLEVEL" + if ([System.Environment]::GetEnvironmentVariable($getEnvName) -ne "" -and $null -ne [System.Environment]::GetEnvironmentVariable($getEnvName)) { + $7zipLevel = [System.Environment]::GetEnvironmentVariable($getEnvName) + Write-Log "Compression level set to '$7zipLevel'" } else { $7zipLevel = "Normal" } - if ($null -ne $env:PETERDOCS_ZIPFORMAT -and $env:PETERDOCS_ZIPFORMAT -ne "") { - $7zipFormat = $env:PETERDOCS_ZIPFORMAT + $getEnvName = $(Get-SoftwareName) + "_ZIPFORMAT" + if ([System.Environment]::GetEnvironmentVariable($getEnvName) -ne "" -and $null -ne [System.Environment]::GetEnvironmentVariable($getEnvName)) { + $7zipFormat = [System.Environment]::GetEnvironmentVariable($getEnvName) + Write-Log "Compression format set to '$7zipFormat'" } else { $7zipFormat= "SevenZip" } @@ -808,7 +812,7 @@ Param( { Write-Log "Archive primary folder is '$SourceFolder'" $firstCompress = $true - Get-ChildItem $SourceFolder| ForEach-Object { + Get-ChildItem $SourceFolder -Force | ForEach-Object { $firstCompress = Invoke-SinglePack -ArchiveFolder $_.FullName -ArchiveFile $ArchiveFile -FileFilter $FileFilter -ZipFormat $7zipFormat -FirstCompress $firstCompress -CompressionLevel $7zipLevel } } else { @@ -821,7 +825,7 @@ Param( if ($_ -ne "") { if ($_.EndsWith("*")) { - Get-ChildItem $_ | ForEach-Object { + Get-ChildItem $_ -Force | ForEach-Object { $firstCompress = Invoke-SinglePack -ArchiveFolder $_.FullName -ArchiveFile $ArchiveFile -FileFilter $FileFilter -ZipFormat $7zipFormat -FirstCompress $firstCompress -CompressionLevel $7zipLevel } } else { @@ -884,19 +888,20 @@ Param( $jsonData.Add("Software",$dataItem) $items = New-Object System.Collections.ArrayList - $dataItem = @{"Reconcile"="$ReconcileFile";"Caption"="File listing of archive for reconciliation";} + $dataItem = @{"Topic"="Reconcile";"Reconcile"="$ReconcileFile";"Caption"="File listing of archive for reconciliation";} $null = $items.Add($dataItem) if ($IncludeExif) { - $ExifFile = Join-Path -Path $global:MetadataPathName -ChildPath $global:default_exifFile - $dataItem = @{"Exif"="$ExifFile";"Caption"="Exif information";} + $fname = [System.IO.Path]::GetFileNameWithoutExtension($ReconcileFile) + $ExifFile = Join-Path -Path $global:MetadataPathName -ChildPath $($fname+"_exif.csv") + $dataItem = @{"Topic"="Exif";"Exif"="$ExifFile";"Caption"="Exif information";} $null = $items.Add($dataItem) } - $dataItem = @{"SecretFile"="$SecretFile";"Caption"="File used for complex password storage with asymmetric key";} + $dataItem = @{"Topic"="SecretFile";"SecretFile"="$SecretFile";"Caption"="File used for complex password storage with asymmetric key";} $null = $items.Add($dataItem) - $dataItem = @{"FileFilter"="$FileFilter";"Caption"="File filter used with Compress";} + $dataItem = @{"Topic"="FileFilter";"FileFilter"="$FileFilter";"Caption"="File filter used with Compress";} $null = $items.Add($dataItem) $jsonData.Add("Links",$items) @@ -1774,6 +1779,9 @@ Param( $restoreFileName = $(Join-Path -Path $RestoreFolder -ChildPath $_.FullName) } If (Test-Path -Path $restoreFileName ) { + + $fileItem = Get-Item -Path $restoreFileName -Force + if (!($ExcludeHash)) { if ($_.Hash -ne "") { $targetHash= (Get-FileHash -Path $restoreFileName).Hash @@ -1795,12 +1803,12 @@ Param( } } - if ((Get-Item -Path $restoreFileName).LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastWriteTime) { - Write-Log "LastWrite mismatch for file '$restoreFileName' with target value $((Get-Item -Path $restoreFileName).LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss")) expected $($_.LastWriteTime)" + if ($fileItem.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastWriteTime) { + Write-Log "LastWrite mismatch for file '$restoreFileName' with target value $($fileItem.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss")) expected $($_.LastWriteTime)" $errorCreateCount = $errorCreateCount + 1 $dateTimeValue = [Datetime]::ParseExact($_.LastWriteTime, 'yyyy-MM-ddTHH:mm:ss', $null) - $fileValue = (Get-Item -Path $restoreFileName).LastWriteTime + $fileValue = $fileItem.LastWriteTime $diff = ($dateTimeValue - $fileValue).Seconds # Allow +/- 2 second discrepancy if (($diff.Seconds -lt -2) -or ($diff.Seconds -gt 2)) { @@ -1815,9 +1823,9 @@ Param( $errorFileLogged = $true } } - if ((Get-Item -Path $restoreFileName).Length -ne $_.Length) { + if ($fileItem.Length -ne $_.Length) { $errorCount = $errorCount + 1 - Write-Log "Length mismatch for file '$restoreFileName' with target value $(Get-Item -Path $restoreFileName).Length) expected $($_.Length)" + Write-Log "Length mismatch for file '$restoreFileName' with target value $($fileItem.Length) expected $($_.Length)" if (!$errorFileLogged) { if (!(Test-Path -Path $global:default_errorListFile)) { @@ -1832,12 +1840,12 @@ Param( # Note that last / write access time is not checked by default as it will commonly be changed after restore if ($extendedCheck) { - if ((Get-Item -Path $restoreFileName).CreationTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.CreationTime) { - Write-Log "Creation mismatch for file '$restoreFileName' with target value $((Get-Item -Path $restoreFileName).CreationTime.ToString("yyyy-MM-ddTHH:mm:ss")) expected $($_.CreationTime)" + if ($fileItem.CreationTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.CreationTime) { + Write-Log "Creation mismatch for file '$restoreFileName' with target value $($fileItem.CreationTime.ToString("yyyy-MM-ddTHH:mm:ss")) expected $($_.CreationTime)" $errorCreateCount = $errorCreateCount + 1 $dateTimeValue = [Datetime]::ParseExact($_.CreationTime, 'yyyy-MM-ddTHH:mm:ss', $null) - $fileValue = (Get-Item -Path $restoreFileName).CreationTime + $fileValue = $fileItem.CreationTime $diff = ($dateTimeValue - $fileValue).Seconds # Allow +/- 2 second discrepancy if (($diff.Seconds -lt -2) -or ($diff.Seconds -gt 2)) { @@ -1854,9 +1862,9 @@ Param( } - if ((Get-Item -Path $restoreFileName).LastAccessTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastAccessTime) { + if ($fileItem.LastAccessTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastAccessTime) { $errorCount = $errorCount + 1 - Write-Log "Last access mismatch for file '$restoreFileName' with target value $((Get-Item -Path $restoreFileName).LastAccessTime.ToString("yyyy-MM-ddTHH:mm:ss"))" + Write-Log "Last access mismatch for file '$restoreFileName' with target value $($fileItem.LastAccessTime.ToString("yyyy-MM-ddTHH:mm:ss"))" if (!$errorFileLogged) { if (!(Test-Path -Path $global:default_errorListFile)) { @@ -1867,9 +1875,9 @@ Param( } } - if ((Get-Item -Path $restoreFileName).LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastWriteTime) { + if ($fileItem.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastWriteTime) { $errorCount = $errorCount + 1 - Write-Log "Last write mismatch for file '$restoreFileName' with target value $((Get-Item -Path $restoreFileName).LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss"))" + Write-Log "Last write mismatch for file '$restoreFileName' with target value $($fileItem.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss"))" if (!$errorFileLogged) { if (!(Test-Path -Path $global:default_errorListFile)) { @@ -1882,7 +1890,7 @@ Param( } } - $totalFileSize = $totalFileSize + (Get-Item -Path $restoreFileName).Length + $totalFileSize = $totalFileSize + $fileItem.Length } else { $missingFileCount = $missingFileCount + 1 $errorCount = $errorCount + 1