Adding certificate backup

install instructions

Docs/Advanced.md

@@ -4,26 +4,26 @@ There are various options for using PeterDocs.  The following sections will cove
 
 ## File Filter
 
-The ``-FileFilter`` parameter allows selection of files tha are to be included into the archive file.
+The ``-FileFilter`` parameter allows selection of files that are to be included into the archive file.
-The parameter only applies to the compress function or buidling the reconciliation file.
+The parameter only applies to the compress function or building the reconciliation file.
 
 For example:
 
 ```powershell
-Compress-Peter -SourceFolder "~/Pictures/" -Secret "c0mpleX%S3cret"  -FileFiletr "*.jpg"
+Compress-Peter -SourceFolder "~/Pictures/" -Secret "c0mpleX%S3cret"  -FileFilter "*.jpg"
 ```
 
-will only include  files with the extension ".jpg"
+will only include files with the extension ".jpg"
 
 ```powershell
-Compress-Peter -SourceFolder "~/Pictures/" -Secret "c0mpleX%S3cret"  -FileFiletr "IMG9*.jpg"
+Compress-Peter -SourceFolder "~/Pictures/" -Secret "c0mpleX%S3cret"  -FileFilter "IMG9*.jpg"
 ```
 
-will only include  files with the extension ".jpg" and starting with the characters "IMG90"
+will only include files with the extension ".jpg" and starting with the characters "IMG90"
 
 ## ReconcileFile
 
-The ``-ReconcileFile`` parameter allows specification of the reocnciliation file if you
+The ``-ReconcileFile`` parameter allows specification of the reconciliation file if you
 wish to select your own name.
 
 For example:
@@ -61,4 +61,21 @@ Expand-Peter -RestoreFolder "c:\backup\pictures"  -RecipientKey "meerkat@merebox
 ## LogPath
 
 The ``-LogPath`` parameter allows definition of the folder that will contain the
-execution log.
+execution log.  The name of the log file is automatically generated for you and
+includes the date.
+
+## Compression Level
+
+By setting the Compression level to a value recognized by the 7Zip4Powershell module you can gain more control
+of the compression.  The main use case here is for documents that are already compressed and would
+not benefit from future compression.  To use this feature you need to set the environment variable.
+
+If all documents are JPEG pictures then setting this value can speed up the compress process
+and potentially save a few kilobytes of 7Zip archive size.
+
+An example for archiving a source folder with already compressed files is:
+
+```powershell
+$env:PETERDOCS_7ZIPLEVEL="None"
+Compress-Peter -SourceFolder "./zip_files" -RecipientKey "meerkat@merebox.com" -ArchiveFile .\myarchive.7z -ExcludeHash
+```

Docs/AlternateUses.md

@@ -0,0 +1,63 @@
+# Alternative Use Cases
+
+While ``PeterDocs`` has been built with the objective to transfer documents from
+one computer to another where the computers are on isolated networks, there are
+alternatives uses.
+
+## Documents on the same network
+
+You can use ``PeterDocs`` to reconcile files transferred using the Windows
+``Robocopy`` command.  Robocopy is installed by default on your Windows
+system.
+
+Robocopy does require your source and target folders to be accessible from
+the computer that is executing the command.
+
+To use ``PeterDocs`` and ``Robocopy`` install PeterDocs from the PowerShell Gallery
+and execute the below commands in a PowerShell terminal, changing the values to suit.
+
+```powershell
+New-PeterReconcile -ReconcileFile .\myrobocopy.csv -SourceFolder <Source> -ExcludeHash 
+robocopy <Source> <Destination> /mt /e /z /j /copy:DAT /dcopy:DAT /r:100 /eta /log+:robocopy_run.log /tee
+Compare-Peter -ReconcileFile .\myrobocopy.csv -RestoreFolder <Destination> -ExcludeHash
+```
+
+The source and destination folders can be network paths i.e. start with \\\\
+
+The above robocopy command retries 100 times failed copies.  The default is a million with a 30 second
+wait time between retries.  Probably not a realistic time before failing.
+
+If you want to verify the HASH for each file copied, then remove the ``-ExcludeHash`` directive.  Be
+warned that generating a hash on both source and destination will take some time if you
+have many files.
+
+Further information on Robocopy can be found on the internet such as:
+
+* [https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy](https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/robocopy)
+* [https://pureinfotech.com/robocopy-transfer-files-fast-network-windows-10/](https://pureinfotech.com/robocopy-transfer-files-fast-network-windows-10/)
+* [https://www.techrepublic.com/article/how-to-quickly-back-up-just-your-data-in-windows-10-with-robocopys-multi-threaded-feature/](https://www.techrepublic.com/article/how-to-quickly-back-up-just-your-data-in-windows-10-with-robocopys-multi-threaded-feature/)
+* [https://www.youtube.com/watch?v=gTzTeHmKMKw](https://www.youtube.com/watch?v=gTzTeHmKMKw)
+
+You need to consider the **security** of the network path when using robocopy.  The copy will use the underlying
+network transport layer and protocol.  If you are not using SMBv3 protocol then the file contents may not
+be secure in transit.
+
+## Picture EXIF data
+
+You can use ``PeterDocs`` to extract EXIF data from your picture files.  To do
+this just install PeterDocs from the PowerShell Gallery and execute the
+below command in a PowerShell terminal, changing the values to suit.
+
+```powershell
+New-PeterReconcile -ReconcileFile .\mypictures_metadata.csv -SourceFolder <Source> -ExcludeHash -IncludeExif
+```
+
+At the conclusion of the execution, you will have a file named ``##peter_exif##.csv`` that
+contains your pictures metadata.  You will also have a CSV file with picture file
+general metadata named ``mypictures_metadata.csv`` such as creation time and size.
+
+Further information on EXIF can be found on the internet such as:
+
+* [https://en.wikipedia.org/wiki/Exif](https://en.wikipedia.org/wiki/Exif)
+* [https://photographylife.com/what-is-exif-data](https://photographylife.com/what-is-exif-data)
+* [https://exiftool.org/](https://exiftool.org/)

Docs/Compress.md

@@ -7,7 +7,7 @@ or cloned to a remote location securely.  The documents can be binary or text do
 including personal photographs or sensitive Microsoft Word documents.
 
 At the remote location a reconciliation can be performed to verify that the documents
-have been recieved and no alteration occurred.
+have been received and no alteration occurred.
 
 ## Why
 
@@ -49,7 +49,7 @@ You can ignore the remaining parameters if you are happy with the defaults.
 
 ## What
 
-The ```Compress-Peter``` compressess the contet of the ```SourceFolder``` and saves the result
+The ```Compress-Peter``` compresses the content of the ```SourceFolder``` and saves the result
 as the encrypted ```ArchiveFile```.  The archive file also contains the reconciliation file
 so that the recipient of the archive is able to reconcile the restore at the remote location.
 
@@ -67,7 +67,7 @@ generate delta archive files.
 
 ## Send Usage
 
-Once the archive file is created you will commonly send or transfer it to anohter
+Once the archive file is created you will commonly send or transfer it to another
 location where it wll be unpacked.
 
 Please read next the documentation on [sending the archive](SendArchive.md)

Docs/Design.md

@@ -3,7 +3,7 @@
 ## Objective
 
 The objective is to provide a simple capability to create a secure
-archive file containiung documents to be restored and reconciled at the destination.
+archive file containing documents to be restored and reconciled at the destination.
 
 The design artefacts must provide all the tools required to pack,
 secure, unpack and reconcile the restored/cloned documents.

Docs/Encryption.md

@@ -22,7 +22,7 @@ encryption.  The secret (or password) needs to be complex and at least
 The complex secret needs to contain lower case letter, upper case letter,
 numeric digit and special symbol.
 
-The secret is used directly on the 7ZIP compresison and you can use the
+The secret is used directly on the 7ZIP compression and you can use the
 secret to decrypt the archive file and check its contents.
 
 Make a note of the secret as you will need it to decrypt the contents.
@@ -34,10 +34,10 @@ file itself.
 ## RecipientKey
 
 The PeterDocs parameter labelled ```RecipientKey``` is used for asymmetric
-keys provided by the Microsft Certificate Manager for encrypting content.
+keys provided by the Microsoft Certificate Manager for encrypting content.
 
 This is the most secure method for transfer as it is secured with the
-certficates.  Using certificates requires:
+certificates.  Using certificates requires:
 
 1. your recipient to send you their public certificate
 2. you to load the public certificate into your Windows Certificate Manager
@@ -49,7 +49,7 @@ contents.
 
 The recipient will need to receive the archive file plus the ```.key```
 file generated by PeterDocs.  Both files must be accessible to the
-recipient for decryptingg the contents.
+recipient for decrypting the contents.
 
 ### Internal process
 
@@ -58,7 +58,7 @@ is saved into the ```.key``` file which is encrypted using the recipients
 public key.
 
 The reason for doing this process is because there are technical limitations
-in encrypting large files using the certficate public keys.  The maximum
+in encrypting large files using the certificate public keys.  The maximum
 size is around 60MB when using public keys.
 
 ### Generating your Certificate
@@ -70,13 +70,31 @@ Manager or executing the below PowerShell command.
     New-SelfSignedCertificate -Subject "CN=PeterDocs" -FriendlyName "PeterDocs"  -KeyDescription "Encryption key for PeterDocs data encipherment" -CertStoreLocation "Cert:\CurrentUser\My" -KeyUsage KeyEncipherment,DataEncipherment, KeyAgreement -Type DocumentEncryptionCert
 ```
 
-To list your current certficates use the below PowerShell command.
+To list your current certificates use the below PowerShell command.
 
 ```powershell
    Get-Childitem -Path "Cert:\CurrentUser\My" -DocumentEncryptionCert     
 ```
 
+**Note**: The certficates from above is assigned to your current logged in user and not the local machine
+
 ### Exporting your Certificate
 
 You will need to export your public key and send it to the person who will generate the archive
 file for you.  That person with your public key will need to import your public key.
+
+On Windows, execute the command ``certmgr`` and export the certificate under "Personal\Certificates" for example.
+
+### Certificate Backup
+
+You should create a backup of your certificate (public and private) in case your local 
+device suffers a failure.  Please secure the backup copy.
+
+You can use the below as a sample code for exporting the default ``PeterDocs`` 
+certificate is give below
+
+```powershell
+   Get-ChildItem -Path "Cert:\CurrentUser\My"  | where{$_.Subject -eq "CN=PeterDocs"}  | Export-Certificate -Type CERT -FilePath C:\Temp\PeterDocs_cert.cer -Force
+```
+
+Change the values and file name to suit your situation.  This file can be imported using the ``certmgr``

Docs/Expand.md

@@ -36,7 +36,7 @@ To expand the archive you will need write access to the ```RestoreFolder``` loca
 ## What
 
 The ```Expand-Peter``` decrypts the archive file and expands the contents into
-the specified restore folder.  It does not peform a reconciliation which is the
+the specified restore folder.  It does not perform a reconciliation which is the
 next step.
 
 ## Reconcile Usage

Docs/Install.md

@@ -1,6 +1,6 @@
 # Install
 
-PeterDocs is a module that can be donwloaded or installed from
+PeterDocs is a module that can be downloaded or installed from
 [PowerShell Gallery](https://xx.com/)
 
 ## Pre-requisites
@@ -28,3 +28,69 @@ need administrator rights.
 ## Compress Usage
 
 Please read next the documentation on [creating an archive file](Compress.md)
+
+## Offline install
+
+If the computer you wish to use PeterDocs module on does not have Internet access,
+a common situation for secured servers, then you will need to install the
+PeterDocs module manually by following the instructions below.
+
+Please familiarize yourself with all the instructions before commencing so that you are aware of
+all steps to be followed.
+
+On a **computer with Internet (PowerShell Gallery) access** do the following steps
+
+1. First check you have PowerShell version 5 or later
+
+```powershell
+$PSVersionTable.PSVersion
+```
+
+2. Download the PeterDocs module
+
+```powershell
+Save-Module -Name PeterDocs -Path C:\Temp
+```
+
+A few folders will be created in C:\Temp.  The names of the folders are
+7Zip4PowerShell, AWS.Tools.Common, AWS.Tools.S3, and PeterDocs
+
+3. Compress the new 7Zip4PowerShell, AWS.Tools.Common, AWS.Tools.S3, PeterDocs folders into a single ZIP file
+4. Copy the ZIP file to the offline computer
+
+On the **computer lacking Internet (PowerShell Gallery) access** do the following steps
+
+1. First check you have PowerShell version 5 or later
+
+```powershell
+$PSVersionTable.PSVersion
+```
+
+2. Run the following command to determine where the ZIP needs to be unpacked
+
+```powershell
+$env:PSModulePath -Split ";"
+```
+
+3. Take a note of the name of the Windows PowerShell Modules folder that is linked to your account.
+We will install the module linking to you account as you may not be authorised to the global location.
+
+4. Unpack the ZIP contents into the folder name noted above.  This should restore the
+7Zip4PowerShell, AWS.Tools.Common, AWS.Tools.S3, PeterDocs folders
+as a child of the Windows PowerShell Module folder
+
+5. To check the module is installed, run the following command
+
+```powershell
+Import-Module PeterDocs
+```
+
+The summary details on PeterDocs should be displayed.
+
+## Uninstall
+
+To uninstall ``PeterDocs``, execute the following PowerShell command
+
+```powershell
+    UnInstall-Module -Name PeterDocs 
+```

Docs/Publish.md

@@ -0,0 +1,20 @@
+# Publishing PeterDocs
+
+To publish the PeterDocs module to the PowerShell Gallery, follow these instructions.
+
+**Note** Only the author or delegate for PeterDocs will be authorized to perform this action.
+
+1. Ensure you have incremented the version number in ``PeterDocs.psm1`` and ``PeterDocs.psd1``
+2. Open PowerShell terminal
+3. Retrieve the PowerShell Gallery API key and set it
+4. Do a Whatif check on the module before publishing
+5. Publish the module
+
+```powershell
+$apiKey = ""
+
+Publish-Module -Path .\PeterDocs\ -NuGetApiKey $apiKey -WhatIf -Verbose
+
+Publish-Module -Name .\PeterDocs\PeterDocs.psd1 -NuGetApiKey $apiKey
+
+```

Docs/ReceiveArchive.md

@@ -3,13 +3,13 @@
 ## Why
 
 Once the archive file is sent, you need to download it from its intermediate
-location if the source and destination locations are not directky connected.
+location if the source and destination locations are not directly connected.
 
 You can user other tools you have available to download the archive file.
 
 ## When
 
-The archive is received after it sent.  The assunmption is that cloud storage
+The archive is received after it sent.  The assumption is that cloud storage
 is being used as an intermediary.
 
 ## How
@@ -36,8 +36,8 @@ The ```SourcePath``` is specified as follows:
 The "s3" prefix is to download from AWS S3.  The "b2" prefix
 is to download from Backblaze.
 
-If you are dowloading from AWS you can specify the AWS profile name
+If you are downloading from AWS you can specify the AWS profile name
-in parameter ```SourceProfile```.  In this situtation the profile
+in parameter ```SourceProfile```.  In this situation the profile
 needs to exist in the AWS credentials on your local device and user profile.
 
 If you are downloading from Backblaze you specify the ```AccountId``` and the
@@ -57,7 +57,7 @@ use other tools to download the archive file and the key file.
 
 The function will not expand or reconcile the restore at the destination.
 
-Please ensure you have sufficent storage to accomodate the local copy of the
+Please ensure you have sufficient storage to accommodate the local copy of the
 archive and space to unpack it.
 
 ## Expand Usage

Docs/Reconcile.md

@@ -4,7 +4,7 @@ A reconcile file is generated as part of the Compress process and packed with th
 
 ## Why
 
-When transferring or cloning documenmts to another location, you will want to
+When transferring or cloning documents to another location, you will want to
 verify that the same documents have been restored unaltered at the destination.
 
 ## When
@@ -44,7 +44,7 @@ The document last update and time is not checked because the value
 will reflect the date and time of restore.
 
 The reconciliation summary is displayed in the terminal and the log
-wil lhave more information.
+will have more information.
 
 If any errors are listed, please investigate the discrepancy.
 

Docs/SendArchive.md

@@ -43,7 +43,7 @@ The "s3" prefix is to upload to AWS S3.  The "b2" prefix
 is to upload to Backblaze.
 
 If you are uploading to AWS you can specify the AWS profile name
-in parameter ```TargetProfile```.  In this situtation the profile
+in parameter ```TargetProfile```.  In this situation the profile
 needs to exist in the AWS credentials on your local device and user profile.
 
 If you are uploading to Backblaze you specify the ```AccountId``` and the

Docs/Usage.md

@@ -0,0 +1,78 @@
+# Usage
+
+## Usage PeterDocs
+
+The syntax is for each function is below:
+
+```powershell
+Compress-Peter
+    -SourceFolder <string>    
+    [-RecipientKey <string>] | [-SecretKey <string>]
+    [-ArchiveFile <string>]
+    [-ReconcileFile <string>]
+    [-FileFilter <string>]
+    [-SecretFile <string>]
+    [-ExcludeHash]
+    [-IncludeExif]
+    [-RootFolder  <string>]
+    [-VolumeSize <string>]
+    [-LogPath <string>]
+    [<CommonParameters>]
+
+Expand-Peter
+    -ArchiveFile <string>
+    -RestoreFolder <string>    
+    [-RecipientKey <string>] | [-SecretKey <string>]
+    [-SecretFile <string>]
+    [-LogPath <string>]
+    [<CommonParameters>]
+
+New-PeterReconcile
+    -SourceFolder <string>    
+    -ReconcileFile <string>
+    [-RootFolder <string>]
+    [-FileFilter <string>]
+    [-ProcessFileCount <long>]
+    [-ExcludeHash]
+    [-IncludeExif]
+    [-Feedback]
+    [-LogPath <string>]
+    [<CommonParameters>]
+
+Compare-Peter
+    -RestoreFolder <string>    
+    [-ReconcileFile <string>]
+    [-RootFolder <string>}
+    [-ExcludeHash]
+    [-ExtendedCheck]
+    [-LogPath <string>]
+    [<CommonParameters>]
+
+```
+
+## Usage PeterTask
+
+For using the PeterTask, syntax is as simple as shown below. Behind the scenes the
+PeterTask calls the functions above:
+
+```powershell
+
+PeterTask
+    -Task {Compress, Expand, Compare, NewReconcile, Put, Get, ArchiveInformation}
+    -Path <string>
+    [-RecipientKey <string>] | [-SecretKey <string>]
+    [-ArchiveFile  <string>]
+    [-RootFolder  <string>]
+    [-FileFilter <string>]
+    [-ReconcileFile <string>]
+    [-SecretFile <string>]
+    [-CloudProfile <string>]
+    [-ExcludeHash]
+    [-IncludeExif]
+    [-LogPath <string>]
+    [<CommonParameters>]
+
+```
+
+**NOTE** Please do not use the VolumeSize parameter as it will fail due to a current
+issue with the underlying 7zip4 Powershell module.

PeterDocs/PeterDocs.psm1

@@ -29,11 +29,12 @@
 #>
 
 $global:default_reconcileFile = "##peter_files##.csv"
-$global:default_exifFile = "##peter_exif##.csv"
 $global:default_metaFile = "##peter##.json"
+$global:default_errorListFile = Join-Path -Path ".\" -ChildPath "##peter_error_list##.txt"
 $global:LogPathName = ""
 $global:MetadataPathName = Join-Path -Path ".\" -ChildPath ".peter-metadata"
-$global:Version = "0.3"
+$global:Version = "0.32"
+
 
 function Open-Log {
     
@@ -44,10 +45,7 @@ function Open-Log {
 
 }
 
-function Write-Log {
+function Get-LogName {
-    param(
-        [String] $LogEntry
-    )
 
     $date = Get-Date -f "yyyy-MM-dd"
     
@@ -55,13 +53,23 @@ function Write-Log {
     {
         $global:LogPathName = Join-Path -Path ".\" -ChildPath "Logs"
     }
-    $logName = $(Get-SoftwareName) + "_$date.log"
-    $sFullPath = Join-Path -Path $global:LogPathName -ChildPath $logName 
 
     if (!(Test-Path -Path $global:LogPathName)) {
         $null = New-Item -Path $global:LogPathName -ItemType Directory
     }
 
+    $logName = $(Get-SoftwareName) + "_$date.log"
+
+    return Join-Path -Path $global:LogPathName -ChildPath $logName 
+}
+
+function Write-Log {
+    param(
+        [String] $LogEntry
+    )
+
+    $sFullPath = Get-LogName 
+
     if (!(Test-Path -Path $sFullPath)) {
         Write-Host "Log path: $sFullPath"
         $null = New-Item -Path $sFullPath -ItemType File
@@ -130,7 +138,7 @@ function Test-FilesExist
         [String] $FileFilter
     ) 
 
-    Get-ChildItem $folderName -Recurse | Where-Object {!$_.PSIsContainer} | ForEach-Object {
+    Get-ChildItem $folderName -Recurse -Force | Where-Object {!$_.PSIsContainer} | ForEach-Object {
         return $true
     }
     
@@ -144,15 +152,15 @@ function Get-ConvenientFileSize
     ) 
  
     
-    if ($totalFileSize -ge 1TB) {
+    if ($Size -ge 1TB) {
         $totalRightLabel = "TB"
         $totalFileXbytes = [math]::Round(($size / 1TB), 2)        
     } else {
-        if ($totalFileSize -ge 1GB) {
+        if ($Size -ge 1GB) {
             $totalRightLabel = "GB"
             $totalFileXbytes = [math]::Round(($size / 1GB), 2)        
         } else { 
-            if ($totalFileSize -ge 1MB) {
+            if ($Size -ge 1MB) {
                 $totalRightLabel = "MB"
                 $totalFileXbytes = [math]::Round(($size / 1MB), 2)        
             } else {
@@ -166,6 +174,60 @@ function Get-ConvenientFileSize
 }
 
 
+function Get-ReverseConvenientFileSize
+{
+    Param( 
+        [Parameter(Mandatory)]
+        [String] $Size
+    ) 
+
+    if ($null -eq $Size -or $size -eq "") {
+        return ""
+    }
+    
+    $found = $false
+    if ($size -like "*TB") {
+        $found = $true
+        $totalSize = [int]::Parse($size.Substring(0, ($size.Length-2))) * 1TB
+    }
+    if ($size -like "*T") {
+        $found = $true
+        $totalSize = [int]::Parse($size.Substring(0, ($size.Length-1))) * 1TB
+    }
+    if ($size -like "*GB") {
+        $found = $true
+        $totalSize = [int]::Parse($size.Substring(0, ($size.Length-2))) * 1GB
+    }
+    if ($size -like "*G") {
+        $found = $true
+        $totalSize = [int]::Parse($size.Substring(0, ($size.Length-1))) * 1GB
+    }
+    if ($size -like "*MB") {
+        $found = $true
+        $totalSize = [int]::Parse($size.Substring(0, ($size.Length-2))) * 1MB
+    }
+    if ($size -like "*M") {
+        $found = $true
+        $totalSize = [int]::Parse($size.Substring(0, ($size.Length-1))) * 1MB
+    }
+    if ($size -like "*KB") {
+        $found = $true
+        $totalSize = [int]::Parse($size.Substring(0, ($size.Length-2))) * 1KB
+    }
+    if ($size -like "*K") {
+        $found = $true
+        $totalSize = [int]::Parse($size.Substring(0, ($size.Length-1))) * 1KB
+    }
+    if (!$found)
+    {
+        $found = $true
+        $totalSize = [int]::Parse($size)
+    } 
+
+    return $totalSize
+}
+
+
 <#
  .Synopsis
   Creates a CSV file for reconciliation at the destination.
@@ -284,16 +346,14 @@ Param(
     if ($SourceFolder.StartsWith("@")) {
         If (!(Test-Path -Path $SourceFolder.Substring(1) )) {    
             Write-Log "File '$($SourceFolder.Substring(1))' does not exist"
-            Write-Host "File '$($SourceFolder.Substring(1))' does not exist" -ForegroundColor Red
             Close-Log
-            Exit
+            Throw "File '$($SourceFolder.Substring(1))' does not exist"
         }
     } else {
         If (!(Test-Path -Path $SourceFolder )) {    
             Write-Log "Folder '$SourceFolder' does not exist"
-            Write-Host "Folder '$SourceFolder' does not exist" -ForegroundColor Red
             Close-Log
-            Exit
+            Throw "Folder '$SourceFolder' does not exist"
         }
     }
 
@@ -305,7 +365,8 @@ Param(
         if (!(Test-Path -Path $dirpath )) {
             $null = New-Item -Path $dirpath -ItemType Directory
         }
-        $ExifFile = Join-Path -Path $dirpath -ChildPath $global:default_exifFile
+        $fname = [System.IO.Path]::GetFileNameWithoutExtension($ReconcileFile)
+        $ExifFile = Join-Path -Path $dirpath -ChildPath $($fname+"_exif.csv")
         Write-Log "Generating Exif file '$ExifFile'"
         Set-Content  -Encoding utf8  -Path $ExifFile  -Value $(Set-ExifCsvHeader)
     }
@@ -337,7 +398,7 @@ Param(
                     Write-Host "Folder/file '$($_)' does not exist" -ForegroundColor Red
                 }
                 else {
-                    Get-ChildItem $_ -Filter $FileFilter -Recurse | Where-Object {!$_.PSIsContainer} | ForEach-Object {
+                    Get-ChildItem $_ -Filter $FileFilter -Recurse -Force | Where-Object {!$_.PSIsContainer} | ForEach-Object {
 
                         $totalFilecount = $totalFileCount + 1
                         $totalFileSize = $totalFileSize + $_.Length 
@@ -354,7 +415,21 @@ Param(
                         if ($ExcludeHash) {
                             $sourceHash = ""
                         } else {
-                            $sourceHash = (Get-FileHash -Path $_.FullName).Hash
+                            $fullN = $_.FullName
+                            try 
+                            {
+                                $sourceHash = (Get-FileHash -Path $fullN -ErrorAction Stop).Hash
+                            } 
+                            catch 
+                            {
+                                $sourceHash = ""
+                                $errorDetail = $_.Exception.Message
+                                Write-Log "Error in hash with message: $errorDetail"
+                                Write-Log "Hash creation failed for '$fullN'. Maybe the file is open in another process"
+                                Close-Log
+                                Throw "Hash creation failed for '$fullN'. Maybe the file is open in another process"
+                            }
+
                         }
                         $record = '"'+$_.FullName.Replace($RootFolder, "")+'","'+$_.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss")+'"'
                         $record = $record + ',"'+$_.CreationTime.ToString("yyyy-MM-ddTHH:mm:ss")+'","'+$_.LastAccessTime.ToString("yyyy-MM-ddTHH:mm:ss")+'"'
@@ -376,7 +451,7 @@ Param(
         }
 
     } else {
-        Get-ChildItem $SourceFolder -Filter $FileFilter -Recurse | Where-Object {!$_.PSIsContainer} | ForEach-Object {
+        Get-ChildItem $SourceFolder -Filter $FileFilter -Recurse -Force| Where-Object {!$_.PSIsContainer} | ForEach-Object {
 
             $totalFilecount = $totalFileCount + 1
             $totalFileSize = $totalFileSize + $_.Length 
@@ -393,7 +468,21 @@ Param(
             if ($ExcludeHash) {
                 $sourceHash = ""
             } else {
-                $sourceHash = (Get-FileHash -Path $_.FullName).Hash
+                $fullN = $_.FullName
+                try 
+                {
+                    $sourceHash = (Get-FileHash -Path $fullN  -ErrorAction Stop).Hash
+                } 
+                catch 
+                {
+                    $sourceHash = ""
+                    $errorDetail = $_.Exception.Message
+                    Write-Log "Error in hash with message: $errorDetail"
+                    Write-Log "Hash creation failed for '$fullN'. Maybe the file is open in another process"
+                    Close-Log
+                    Throw "Hash creation failed for '$fullN'. Maybe the file is open in another process"
+                }
+            
             }
             $record = '"'+$_.FullName.Replace($RootFolder, "")+'","'+$_.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss")+'"'
             $record = $record + ',"'+$_.CreationTime.ToString("yyyy-MM-ddTHH:mm:ss")+'","'+$_.LastAccessTime.ToString("yyyy-MM-ddTHH:mm:ss")+'"'
@@ -428,6 +517,9 @@ function Invoke-SinglePack
         [Parameter(Mandatory)][String] $ArchiveFolder,
         [Parameter(Mandatory)][String] $ArchiveFileName,
         [String] $FileFilter,
+        [String] $ZipFormat = "SevenZip",
+        [String] $CompressionLevel = "Normal",
+        [String] $VolumeSize = "-1",
         [Boolean] $FirstCompress
     ) 
 
@@ -438,14 +530,14 @@ function Invoke-SinglePack
     if (Test-FilesExist -FolderName $ArchiveFolder -FileFilter $FileFilter) {
         try {
             if ($FirstCompress) {
-                Compress-7Zip -Path $ArchiveFolder -ArchiveFileName $ArchiveFileName -Format SevenZip -PreserveDirectoryRoot -Filter $FileFilter   
+                Compress-7Zip -Path $ArchiveFolder -ArchiveFileName $ArchiveFileName -Format $ZipFormat -CompressionLevel $7zipLevel -PreserveDirectoryRoot -Filter $FileFilter -Volume (Get-ReverseConvenientFileSize $VolumeSize) 
             } else {
-                Compress-7Zip -Path $ArchiveFolder -ArchiveFileName $ArchiveFileName -Format SevenZip -PreserveDirectoryRoot -Filter $FileFilter -Append    
+                Compress-7Zip -Path $ArchiveFolder -ArchiveFileName $ArchiveFileName -Format $ZipFormat -CompressionLevel $7zipLevel -PreserveDirectoryRoot -Filter $FileFilter -Volume (Get-ReverseConvenientFileSize $VolumeSize) -Append    
             }
             $FirstCompress = $false
         } catch {
             Write-Log "Compress error with folder/file '$ArchiveFolder'.  See any previous errors.  $Error"
-            Write-Host "Compress error with folder/file '$ArchiveFolder'.  See any previous errors.  $Error" -ForegroundColor Red
+            Throw "Compress error with folder/file '$ArchiveFolder'. Please refer to log '$(Get-LogName)' for details" 
         }
     } else {
         Write-Log "Empty folder/file '$ArchiveFolder'"
@@ -575,8 +667,22 @@ function Invoke-SinglePack
   The following environment variables are supported:
   - PETERDOCS_RECIPIENTKEY
   - PETERDOCS_SECRETKEY
+  - PETERDOCS_7ZIPLEVEL
+  - PETERDOCS_ZIPFORMAT
   - PETERDOCS_LOGPATH
 
+  The environment variable _PETERDOCS_7ZIPLEVEL_ is used to override the default
+  7ZIP compression level setting.  This is useful if you already for example when 
+  you know that the binary files are compressed or have no benefit in compression
+  saving time. For example
+
+  ```PETERDOCS_7ZIPLEVEL=None```
+
+  The environment variable PETERDOCS_ZIPFORMAT is used to override the default
+  7ZIP format value.  Using this option may invalidate other settings. For example
+
+  ```PETERDOCS_ZIPFORMAT=SevenZip```
+
  .Example
    # Pack and encrypt all files in folder ".\transferpack\" using a private-public key
    # A default archive named file is created which includes a date and time in the name.
@@ -598,6 +704,7 @@ Param(
     [switch] $ExcludeHash,
     [switch] $IncludeExif,
     [String] $RootFolder,
+    [String] $VolumeSize = "-1",
     [String] $LogPath
 
 ) 
@@ -617,31 +724,46 @@ Param(
     } else {
         Write-Log "Parameter: SecretKey   Value: ************** "
     }
+
     Write-Log "Parameter: ArchiveFile   Value: $ArchiveFile "
     Write-Log "Parameter: ReconcileFile   Value: $ReconcileFile "
     Write-Log "Parameter: FileFilter   Value: $FileFilter "
     Write-Log "Parameter: SecretFile   Value: $SecretFile "
     Write-Log "Parameter: ExcludeHash   Value: $ExcludeHash "
     Write-Log "Parameter: IncludeExif   Value: $IncludeExif "
+    Write-Log "Parameter: VolumeSize   Value: $VolumeSize "
     Write-Log "Parameter: LogPath   Value: $LogPath "
     Write-Log ""
 
     if ($SourceFolder.StartsWith("@")) {
         If (!(Test-Path -Path $SourceFolder.Substring(1) )) {    
             Write-Log "File '$($SourceFolder.Substring(1))' does not exist"
-            Write-Host "File '$($SourceFolder.Substring(1))' does not exist" -ForegroundColor Red
             Close-Log
-            Exit
+            Throw "File '$($SourceFolder.Substring(1))' does not exist"
         }
     } else {
         If (!(Test-Path -Path $SourceFolder )) {    
             Write-Log "Folder '$SourceFolder' does not exist"
-            Write-Host "Folder '$SourceFolder' does not exist" -ForegroundColor Red
             Close-Log
-            Exit
+            Throw "Folder '$SourceFolder' does not exist"
         }
     }
 
+    $getEnvName = $(Get-SoftwareName) + "_7ZIPLEVEL"
+    if ([System.Environment]::GetEnvironmentVariable($getEnvName) -ne "" -and $null -ne [System.Environment]::GetEnvironmentVariable($getEnvName)) {
+        $7zipLevel = [System.Environment]::GetEnvironmentVariable($getEnvName)
+        Write-Log "Compression level set to '$7zipLevel'"
+    } else {
+        $7zipLevel = "Normal"
+    }
+
+    $getEnvName = $(Get-SoftwareName) + "_ZIPFORMAT"
+    if ([System.Environment]::GetEnvironmentVariable($getEnvName) -ne "" -and $null -ne [System.Environment]::GetEnvironmentVariable($getEnvName)) {
+        $7zipFormat = [System.Environment]::GetEnvironmentVariable($getEnvName)
+        Write-Log "Compression format set to '$7zipFormat'"
+    } else {
+        $7zipFormat= "SevenZip"
+    }
 
     if ($RecipientKey -eq "") {
         $getEnvName = $(Get-SoftwareName) + "_RECIPIENTKEY"
@@ -659,17 +781,15 @@ Param(
 
     if (($RecipientKey -eq "") -and ($SecretKey -eq "")) {
         Write-Log "Recipient Key or Secret Key required for packing" 
-        Write-Host "Recipient Key or Secret Key required for packing"  -ForegroundColor Red
         Close-Log
-        return
+        Throw "Recipient Key or Secret Key required for packing"
     } 
     
     if ($RootFolder -eq "") {
         if ($SourceFolder.EndsWith("*")) {
             Write-Log "Root folder required for packing when using wild card for Source Folder" 
-            Write-Host "Root folder required for packing when using wild card for Source Folder"  -ForegroundColor Red
             Close-Log
-            return
+            Throw "Root folder required for packing when using wild card for Source Folder"
         } else {
             $RootFolder = $SourceFolder
         }
@@ -679,19 +799,22 @@ Param(
         $ArchiveFile = $(Get-SoftwareName) + $(Get-Date -Format "yyyyMMdd_HHmm") + ".7z"
     }
 
-    if ($SecretKey -eq "") {
+    if ($SecrettKey -eq "") {
-        if ($SecretFile -eq "")
+        if ($SecretFile -eq "") {
-        {
             $SecretFile = $ArchiveFile + ".key"
         }
         $secret = New-RandomPassword -Length 80
         Protect-CmsMessage -To $recipientKey -OutFile $SecretFile -Content $secret 
+        if (!(Test-Path -Path $SecretFile)) {
+            Write-Log "Secret file '$SecretFile' not found" 
+            Close-Log
+            Throw "Secret file '$SecretFile' not found"
+        }
     } else {
         if (!(Test-PasswordQuality -TestPassword $SecretKey)) {
             Write-Log "Secret Key does not meet complexity rules" 
-            Write-Host "Secret Key does not meet complexity rules"  -ForegroundColor Red
             Close-Log
-            return    
+            Throw "Secret Key does not meet complexity rules"
         }
         $secret = $SecretKey
     }
@@ -700,25 +823,22 @@ Param(
     Write-Log "Saving folders/files to archive file '$ArchiveFile'"
     Write-Host "Saving folders/files to archive file '$ArchiveFile'"
 
-    if ($ReconcileFile -eq "")
+    if ($ReconcileFile -eq "") {
-    {
         if (!(Test-Path -Path $global:MetadataPathName)) {
             $null = New-Item -Path $global:MetadataPathName -ItemType Directory
         }
         $ReconcileFile = Join-Path -Path $global:MetadataPathName -ChildPath $default_reconcileFile
     }
 
-    if ($FileFilter -eq "")
+    if ($FileFilter -eq "") {
-    {
         $FileFilter = "*"
     }
 
-    if ($SourceFolder.EndsWith("*"))
+    if ($SourceFolder.EndsWith("*")) {
-    {
         Write-Log "Archive primary folder is '$SourceFolder'"
         $firstCompress = $true
-        Get-ChildItem $SourceFolder| ForEach-Object {
+        Get-ChildItem $SourceFolder -Force | ForEach-Object {
-            $firstCompress = Invoke-SinglePack -ArchiveFolder $_.FullName -ArchiveFile $ArchiveFile -FileFilter $FileFilter -FirstCompress $firstCompress
+            $firstCompress = Invoke-SinglePack -ArchiveFolder $_.FullName -ArchiveFile $ArchiveFile -FileFilter $FileFilter -ZipFormat $7zipFormat -FirstCompress $firstCompress  -CompressionLevel $7zipLevel
         }
     } else {
         if ($SourceFolder.StartsWith("@")) {
@@ -730,8 +850,8 @@ Param(
                 if ($_ -ne "") {
 
                     if ($_.EndsWith("*")) {
-                        Get-ChildItem $_ | ForEach-Object {
+                        Get-ChildItem $_  -Force | ForEach-Object {
-                            $firstCompress = Invoke-SinglePack -ArchiveFolder $_.FullName -ArchiveFile $ArchiveFile -FileFilter $FileFilter -FirstCompress $firstCompress
+                            $firstCompress = Invoke-SinglePack -ArchiveFolder $_.FullName -ArchiveFile $ArchiveFile -FileFilter $FileFilter -ZipFormat $7zipFormat -FirstCompress $firstCompress -CompressionLevel $7zipLevel
                         }
                     } else {
                 
@@ -740,7 +860,7 @@ Param(
                             Write-Host "Folder/file '$($_)' does not exist" -ForegroundColor Red
                         }
                         else {
-                            $firstCompress = Invoke-SinglePack -ArchiveFolder $_ -ArchiveFile $ArchiveFile -FileFilter $FileFilter -FirstCompress $firstCompress
+                            $firstCompress = Invoke-SinglePack -ArchiveFolder $_ -ArchiveFile $ArchiveFile -FileFilter $FileFilter -ZipFormat $7zipFormat -FirstCompress $firstCompress  -CompressionLevel $7zipLevel
                         }
                     }
                 }
@@ -748,27 +868,38 @@ Param(
         } else {
             Write-Log "Archive folder '$SourceFolder'"
             Write-Host "Archive folder '$SourceFolder'"
-            Compress-7Zip -Path $SourceFolder -ArchiveFileName $ArchiveFile -Format SevenZip -Filter $FileFilter    
+            Compress-7Zip -Path $SourceFolder -ArchiveFileName $ArchiveFile  -Format $7zipFormat -CompressionLevel $7zipLevel -Filter $FileFilter -Volume (Get-ReverseConvenientFileSize $VolumeSize)
         }
     }
 
+    $multiVolume = $false
     If (!(Test-Path -Path $ArchiveFile )) {    
-        Write-Log "Archive file '$ArchiveFile' was not created.  See any previous errors"
+        # Check for volume 
-        Write-Host "Archive file '$ArchiveFile' was not created.  See any previous errors" -ForegroundColor Red
+        If (!(Test-Path -Path $($ArchiveFile+".001") )) {    
-        Close-Log
+            Write-Log "Archive file '$ArchiveFile' was not created.  See any previous errors"
-        Exit
+            Close-Log
+            Throw "Archive file '$ArchiveFile' was not created. Please refer to log '$(Get-LogName)' for details"
+        } else {
+            $multiVolume = $true
+            Write-Log "Multi volume archive file '$ArchiveFile' created."
+            Write-Host "Multi volume archive file '$ArchiveFile' created."
+        }
     }
 
-    
+    if ($multiVolume) {
-    $archiveInfo = Get-7ZipInformation -ArchiveFileName $ArchiveFile
+        $fullZipName = (Get-Item $($ArchiveFile+".001")).FullName
-    [int] $archiveFileCount = $archiveInfo.FilesCount
+        $archiveInfo = Get-7ZipInformation -ArchiveFileName $fullZipName
+        [long] $archiveFileCount = $archiveInfo.FilesCount
+    } else {
+        $archiveInfo = Get-7ZipInformation -ArchiveFileName $ArchiveFile
+        [long] $archiveFileCount = $archiveInfo.FilesCount
+    }
 
     New-PeterReconcile -ReconcileFile $ReconcileFile -SourceFolder $SourceFolder -FileFilter $FileFilter -RootFolder $rootFolder -ExcludeHash:$ExcludeHash -ProcessFileCount $archiveFileCount -IncludeExif:$IncludeExif
     If (!(Test-Path -Path $ReconcileFile )) {    
         Write-Log "Reconcile file '$ReconcileFile' was not created.  See any previous errors"
-        Write-Host "Reconcile file '$ReconcileFile' was not created.  See any previous errors" -ForegroundColor Red
         Close-Log
-        return
+        Throw "Reconcile file '$ReconcileFile' was not created.  Please refer to log '$(Get-LogName)' for details"
     }
 
     # Write Json file as links
@@ -782,19 +913,20 @@ Param(
     $jsonData.Add("Software",$dataItem)
 
     $items = New-Object System.Collections.ArrayList
-    $dataItem = @{"Reconcile"="$ReconcileFile";"Caption"="File listing of archive for reconciliation";}
+    $dataItem = @{"Topic"="Reconcile";"Reconcile"="$ReconcileFile";"Caption"="File listing of archive for reconciliation";}
     $null = $items.Add($dataItem)
     
     if ($IncludeExif) {
-        $ExifFile = Join-Path -Path $global:MetadataPathName -ChildPath $global:default_exifFile
+        $fname = [System.IO.Path]::GetFileNameWithoutExtension($ReconcileFile)
-        $dataItem = @{"Exif"="$ExifFile";"Caption"="Exif information";}
+        $ExifFile = Join-Path -Path $global:MetadataPathName -ChildPath $($fname+"_exif.csv")
+        $dataItem = @{"Topic"="Exif";"Exif"="$ExifFile";"Caption"="Exif information";}
         $null = $items.Add($dataItem)
     }
 
-    $dataItem = @{"SecretFile"="$SecretFile";"Caption"="File used for complex password storage with asymmetric key";}
+    $dataItem = @{"Topic"="SecretFile";"SecretFile"="$SecretFile";"Caption"="File used for complex password storage with asymmetric key";}
     $null = $items.Add($dataItem)
 
-    $dataItem = @{"FileFilter"="$FileFilter";"Caption"="File filter used with Compress";}
+    $dataItem = @{"Topic"="FileFilter";"FileFilter"="$FileFilter";"Caption"="File filter used with Compress";}
     $null = $items.Add($dataItem)
 
     $jsonData.Add("Links",$items)
@@ -802,9 +934,18 @@ Param(
 
     Write-Log "Add folder '$global:MetadataPathName' to file '$ArchiveFile'"
     $fullMetadatName = (Get-Item $global:MetadataPathName).FullName
-    $fullZipName = (Get-Item $ArchiveFile).FullName
+    if ($multiVolume) {
-    Compress-7Zip -Path $fullMetadatName -ArchiveFileName $fullZipName -PreserveDirectoryRoot -Format SevenZip -Append -Password $secret -EncryptFilenames
+        $fext = (Get-ChildItem $ArchiveFile).Extension
-    Remove-Item $fullMetadatName -Recurse
+        $fname = [System.IO.Path]::GetFileNameWithoutExtension($ArchiveFile)
+        $fullZipName = (Get-Item $($fname+"_meta"+$fext)).FullName
+        Compress-7Zip -Path $fullMetadatName -ArchiveFileName $fullZipName -PreserveDirectoryRoot -Format SevenZip -Append -Password $secret -EncryptFilenames
+        # TODO: Change for volumes
+        # -Volume (Get-ReverseConvenientFileSize $VolumeSize)
+    } else {
+        $fullZipName = (Get-Item $ArchiveFile).FullName
+        Compress-7Zip -Path $fullMetadatName -ArchiveFileName $fullZipName -PreserveDirectoryRoot -Format SevenZip -Append -Password $secret -EncryptFilenames -Volume (Get-ReverseConvenientFileSize $VolumeSize)
+        Remove-Item $fullMetadatName -Recurse
+    }
 
     Write-Log "Archive file '$ArchiveFile' created from folder '$SourceFolder'"
     Write-Host "Archive file '$ArchiveFile' created from folder '$SourceFolder'"  -ForegroundColor Green
@@ -943,9 +1084,8 @@ Param(
 
     if (!(Test-Path -Path $ArchiveFile )) {
         Write-Log "Archive file '$ArchiveFile' not found"
-        Write-Host "Archive file '$ArchiveFile' not found"  -ForegroundColor Red
         Close-Log
-        return
+        Throw "Archive file '$ArchiveFile' not found"
     }
     if ($SecretFile -eq "") {
         $SecretFile = $ArchiveFile + ".key"
@@ -963,9 +1103,8 @@ Param(
 
         if ($bucketHost -eq "") {
             Write-Log "Bucket name required" 
-            Write-Host "Bucket name required"  -ForegroundColor Red
             Close-Log
-            return
+            Throw "Bucket name required"
         }
 
         Try {
@@ -986,7 +1125,8 @@ Param(
             Write-Host "Archive file '$ArchiveFile' stored on AWS S3 bucket '$bucketHost' at '$targetObject'" -ForegroundColor Green
         } Catch {
             Write-Log "Error in sending archive file '$ArchiveFile' to AWS S3 with error: $($_)"
-            Write-Host "Error in sending archive file '$ArchiveFile' to AWS S3 with error: $($_)" -ForegroundColor Red
+            Close-Log
+            Throw "Error in sending archive file '$ArchiveFile' to AWS S3 with error: $($_)"
         }
     }
 
@@ -1011,9 +1151,8 @@ Param(
             } 
             if ($null -eq $accountKey -or $accountKey -eq "") {
                 Write-Log "Account key required" 
-                Write-Host "Account key required"  -ForegroundColor Red
                 Close-Log
-                return
+                Throw "Account key required"
             }
         }
 
@@ -1023,25 +1162,22 @@ Param(
             $b2ApiToken = Get-B2ApiToken -AccountId $AccountId -AccountKey $AccountKey
         } Catch {
             Write-Log "Authentication error with account '$AccountID'" 
-            Write-Host "Authentication error with account '$AccountID'" -ForegroundColor Red
             Close-Log
-            return
+            Throw "Authentication error with account '$AccountID'"
         }
 
         if ($null -eq $b2ApiToken.Token -or $b2ApiToken.Token -eq "")
         {
             Write-Log "Authentication error with account '$AccountID' as no API Token" 
-            Write-Host "Authentication error with account '$AccountID' as no API Token" -ForegroundColor Red
             Close-Log
-            return
+            Throw "Authentication error with account '$AccountID' as no API Token"
         }
         
         $b2Bucket = Get-B2Bucket -ApiToken $b2ApiToken.Token -AccountId $b2ApiToken.accountId -ApiUri $b2ApiToken.ApiUri -BucketHost $bucketHost
         if ($null -eq $b2Bucket -or $b2Bucket.BucketID -eq "") {
             Write-Log "Bucket '$bucketHost' not found" 
-            Write-Host "Bucket '$bucketHost' not found" -ForegroundColor Red
             Close-Log
-            return
+            Throw "Bucket '$bucketHost' not found"
         }
 
         $b2UploadUri = Get-B2UploadUri -BucketHost $b2Bucket.bucketId -FileName $ArchiveFile -ApiUri $b2ApiToken.ApiUri -ApiToken $b2ApiToken.Token 
@@ -1065,7 +1201,6 @@ Param(
 
     if (!($remoteType)) {
         Write-Log "Unknown remote path '$TargetPath.'.  No transfer performed" 
-        Write-Host "Unknown remote path '$TargetPath.'.  No transfer performed"  -ForegroundColor Red
         Write-Host "Recognised transfer prefixes: "
         Write-Host "    s3://         : Send to AWS S3 location"
         Write-Host "    b3://         : Send to Backblaze location"
@@ -1073,6 +1208,8 @@ Param(
         Write-Host "If you are saving to local drives or network shared folders,"    
         Write-Host "please use your OS tools to move the file"    
         Write-Host " "    
+        Close-Log
+        Throw "Unknown remote path '$TargetPath.'.  No transfer performed"
     }
 
     Close-Log
@@ -1226,7 +1363,8 @@ Param(
         $null = Read-S3Object -BucketName $bucketHost -File $ArchiveFile -Key $sourceObject
         if (!(Test-Path -Path $ArchiveFile)) {
             Write-Log "Archive file '$sourceObject' not found." 
-            Write-Host "Archive file '$sourceObject' not found."  -ForegroundColor Red
+            Close-Log
+            Throw "Archive file '$sourceObject' not found." 
         } else {
             $sourceObject = $SourcePath.Substring($offset) + ".key"
             $secretFile = $ArchiveFile + ".key"
@@ -1265,9 +1403,8 @@ Param(
             } 
             if ($null -eq $AccountKey -or $AccountKey -eq "") {
                 Write-Log "Account key required" 
-                Write-Host "Account key required"  -ForegroundColor Red
                 Close-Log
-                return
+                Throw "Account key required"
             }
         }
 
@@ -1275,25 +1412,22 @@ Param(
             $b2ApiToken = Get-B2ApiToken -AccountId $AccountId -AccountKey $AccountKey
         } Catch {
             Write-Log "Authentication error with account '$AccountID'" 
-            Write-Host "Authentication error with account '$AccountID'" -ForegroundColor Red
             Close-Log
-            return
+            Throw "Authentication error with account '$AccountID'"
         }
 
         if ($null -eq $b2ApiToken.Token -or $b2ApiToken.Token -eq "")
         {
             Write-Log "Authentication error with account '$AccountID' as no API Token" 
-            Write-Host "Authentication error with account '$AccountID' as no API Token" -ForegroundColor Red
             Close-Log
-            return
+            Throw "Authentication error with account '$AccountID' as no API Token"
         }
 
         $b2Bucket = Get-B2Bucket -ApiToken $b2ApiToken.Token -AccountId $b2ApiToken.accountId -ApiUri $b2ApiToken.ApiUri -BucketHost $bucketHost
         if ($null -eq $b2Bucket -or $b2Bucket.BucketID -eq "") {
             Write-Log "Bucket '$bucketHost' not found" 
-            Write-Host "Bucket '$bucketHost' not found" -ForegroundColor Red
             Close-Log
-            return
+            Throw "Bucket '$bucketHost' not found"
         }
 
         $sourceObject = $SourcePath.Substring($offset)
@@ -1302,7 +1436,8 @@ Param(
         Receive-B2Download -BucketHost $bucketHost -SourcePath $sourceObject -FileName $ArchiveFile -ApiDownloadUri $b2ApiToken.DownloadUri -ApiToken $b2ApiToken.Token
         if (!(Test-Path -Path $ArchiveFile)) {
             Write-Log "Archive file '$sourceObject' not found." 
-            Write-Host "Archive file '$sourceObject' not found."  -ForegroundColor Red
+            Close-Log
+            Throw "Archive file '$sourceObject' not found."
         } else {
             $sourceObject = $SourcePath.Substring($offset) + ".key"
             $secretFile = $ArchiveFile + ".key"
@@ -1323,7 +1458,6 @@ Param(
 
     if (!($remoteType)) {
         Write-Log "Unknown remote path '$SourcePath'.  No get performed" 
-        Write-Host "Unknown remote path '$SourcePath'.  No get performed"  -ForegroundColor Red
         Write-Host "Recognised transfer prefixes: "
         Write-Host "    s3://bucket/path/path     : Fetch from AWS S3 location"
         Write-Host "    b2://bucket/path/path     : Fetch from Backblaze location"
@@ -1331,6 +1465,8 @@ Param(
         Write-Host "If you are fetching from local drives or network shared folders,"    
         Write-Host "please use your OS tools to move the file"    
         Write-Host " "    
+        Close-Log
+        Throw "Unknown remote path '$SourcePath'.  No get performed"
     }
 
     Close-Log
@@ -1459,9 +1595,8 @@ Param(
     
     If (!(Test-Path -Path $ArchiveFile )) {    
         Write-Log "Archive file '$ArchiveFile' does not exist"
-        Write-Host "Archive file '$ArchiveFile' does not exist" -ForegroundColor Red
         Close-Log
-        return
+        Throw "Archive file '$ArchiveFile' does not exist"
     }
 
     if ($RecipientKey -eq "") {
@@ -1480,9 +1615,8 @@ Param(
 
     if (($RecipientKey -eq "") -and ($SecretKey -eq "")) {
         Write-Log "Recipient Key name or Secret Key required for unpacking" 
-        Write-Host "Recipient Key name or Secret Key required for unpacking" -ForegroundColor Red
         Close-Log
-        return
+        Throw "Recipient Key name or Secret Key required for unpacking"
     } 
     
     if ($SecretKey -eq "") {
@@ -1490,14 +1624,11 @@ Param(
         {
             $SecretFile = $ArchiveFile + ".key"
         }
-    
         If (!(Test-Path -Path $SecretFile )) {    
             Write-Log "Secret file '$SecretFile' does not exist"
-            Write-Host "Secret file '$SecretFile' does not exist" -ForegroundColor Red
             Close-Log
-            return
+            Throw "Secret file '$SecretFile' does not exist"
-        }
+        }    
-    
         $secret = Unprotect-CmsMessage -To $RecipientKey -Path $SecretFile
     } else {
         $secret = $SecretKey
@@ -1599,9 +1730,8 @@ Param(
 
     If (!(Test-Path -Path $RestoreFolder )) {    
         Write-Log "Folder '$RestoreFolder' does not exist"
-        Write-Host "Folder '$RestoreFolder' does not exist"  -ForegroundColor Red
         Close-Log
-        return
+        Throw "Folder '$RestoreFolder' does not exist"
     }
 
 
@@ -1641,9 +1771,8 @@ Param(
 
     If (!(Test-Path -Path $ReconcileFile )) {    
         Write-Log "Reconciliation file '$ReconcileFile' does not exist"
-        Write-Host "Reconciliation file '$ReconcileFile' does not exist" -ForegroundColor Red
         Close-Log
-        return
+        Throw "Reconciliation file '$ReconcileFile' does not exist"
     }
 
     Write-Log "Reconciling documents transferred"
@@ -1667,6 +1796,7 @@ Param(
 
     Import-Csv $ReconcileFile | ForEach-Object {
         $totalFileCount = $totalFileCount +1 
+        $errorFileLogged = $false
         if ($RootFolder -ne "") {
             $adjustedName = $_.FullName.Replace($RootFolder, "\")
             $restoreFileName = $(Join-Path -Path $RestoreFolder -ChildPath $adjustedName)    
@@ -1674,66 +1804,131 @@ Param(
             $restoreFileName = $(Join-Path -Path $RestoreFolder -ChildPath $_.FullName)    
         }
         If (Test-Path -Path $restoreFileName ) {    
+
+            $fileItem = Get-Item -Path $restoreFileName -Force
+
             if (!($ExcludeHash)) {
                 if ($_.Hash -ne "") {
                     $targetHash= (Get-FileHash -Path $restoreFileName).Hash
                     if ($_.Hash -ne $targetHash) {
                         $errorCount = $errorCount + 1
                         Write-Log "Hash mismatch for file '$restoreFileName' with target value $targetHash"
+                                    
+                        if (!$errorFileLogged) {
+                            if (!(Test-Path -Path $global:default_errorListFile)) {
+                                $null = New-Item -Path $global:default_errorListFile -ItemType File
+                            }
+                            Add-Content -Path $global:default_errorListFile -Value "$restoreFileName"
+                            $errorFileLogged = $true
+                        }
+
                     }
                 } else {
                     $missingHash = $true
                 }
             }
 
-            if ((Get-Item -Path $restoreFileName).LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastWriteTime) {
+            if ($fileItem.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastWriteTime) {
-                Write-Log "LastWrite mismatch for file '$restoreFileName' with target value $((Get-Item -Path $restoreFileName).LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss")) expected $($_.LastWriteTime)"
+                Write-Log "LastWrite mismatch for file '$restoreFileName' with target value $($fileItem.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss")) expected $($_.LastWriteTime)"
                 $errorCreateCount = $errorCreateCount + 1
 
                 $dateTimeValue = [Datetime]::ParseExact($_.LastWriteTime, 'yyyy-MM-ddTHH:mm:ss', $null)
-                $fileValue = (Get-Item -Path $restoreFileName).LastWriteTime
+                $fileValue = $fileItem.LastWriteTime
                 $diff = ($dateTimeValue - $fileValue).Seconds
                 # Allow +/- 2 second discrepancy
                 if (($diff.Seconds -lt -2) -or ($diff.Seconds -gt 2)) {
                     $errorCount = $errorCount + 1
                 }
+                                    
+                if (!$errorFileLogged) {
+                    if (!(Test-Path -Path $global:default_errorListFile)) {
+                        $null = New-Item -Path $global:default_errorListFile -ItemType File
+                    }
+                    Add-Content -Path $global:default_errorListFile -Value "$restoreFileName"
+                    $errorFileLogged = $true
+                }
             }
-            if ((Get-Item -Path $restoreFileName).Length -ne $_.Length) {
+            if ($fileItem.Length -ne $_.Length) {
                 $errorCount = $errorCount + 1
-                Write-Log "Length mismatch for file '$restoreFileName' with target value $(Get-Item -Path $restoreFileName).Length) expected $($_.Length)"
+                Write-Log "Length mismatch for file '$restoreFileName' with target value $($fileItem.Length) expected $($_.Length)"
+                                    
+                if (!$errorFileLogged) {
+                    if (!(Test-Path -Path $global:default_errorListFile)) {
+                        $null = New-Item -Path $global:default_errorListFile -ItemType File
+                    }
+                    Add-Content -Path $global:default_errorListFile -Value "$restoreFileName"
+                    $errorFileLogged = $true
+                }
+
             }
 
             # Note that last / write access time is not checked by default as it will commonly be changed after restore
             if ($extendedCheck) {
 
-                if ((Get-Item -Path $restoreFileName).CreationTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.CreationTime) {
+                if ($fileItem.CreationTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.CreationTime) {
-                    Write-Log "Creation mismatch for file '$restoreFileName' with target value $((Get-Item -Path $restoreFileName).CreationTime.ToString("yyyy-MM-ddTHH:mm:ss")) expected $($_.CreationTime)"
+                    Write-Log "Creation mismatch for file '$restoreFileName' with target value $($fileItem.CreationTime.ToString("yyyy-MM-ddTHH:mm:ss")) expected $($_.CreationTime)"
                     $errorCreateCount = $errorCreateCount + 1
     
                     $dateTimeValue = [Datetime]::ParseExact($_.CreationTime, 'yyyy-MM-ddTHH:mm:ss', $null)
-                    $fileValue = (Get-Item -Path $restoreFileName).CreationTime
+                    $fileValue = $fileItem.CreationTime
                     $diff = ($dateTimeValue - $fileValue).Seconds
                     # Allow +/- 2 second discrepancy
                     if (($diff.Seconds -lt -2) -or ($diff.Seconds -gt 2)) {
                         $errorCount = $errorCount + 1
                     }
+                                    
+                    if (!$errorFileLogged) {
+                        if (!(Test-Path -Path $global:default_errorListFile)) {
+                            $null = New-Item -Path $global:default_errorListFile -ItemType File
+                        }
+                        Add-Content -Path $global:default_errorListFile -Value "$restoreFileName"
+                        $errorFileLogged = $true
+                    }
+
                 }
     
-                if ((Get-Item -Path $restoreFileName).LastAccessTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastAccessTime) {
+                if ($fileItem.LastAccessTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastAccessTime) {
                     $errorCount = $errorCount + 1
-                    Write-Log "Last access mismatch for file '$restoreFileName' with target value $((Get-Item -Path $restoreFileName).LastAccessTime.ToString("yyyy-MM-ddTHH:mm:ss"))"
+                    Write-Log "Last access mismatch for file '$restoreFileName' with target value $($fileItem.LastAccessTime.ToString("yyyy-MM-ddTHH:mm:ss"))"
+                                    
+                    if (!$errorFileLogged) {
+                        if (!(Test-Path -Path $global:default_errorListFile)) {
+                            $null = New-Item -Path $global:default_errorListFile -ItemType File
+                        }
+                        Add-Content -Path $global:default_errorListFile -Value "$restoreFileName"
+                        $errorFileLogged = $true
+                    }
+
                 }
-                if ((Get-Item -Path $restoreFileName).LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastWriteTime) {
+                if ($fileItem.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss") -ne $_.LastWriteTime) {
                     $errorCount = $errorCount + 1
-                    Write-Log "Last write mismatch for file '$restoreFileName' with target value $((Get-Item -Path $restoreFileName).LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss"))"
+                    Write-Log "Last write mismatch for file '$restoreFileName' with target value $($fileItem.LastWriteTime.ToString("yyyy-MM-ddTHH:mm:ss"))"
+                                    
+                    if (!$errorFileLogged) {
+                        if (!(Test-Path -Path $global:default_errorListFile)) {
+                            $null = New-Item -Path $global:default_errorListFile -ItemType File
+                        }
+                        Add-Content -Path $global:default_errorListFile -Value "$restoreFileName"
+                        $errorFileLogged = $true
+                    }
+
                 }
             }
 
-            $totalFileSize = $totalFileSize + (Get-Item -Path $restoreFileName).Length             
+            $totalFileSize = $totalFileSize + $fileItem.Length             
         } else {
             $missingFileCount = $missingFileCount + 1
             $errorCount = $errorCount + 1
             Write-Log "Non existant target file '$restoreFileName'"
+                                    
+            if (!$errorFileLogged) {
+                if (!(Test-Path -Path $global:default_errorListFile)) {
+                    $null = New-Item -Path $global:default_errorListFile -ItemType File
+                }
+                Add-Content -Path $global:default_errorListFile -Value "$restoreFileName"
+                $errorFileLogged = $true
+            }
+            
         }
 
         if ( $ProcessFileCount -gt 0) {
@@ -1757,21 +1952,28 @@ Param(
     Write-Log "Total file count is $totalFileCount with $errorCount errors"
     Write-Log "There are $missingFileCount missing files"
 
+    $errorDetected = $false
     if ($errorCreateCount -gt 0) {
+        $errorDetected = $true
         Write-Log "File create mismatch count is $errorCreateCount" 
         Write-Host "File create mismatch count is $errorCreateCount" -ForegroundColor Red
     }
 
     if ($errorCount -gt 0) {
+        $errorDetected = $true
         Write-Host "Total file count is $totalFileCount with $errorCount errors"  -ForegroundColor Red
     } else {
         Write-Host "Total file count is $totalFileCount with $errorCount errors"  -ForegroundColor Green
     }
     if ($missingFileCount -gt 0) {
+        $errorDetected = $true
         Write-Host "There are $missingFileCount missing files"  -ForegroundColor Red
     }
 
     Close-Log
+    if ($errorDetected) {
+        Throw "Compare mismatch error detected. Please refer to log '$(Get-LogName)' for details"
+    }
 }
 
 $getEnvName = $(Get-SoftwareName) + "_LOGPATH"

PeterTask.ps1

@@ -239,6 +239,8 @@ param (
 
     [Alias("Exif")]
     [switch] $IncludeExif,
+       
+    [String] $VolumeSize = "-1",
 
     [String] $LogPath = ""
 
@@ -250,7 +252,7 @@ Import-Module .\PeterDocs
 
     if ($task -eq "Compress") {
         $actioned = $true
-        Compress-Peter -SourceFolder $path -SecretKey $SecretKey  -SecretFile $SecretFile -ArchiveFile $archiveFile -ReconcileFile $reconcileFile -RootFolder $rootFolder -FileFilter $fileFilter -LogPath $LogPath -ExcludeHash:$ExcludeHash -IncludeExif:$IncludeExif
+        Compress-Peter -SourceFolder $path -SecretKey $SecretKey  -SecretFile $SecretFile -RecipientKey $RecipientKey -ArchiveFile $archiveFile -ReconcileFile $reconcileFile -RootFolder $rootFolder -FileFilter $fileFilter -VolumeSize $VolumeSize -LogPath $LogPath -ExcludeHash:$ExcludeHash -IncludeExif:$IncludeExif
     }
 
 
@@ -268,7 +270,7 @@ Import-Module .\PeterDocs
 
     if ($task -eq "Expand") {
         $actioned = $true
-        Expand-Peter -RestoreFolder $path -SecretKey $secretKey -SecretFile $secretFile -ArchiveFile $ArchiveFile -LogPath $LogPath 
+        Expand-Peter -RestoreFolder $path -SecretKey $secretKey -SecretFile $secretFile  -RecipientKey $RecipientKey -ArchiveFile $ArchiveFile -LogPath $LogPath 
     }
 
 
@@ -299,6 +301,12 @@ Import-Module .\PeterDocs
             {
                 $SecretFile = $ArchiveFileName + ".key"
             }
+            if (!(Test-Path -Path $SecretFile)) {
+              Write-Log "Secret file '$SecretFile' not found" 
+              Write-Host "Secret file '$SecretFile' not found"  -ForegroundColor Red
+              Close-Log
+              return
+            }
             $secret = Unprotect-CmsMessage -To $RecipientKey -Path $SecretFile
         } else {
             $secret = $SecretKey

README.md

@@ -1,9 +1,10 @@
-# PeterDocs - Protect, Transfer, Reconcile Dcouments
+# PeterDocs - Protect, Transfer, Reconcile Documents
 
 ## Summary
 
 PeterDocs is for [Protecting](Encryption.md), [Transferring](SendArchive.md) and [Reconciling](Reconcile.md) documents
-on a remote computer where the computers are isolated or on different networks.
+on a remote computer where the computers are isolated or on different networks and not accessible via
+file network shares.
 
 The PowerShell module is available on [PowerShell Gallery](https://www.powershellgallery.com/packages/PeterDocs)
 
@@ -13,14 +14,14 @@ that execute the code are required to have Windows PowerShell installed.
 Use the script to create an encrypted archive of the source folder and its contents, then
 transfer the archive file to your target, where the content are unpacked using the decryption
 key. After archive contents are restored you can execute the reconcile function
-to veriy that the contents are transferred, unaltered.
+to verify that the contents are transferred, unaltered.
 
 See [Quick Start](QuickStart.md) if you are ready to start and don't need the details.
 
-If you have access to both source and target folders, then you should consider
+If you have access to both source and target folders as shared folders or even
-using tools such as:
+on the same computer, then you should consider using tools such as:
 
-* Microsoft ROBOCOPY
+* Microsoft ROBOCOPY  - See [Alternate Uses](./Docs/AlternateUses.md)
 * rsync
 
 Alternatively, you can use backup and restore utilities on the folder, and rely that
@@ -89,11 +90,11 @@ will need write access on the target storage. A log file is written at execution
 to record activity.
 
 Your bulk file transfer is encrypted in transit.  Note that if you use the
-SecretKey method the ecnrypted contents will only be as secure as the strength
+SecretKey method the encrypted contents will only be as secure as the strength
 of your secret.
 
 You can use storage providers such as Dropbox, AWS S3, Google Drive, OneDrive or BackBlaze
-and your documents have additonal protection.
+and your documents have additional protection.
 
 A log file is produced on execution.  Repeated executions on the same day
 will add text content to the same log file.  The default log name takes the form:
@@ -104,6 +105,26 @@ via local file NuGet package file if Internet access is limited.
 
 See the [Advanced Usage](Docs/Advanced.md) for more advanced options.
 
+## Limitations
+
+### Secure string
+
+The current version does not use secure strings for password protection 
+within the code.  You data is stil protected with encryption.
+
+## Path length
+
+There is a limitation with the PowerShell functions used within PeterDocs
+of file paths having to be 260 characters in length or less.
+
+If you have long file paths, the processing will fail.  A possible 
+work around is to use mapped net work drive even on your local sourced
+file.  The command in PowerShell would be something like:
+
+```powershell
+New-PSDrive "X" -PSProvider FileSysytem -Root "$Source" 
+```
+
 ## Further Reading
 
 [Design](Docs/Design.md)

samples/Robocopy_sample.ps1

@@ -0,0 +1,34 @@
+param (
+    [Parameter(Mandatory)]
+    [String] $Source, 
+    [Parameter(Mandatory)]
+    [String] $Destination
+)
+
+# Note that there is a path limitation for files of 260 characters
+# beyond which PeterDocs will fail
+# You could also use drive mapping to overcome this
+#  New-PSDrive "X" -PSProvider FileSysytem -Root "$Source" 
+
+$step ="Starting"
+Try {
+    $step ="Creating initial reconcile"
+    New-PeterReconcile -ReconcileFile .\myrobocopy.csv -SourceFolder $Source 
+    $step ="Running robocopy"
+    Write-Host "Running robocopy for source '$Source' and destination '$Destination'"
+    # Change the command line switches to suit
+    robocopy `"$Source`"   "$Destination" /e /copy:DAT /dcopy:DAT /log+:./robocopy.log /r:1000 /w:10
+    if ($LastExitCode -lt 8) {
+        Write-Host "Robocopy succeeded"
+    } else {
+        Write-Host "Robocopy failed with exit code:" $LastExitCode
+        throw "Robocopy error"
+    }
+    $step ="Running copy reconcile"
+    Compare-Peter -ReconcileFile .\myrobocopy.csv -RestoreFolder $Destination 
+    # You can modify the code here to add a success email notification
+} Catch {
+    Write-Host "Error: $_"
+    Write-Error "Processing encountered error at step '$step'"
+    # You can modify the catch to add a simlpe email notification on errors
+}

samples/Samples.md

@@ -0,0 +1,6 @@
+# Samples
+
+This folder contains sample scripts using ``PeterDocs``
+
+The samples are intended to be self documenting.  Please adjust
+the code to suit your circumstances.