HCL Domino DAOS is a fantastic deduplication technology that makes a big difference to running and backing up Domino servers. Only one copy of each attachment is stored on the server no matter how many documents contain that attachment. These attachments are stored as “nlo” files outside of the application databases. It very significantly reduces the volume of the databases and reduces backup loads and volume. We have been using it for over 5 years with no problems.
I have always wondered how we would restore multiple damaged or missing NLO files if the need arose. Today I had to do this for @400 NLO files after they were inadvertently deleted from a test environment. I could have simply restored all of the NLOs but I wanted to see if i could “cherry pick” just the missing ones from backup. In production we would easily have over 500K NLO files.
Step 1 – Which NLOs are missing
From the Domino console run this command as set out in this HCL help note
tell daosmgr listnlo missing dev\xxxxx\fap\demo001.nsf
you will get a response like this telling you that there are 4 nlo files missing :
DAOSMGR: ListNLO started for NSF:e:\IBM\Domino\Data1\dev\xxxxxx\fap\demo001.nsf
DAOSMGR: ListNLO output file: e:\IBM\Domino\Data1\listnlo.txt
DAOSMGR: ListNLO completed – 4 entries processed
The missing files will be recorded in …\Domino\Data1\listnlo.txt but you can specify an alternate location using the -o parameter.
Note that Domino is very fussy about the order of the parameters
tell daosmgr listnlo -o missing.txt missing dev\xxxxxx\fap\demo001.nsf
You will get a file in the data directory on the server that looks something like this :
f:\Domproc1\daos\0001\1611979BE338D04BB1409652E61CAAAD01349DDC001CAB54.nlo
f:\Domproc1\daos\0001\4CDDAB3BBD803FC20AC26CC857E15117371BF5FC00079348.nlo
f:\Domproc1\daos\0001\A309066ADFDF2A4C8FF345F378564662FC8333C5000BDE0F.nlo
f:\Domproc1\daos\0001\353E0802F68CC8A39B483D758F5396926268BEF0004FE864.nlo
Step 2 – Locating the backup files in Azure Blob Storage
In my case the NLO files are backed up to Azure Blog Storage.
The nlo files are stored in a dedicated central DAOS blob container and also, as a precaution, in the appropriate daily and monthly containers.
We use the central DAOS blob container to simplify disaster recovery if needed.
In this case the files could be easily recovered but I wanted to try the scenario of the files only being available from the daily backup folders.
To my surprise I found that there is no easy way to search inside nested folders in Azure blog storage so I had to write a script to search recursively using PowerShell.
I am no PowerShell expert but this is it. I use the PowerShell ISE GUI to do this.
## Input Parameters
$searchFile = ‘1B9EB097A4B4024BF007499C14E2BB2DA89D4EF500044XXX.nlo’
$exportCsv = ‘d:\files.csv’
$resourceGroupName=”xxxxxxxxxxx”
$storageAccName=”yyyyyyy”## Connect to Azure Account
Connect-AzAccount## Function to get the blobs
Function GetBlobs
{
Write-Host -ForegroundColor Green “Retrieving blobs from storage container..”
## Get the storage account
$storageAcc=Get-AzStorageAccount -ResourceGroupName $resourceGroupName -Name $storageAccName
## Get the storage account context
$ctx=$storageAcc.Context
$saKey = Get-AzStorageAccountKey -ResourceGroupName $storageAcc.ResourceGroupName -Name $storageAcc.StorageAccountName
$saContext = New-AzStorageContext -StorageAccountName $storageAcc.StorageAccountName -StorageAccountKey $saKey[1].Value
$saContainer = Get-AzStorageContainer -Context $saContext
foreach ($saFolder in $saContainer)
{
write-host -Foregroundcolor Yellow ‘Checking Folder ‘ + $saFolder.Name
$files = Get-AzStorageBlob -container $saFolder.Name -Context $saContext | Where-Object {$_.Name -match $searchFile}
$files | Select Name, @{N=’StorageAccountName’;E={$_.Context.StorageAccountName}}, @{N=’Container’;E={$saFolder.Name}} | Export-Csv -NoTypeInformation -Path $exportCsv -Append
}
}GetBlobs
## Disconnect from Azure Account
Disconnect-AzAccount
This code reports the location of the specific NLO specified. It could be expanded to find all of the missing NLOs but it would take a long time as it must scan each folder for each missing NLO because there is no native recursive search functionality.
Step 3 – Copying the NLO files from Azure Blob storage
In Step 1 we created a file listing the missing NLOs
In Step 2 we identified where these NLOs were in the backup system ( may need to be expanded )
In Step 3 we will copy these files from backup. In my case I needed @400 files from the 400k files in backup.
I recovered the files to my local desktop but you could do this directly onto the server.
You may need to add AZ to Powershell
See the instructions here
I retrieved the 400 files using PowerShell and a batch file. The batch file must use the current Blog store location so in my case I had to globally replace
f:\Domproc1\daos\0001\
with
2023/0001/0001/
the direction of the / is important.
f:\Domproc1\daos\0001\1611979BE338D04BB1409652E61CAAAD01349DDC001CAXXX.nlo becomes
2023/0001/0001/1611979BE338D04BB1409652E61CAAAD01349DDC001CAXXX.nlo
This is the script file :
## Input Parameters
$resourceGroupName=”xxxxxx”
$storageAccName=”yyyyyyy”
$ContainerName = “daos”
$ImportList = “D:\missingfiles.csv”
$Destination = “D:\temp\005”## Connect to Azure Account
Connect-AzAccount$storageAcc=Get-AzStorageAccount -ResourceGroupName $resourceGroupName -Name $storageAccName
## Get the storage account context
$ctx=$storageAcc.Context
$saKey = Get-AzStorageAccountKey -ResourceGroupName $storageAcc.ResourceGroupName -Name $storageAcc.StorageAccountName
$saContext = New-AzStorageContext -StorageAccountName $storageAcc.StorageAccountName -StorageAccountKey $saKey[1].Value
$saContainer = Get-AzStorageContainer -Context $saContext$files = Get-Content -Path $ImportList
ForEach ($file in $files) {
Get-AzStorageBlobContent -Container $ContainerName -Blob $File -Destination $Destination -Context $saContext -Force
}
The -Force attribute in the last line tells the system to overwrite any previously downloaded files if necessary. This was needed because Domino does record the same NLO as missing multiple times if multiple Notes documents are affected.
Step 4 – Copy the files back to Domino
Now that we have the missing NLO files we need to copy them back to domino.
For me the easiest way to do this is via an RDP session.
Ideally drop the NLO files back into the folders that they were missing from. You can add them to different DAOS folders but then you will need to force a resync with
Tell DAOSMgr Resync Force
So that’s it. For a few missing NLOs you can find and replace them manually but this scripted technique is useful where you need to replace many NLOs but not all.
#SNTT #writeItDownSoIDontForget #dominoforever