Friday, February 2, 2024

[SOLVED]: How to fix DPM 40003 error (Data Protection Manager 2016)

Summary: DPM fails with error 40003 complaining either on DPM agent's or DPM server's storage.
It seems the real reason (in my case) behind this error was that the last backup taken was corrupted!
(Furthermore, in my case it turned out that RAID array was in the degraded state on DPM server's storage side. So it's better for you to CHECK YOUR DISK SUBSYSTEM and RAID STATUS!!! of both client and server). 

To fix it you need to delete last (possibly corrupted) backup (or Recovery Point in terms of DPM).
However you can't do it via GUI, only from Powershell.
Basic concept to do it is as follows:
* Determine Protection group of that RecoveryPoint
* Determine Datasource of that RecoveryPoint
* Determine the last RecoveryPoint available for that Datasource
* Delete it.
This helped me a lot of times (even with degraded RAID!).

Don't forget to launch Powershell with elevated/administrator privileges.

$pg = Get-ProtectionGroup -DPMServerName your_DPM_server | where {$_.Name -eq "MSSQL System Databases" }

Get-Datasource -ProtectionGroup $pg | where {$_.Computer -like "*computer_with_DPM_agent*"  } |where {$_.Name -eq "msdb" }

$ds = Get-Datasource -ProtectionGroup $pg | where {$_.Computer -like "*computer_with_DPM_agent*"  } |where {$_.Name -eq "msdb" }

Get-RecoveryPoint -Datasource $ds |Sort-Object -property BackupTime

Get-RecoveryPoint -Datasource $ds |Sort-Object -property BackupTime |select-object -last 1

$rp = Get-RecoveryPoint -Datasource $ds |Sort-Object -property BackupTime |select-object -last 1

echo $rp

Remove-RecoveryPoint -RecoveryPoint $rp 

Update: I noticed that after rebooting DPM server this error almost gone (while still running with degraded RAID 5!). I guess it's because of hard swapping issues and hence slow disk subsystem due to degraded RAID - it was partially solved by reboot.
So simple reboot of DPM server may also help you!

Original idea taken from here:
[1] https://social.technet.microsoft.com/Forums/en-US/30f8ef3f-1d20-42ae-bffb-dab49fdea0fc/quoterror-40003-the-storage-involving-the-current-operation-could-not-be-read-from-or-written?forum=dataprotectionmanager
[2] https://www.jfe.cloud/how-to-delete-a-specific-recovery-point-in-dpm/