A few weeks ago my desktop computer suffered catastrophic hard drive failure. Not only did it not boot, it soon got to the point where even the BIOS would fail to recognise the device. At first I did not worry too much. Although I was not doing automatic backups, I was still doing irregular weekly manual backups of my home directory with tarsnap and I had performed one about three days prior. I was not specifically making backups of my NixOS system and user configuration, but I had some old copies. The configuration files do not change much and they are less important. Importantly, I had backups of my tarsnap keys stored in other places, such as my shell account.
While waiting for a replacement drive to arrive, I realized I had a serious problem. My tarsnap keys were encrypted with my PGP key. I had two specific places where I kept backup of my PGP keys. One place was a USB drive in a safe deposit box. However, at some point I had taken that one out to update it, and then misplaced it before putting it back. Naturally, I had been meaning to get around to replacing that USB drive and the data on it, for some number of years. Also, to my surprise, I had never actually placed my PGP key in my secondary spot.
I was sunk. I had some very old hard drive images with older versions of my PGP key on it, but because I rotate my encryption keys every five years, they were not useful. Within the last five years I had started using full disk encryption. I had some newer hard drive images that also have my PGP keys but I need the passphrases to decrypt these images. I had copies of the passphrase around, but, of course, they were all encrypted with my PGP keys.
After an emotional day and some meditation, slowly my old passphrase came back to me and I was able to decrypt one of my disk images. I was able to rescue my PGP keys and from there I was able to recover everything I had.
I plan to get a bit more serious about distributing copies of my PGP key since I use it so widely. With my PGP key I should be able to always recover everything since I keep all my other key material encrypted with it. Instead of a single USB drive in a safe deposit box, I want to keep two identical USB keys, one at home and one in the box. When I want to update the data, I will update the one at home, swap it with the one in the box, and update the second one and keep it at home until the next update is needed.
I have also gotten more serious about automatic backup.
Turns out that NixOS already comes with a tarsnap system service.
All that one has to do is place one’s write-only tarsnap key in the appropriate place and specify which directories to back up.
I am hoping to make system recovery even easier by also backing up my ~/.nix-profile/manifest.nix, /root/.nix-profile/manifest.nix,/nix/var/nix/profiles/default/manifest.nix, /nix/var/nix/profiles/per-user/*/profile/manifest.nix, /etc/nixos and /var/lib.There are probably a few more things I should back up, like my user profiles, but I am not yet sure how best to do that.
I also want to restart my programme of escrow for my passwords in case something happens to me. I need to improve my documentation of password list to make it easier for others to use. I will use ssss to split my master password and distribute among my escrow agents. The nice thing about public-key cryptography is that I can assign escrow agents without requiring anything from them beyond the fact that they already possess and use PGP keys. I do not even need to inform them that they are my escrow agents. The encrypted components will be stored on my USB drive in the safe deposit box.
Overall, I am glad to have narrowly avoid disaster and have definitely learned some lessons. Check your backup policy everyone!