The storage drive on my Framework failed this weekend. Thankfully, former me was wise enough to schedule nightly backups with Restic, which I was able to restore from.
Problem 🔗
My SSD failed on my Framework laptop this weekend. I had been working on it, plugged it in and left for an errand, and when I came back, the fan was on high and the screen was black. I shutdown the computer, and when I turned it back on, I got a blue screen on startup that said:
Default Boot Device Missing or Boot Failed
I found a knowledge base article on the framework support site which suggestted that updating the firmware on the drive may fix it. I had the exact drive that the KB article was calling out
- Western Digital WD_BLACK SN850 SSD:
- Download Western Digital Dashboard: https://support.wdc.com/downloads.aspx?p=279
- Run the installer to install the Dashboard.
- If the “Firmware Version” shows that there is a warning of a newer firmware available, update it by clicking on “NEW FIRMWARE AVAILABLE”
The Western Digital Dashboard that supports updating firmware is only available on Windows. Luckily, I have an old machine in the basement that I keep around for preciseily this type of occasion - the rare occasion I need Windows.
I pulled the drive and plugged it into an enclosure, and plugged that into the Windows machine running the Western Digital Dashboard, but neither Windows nor the Dashboard could not read the drive.
I’m not a big hardware guy, so that was about the extent of my troubleshooting. Just in case it was a mere formatting issue, I plugged it into a macbook, just to see if the drive would be recognized, but OSX also failed to recognize the existance of the drive.
Maybe later I’ll attempt some more diagnostics on the failed drive to see if it really is a firmware issue. I’m not really sure how to tell if the thing is just dead.
R.I.P.
Restoring 🔗
I installed a new WD_BLACK SN850X SSD and flashed Ubuntu on it. (I love how easy the Framework made this.)
My backups are on Wasabi. So, the first thing was to get a new set of access keys.
export AWS_ACCESS_KEY_ID=<access_key>
export AWS_SECRET_ACCESS_KEY=<secret_key>
I, unfortunately, was not quite wise enough to document which bucket I had been backing up to, nor to save the password to my password manager. Therefore, the next step was to figure out which bucket I had been backing up to. Once I thought I had it, I started trying to list snapshots in order figure out the password.
restic snapshots -r s3:https://s3.wasabisys.com/<bucket-name>
Thank goodness, I was able to guess the password!
Finally, I was able to restore.
restic restore -r s3:https://s3.wasabisys.com/<bucket-name> --target / latest
Suggestions 🔗
- Start backing up! Restic is great, and doesn’t take too much effort to get started.
- Automate the backup process.
- Save your credentials somewhere not on the machine being backed up!
- Leave yourself instructions on how to backup/restore.
I now have a script at ~/.backup.sh that is triggered nightly on a cron
0 1 * * * /home/andrew/.backup.sh 2>&1 | /usr/bin/logger -t RESTIC`
#!/bin/sh
#
# ~/.backup.sh
#
# This is the script that backs up the homedir with restic via cron.
# When restoring to a new hard drive, recreate the cron:
#
# 0 1 * * * /home/andrew/.backup.sh 2>&1 | /usr/bin/logger --tag RESTIC
#
# Output of the script goes to /var/log/syslog
restic backup \
--tag cron \
--repository-file /home/andrew/.config/restic/repository \
--password-file /home/andrew/.config/restic/password \
/home/andrew