So, I moved my nextcloud directory from a local SATA drive to a NFS mount from a nvme array on a 10G network
“I just need to change /docker/nextcloud
to /mnt/nfs/nextcloud
in the docker-compose.yml
, what’s the issue, i do it live” - i tell myself
So i stop the container, copy /docker/nextcloud
to /mnt/nfs/nextcloud
, then edit the docker-compose.yml
… and… because I’m doing it during a phone call without paying too much attention i change the main directory to /docker
I rebuild the container and I immediately hear a flood of telegram notifications from my uptime-kuma bot… oh oh…
Looks like the nextcloud docker image has an initialization script that if it doesn’t find the files in the directory, it will delete everything and install a fresh copy of nextcloud… so it deleted everything on my server
Luckily i had a very recent full borg backup and i’m restoring it (i kinda love-hate borg, i always forget the restore commands when in panic and the docs are a bit cryptic for me)
Lessons learned:
-
always double check everything
-
offsite backups are a must (if i accidentally wrote
/
as path, i would have lost also the borg backups!) -
offsite backups should not be permanently mounted, otherwise they would have been wiped as well
-
learn how to use and schedule filesystem snapshots, so the recovery wouldn’t take ages like it’s taking right now (2+ hours and i’m not even half way…)
If anyone else is reading this in the future:
After 8 hours the backup is restored (of course i stored it on a wd green…) but then nextcloud gave error 500 with lots of errors like:
DON’T PANIC. You didn’t screw up the restore!
Just access your database with heidisql or your favorite tool and query
FLUSH HOSTS;
My guy just documenting for his future self, if it ever happens again.
Haha
Been there 🤣
Can’t tell you how many times I’ve googled things and found my own posts and bug reports.
@Moonrise2473 putting this in my notebook. Plus, having it on hand will prevent me ever needing it.