Rebuilding a Home Server After SD Card Failure: A NixOS Survival Story

A Raspberry Pi 4B home server died when its microSD card finally gave out after years of 24/7 operation. The author, running NixOS, used the failure as an opportunity to rebuild with a focus on durability and redundancy — and documented the entire process.

The root cause was a dead microSD card, likely accelerated by a power outage that delivered the final blow. The author had taken few precautions to minimize writes, and the card had been mounted as the root filesystem for years. The only real losses were .torrent files and service caches — all easily regenerated. The real lesson: SD cards are not suitable for long-term root filesystem duty on a server.

Minimizing SD Card Writes

The new setup uses zram for swap instead of swapping to the microSD. zram creates a compressed block device in RAM, which is far faster and avoids wearing out the card. The author also configured /tmp as a RAM disk and set journald to store logs in memory (storage = "volatile"). The root filesystem is mounted with noatime to prevent access-time updates from causing writes.

{
  zramSwap.enable = true;
  boot.tmp.useTmpfs = true;
  services.journald.storage = "volatile";
  fileSystems."/" = {
    device = "/dev/disk/by-label/takodachi";
    fsType = "ext4";
    options = [ "noatime" ];
  };
}

Btrfs RAID1 on Scavenged Drives

Instead of buying new hardware in a market inflated by AI demand, the author repurposed two old 2.5" HDDs — a 500GB and a 320GB — into a btrfs RAID1 pool. The pool is named "ponkotsu" (Japanese for "piece of junk"). Btrfs was chosen for its flexibility: subvolumes, online expansion, and mixed-size drives.

sudo mkfs.btrfs --data raid1 --metadata raid1 --label ponkotsu /dev/sdX /dev/sdY

Declarative Subvolume Management with NixOS

The author created a custom NixOS module called autosubvol to declaratively manage btrfs subvolumes per service. Each subvolume gets its own systemd mount unit, with dependencies tied to the service it supports. This keeps the configuration clean and ensures subvolumes are created automatically on first boot.

{
  services.autosubvol = {
    enable = true;
    disks.ponkotsu = {
      device = "/dev/disk/by-uuid/<pool uuid>";
      subvolumes.immich = {
        mountPoint = "/var/lib/immich";
        mountOptions = [ "noatime" ];
        requiredBy = [ "immich-server.service" ];
      };
    };
  };
}

Backups with Restic and sops-nix

Backups are handled by a Nix module that bridges sops-nix (for secrets) with the restic nixpkgs module. Each service's data directory is backed up to an S3 bucket, with pruning rules to keep daily, weekly, and monthly snapshots. The author also created a subvolume for /var/cache to keep volatile data off the SD card.

{
  internal.backups.restic.immich = {
    paths = [ "/var/lib/immich" ];
    exclude = [
      "/var/lib/immich/encoded-video"
      "/var/lib/immich/thumbs"
    ];
    pruneOpts = [
      "--keep-daily 7"
      "--keep-weekly 4"
      "--keep-monthly 3"
    ];
  };
}

The approach is pragmatic: use what you have, automate everything, and treat your SD card as disposable. The full NixOS configuration and the autosubvol module are available in the author's Git repository.