Garage S3: LMDB Corruption Post-Mortem

Yesterday I was migrating my data from an rclone-mounted bucket to a proper OpenCloud setup backed by my home Garage S3 cluster. Things were going smoothly: files copying over, users importing, the new stack finally taking shape.

Then the power went out.

When the server came back, Garage refused to start. The LMDB metadata database was completely corrupted, rendering the cluster inoperable. The block data (781k objects, ~57 GB) remained intact on a Btrfs RAID1, but without the LMDB index the mapping from S3 keys to block hashes was lost. The data in my personal bucket and the one used by OpenCloud as its storage backend: gone.

It could have been worse. My most important data lives outside S3, backed by restic to a different target, and that backup pipeline was unaffected. Still, losing two buckets stings.

That said, I didn’t choose S3 on a whim. This bucket needed to be accessed from both my homelab and several VPSs hosting public services, a mix of edge and cloud. S3 gives me a single consistent API and authentication layer everywhere, and Garage’s S3 gateway exposes it without vendor lock-in (unlike, cough, certain other open-source S3 implementations). A local filesystem wouldn’t solve that. I like the option to add nodes for replication down the road, but a crash like this makes you question whether the complexity is worth it for a single-node deployment.

TL;DR: LMDB is known to be vulnerable to corruption after an unclean shutdown. Garage provides a built-in snapshot mechanism (metadata_auto_snapshot_interval) exactly for this scenario. I hadn’t enabled it. I also had no Btrfs snapshots on the metadata partition. If you run Garage on LMDB, enable snapshots today.

System Layout

• Metadata LMDB: single-device Btrfs partition (/dev/nvme0n1p2, 238 GiB NVMe, 98% full)
• Block data: Btrfs RAID1 (/mnt/raid/, 5.5 TiB, HDDs)
• Garage version: v2.x, db_engine = "lmdb"
• COW: enabled on both partitions
• Snapshots/backups: none for the metadata directory

Timeline

• T-unknown: Power outage. Server loses power mid-operation. Garage was writing to the LMDB database.
• T=0: Server boots back up. Garage refuses to start. Attempting to inspect the LMDB file:

$ mdb_stat -e /var/lib/garage/meta/db.lmdb/data.mdb
mdb_stat: src/mdb.c:...: Assertion 'IS_BRANCH(mc->mc_pg[mc->mc_top])' failed.
Aborted

$ mdb_copy -c /var/lib/garage/meta/db.lmdb/data.mdb /tmp/recovery.mdb
mdb_copy: src/mdb.c:...: Assertion 'IS_BRANCH(mc->mc_pg[mc->mc_top])' failed.
Aborted

The LMDB b-tree is structurally gone. I sat there staring at the terminal for a good minute. No error message to decode, no partial recovery: just a dead database and 57 GB of unlabeled blocks on the RAID. Garage’s own docs warn about this:

LMDB is prone to database corruption after an unclean shutdown (e.g. a process kill or a power outage).

• T+1h: Garage service migrated to a fresh cluster on a different machine. Block data copied from the RAID. OpenCloud is re-targeted.
• T+2h: Btrfs COW recovery attempted. All 10 candidate tree roots (generations 2477737–2477773) contain the same corrupt extents.
• T+4h: LMDB raw page scan. 956k pages, all flags=0x0000, lower=1, upper≈24. No valid root page. The database was effectively erased.
• T+24h: Recovery abandoned. Data re-seeded from scratch where possible.

Root Cause

A power outage interrupted Garage mid-write on the LMDB metadata database. LMDB uses mmap’d I/O and, by default, operates with MDB_NOMETASYNC and MDB_NOSYNC (Garage sets metadata_fsync = false by default). Without fsync, a power loss during a transaction commit leaves the b-tree in an inconsistent state: torn pages, missing branches, zero-filled root pointers.

This is a well-documented limitation of LMDB, not a Garage bug. The recovery path Garage documents is:

1. Replication (requires replication_factor ≥ 2 and multiple nodes): delete the corrupt DB, restart, and run garage repair -a --yes tables to resync from peers.
2. Metadata snapshots (metadata_auto_snapshot_interval): restore from a Garage-generated snapshot, then repair.
3. Filesystem snapshots (Btrfs/ZFS): restore from an old snapshot of the metadata directory.

I had none of these. I didn’t know about the snapshot feature; I hadn’t read the docs carefully enough. If I had, this post wouldn’t exist.

Recovery Attempts

Btrfs COW recovery

The LMDB file had COW enabled. Each page write creates a new extent at a different physical location; old page data remains on disk until overwritten.

$ btrfs-find-root /dev/nvme0n1p2
Well block 174354268160 (gen: 2477773)
Well block 174354300928 (gen: 2477772)
...
Found tree root at 174435041280 gen 2477773
Found tree root at 174433533952 gen 2477772
Found tree root at 172099076096 gen 2477739
...

10 candidate tree roots identified, spanning generations 2477737–2477773.

Each root was walked to find EXTENT_DATA items for the LMDB inode. The extent map had not changed significantly between generations; only the final extent was COW’d at gen 2477738, and all earlier extents pointed to the same (corrupt) physical blocks.

$ btrfs restore -t <root> /dev/nvme0n1p2 /tmp/recovery/
# btrfs restore cannot traverse FS_TREE roots at b-tree level 3

Direct file extraction from old generations was not possible.

$ btrfs restore -r /dev/nvme0n1p2 /tmp/recovery/
# Produces the same corrupt file; the inode's extent map had not reverted

Conclusion: The corruption happened before any Btrfs snapshot could diverge. The old page data was in Btrfs free space with no index to locate it. COW alone is not a backup. You need explicit snapshots.

LMDB raw page analysis

A custom scanner was written to walk the LMDB file and recover any valid b-tree pages. This felt promising at first: surely some pages survived. Scanning through the raw bytes, looking for recognizable LMDB page headers, hoping for a valid root.

This is exactly the kind of task where AI tools shine. You have a stable read-only copy of the corrupt file, a well-defined binary format, and a one-off analysis script you’ll never need again. Perfect for throwing at an LLM and iterating on the output.

Total pages:    ~956,000
  Leaf pages:    983
  Branch pages:  304
  Overflow:      1,938

Root pages:
  Active root at position 956,702  →  points to zero-filled page
  Last root at position 956,708    →  points to zero-filled page

Freelist:
  freeid counters: 1, 2  →  fully-allocated file, no released pages

All pages:
  flags  = 0x0000
  lower  = 1
  upper  ≈ 24    ← metadata pointer structure erased

The corruption pattern is consistent with mdb_env_open(MDB_CREATE) initializing a new empty database over the existing file. The file was re-opened after the crash with no readable b-tree root.

This is a deliberate LMDB design choice: MDB_CREATE makes mdb_env_open silently create or overwrite the database file if the header is unreadable, rather than returning an error. The rationale is performance: LMDB avoids a full integrity scan at startup and trusts the OS to have flushed pages correctly. If you used metadata_fsync = true, LMDB would have at least synced the meta page, giving the header enough integrity to survive a crash. But with the default false and MDB_NOSYNC in play, a torn meta page looks exactly like an uninitialized file, and LMDB’s fail-safe is to start fresh rather than halt. It’s a speed-over-safety tradeoff that makes sense for the read-optimized workloads LMDB was built for, but backfires catastrophically when power is lost mid-write.

Orphaned block extraction (not executed)

781k orphaned zstd-compressed block files (~57 GB) exist on the RAID. I could see them: ls /mnt/raid/garage/data/ dumped thousands of hash-named files. But without the LMDB mapping of (bucket, key) → block hash, blocks cannot be attributed to objects. A content-based classification (magic bytes → file extension) was considered but, at some point, you have to accept the loss and move on.

Lessons Learned

What went wrong

1. No metadata snapshots: Garage has provided metadata_auto_snapshot_interval since v0.9.4. I hadn’t enabled it. This single config line would have made recovery trivial: stop Garage, swap in the snapshot, run garage repair -a --yes tables, done. Honestly, with LMDB being the default engine and its corruption-on-power-loss behavior being documented, this option should be enabled by default.

2. No Btrfs snapshots on the metadata partition: the RAID had snapper timeline snapshots, but the metadata partition on the NVMe didn’t. A Btrfs snapshot on the metadata directory would have been a second line of defense.

3. No replication: a single-node cluster with replication_factor = 1 means no peer to resync from. With just one additional node, I could have deleted the corrupt DB and recovered automatically.

4. No monitoring: no alerting for Garage crash loops or LMDB assertion failures. I discovered the issue when my laptop was showing an empty folder instead of the mounted bucket. Then OpenCloud started returning 503s.

What went well

1. Btrfs RAID1 preserved block data: all block data (~57 GB) was intact and transferable.
2. restic backups outside S3: my critical data lives outside S3 and is backed by restic to a separate target. That pipeline was unaffected. The data I actually care about is safe.
3. Snapper timelines on RAID: /mnt/raid/ subvolumes had Btrfs snapshot timelines (though the metadata wasn’t on RAID, so this didn’t help).

Preventive Measures Implemented

Garage metadata snapshots

Enabled Garage’s built-in snapshot mechanism. The snapshots are stored on the same RAID subvolume that already has snapper timeline snapshots, so they inherit Btrfs-level protection automatically:

# /etc/garage/config.toml
metadata_auto_snapshot_interval = "6h"
metadata_snapshots_dir = "/mnt/raid/garage/snapshots"
metadata_dir = "/var/lib/garage/meta"
data_dir = "/mnt/raid/garage/data"

Garage keeps the two most recent snapshots and deletes older ones automatically. Recovery from a snapshot is straightforward:

cd /var/lib/garage/meta
mv db.lmdb db.lmdb.bak
cp -r snapshots/2026-07-22T06:00:00Z db.lmdb
systemctl restart garage
garage repair -a --yes tables

Note: Garage docs point out that filesystem-level snapshots may be corrupted if taken during a write. Garage’s own snapshots are crash-consistent and should be your primary recovery path. Btrfs snapshots are a best-effort secondary line.

Future Recommendations

• Btrfs snapshots on the metadata partition: add snapper timeline snapshots on /var/lib/garage/meta as a second layer of defense:

snapper -c garage-meta create-config /var/lib/garage/meta

• Scheduled metadata export: garage meta dump to a restic or S3 target:

#!/usr/bin/env bash
garage meta dump /tmp/garage-meta-dump.json
restic backup /tmp/garage-meta-dump.json

• Monitoring: alert on Garage process crashes and LMDB integrity. A simple systemd OnFailure hook would catch crash loops.
• Multi-node replication: even a second low-power node with replication_factor = 2 would eliminate the single point of failure.
• Move metadata to RAID: migrating the metadata directory to the Btrfs RAID alongside block data would simplify backup and recovery to a single snapper configuration.

Roundup

LMDB corruption after power loss is a known issue. Garage ships with the tools to prevent this (metadata_auto_snapshot_interval, Btrfs/ZFS snapshot support, and multi-node replication), but none of them help if they aren’t configured. And I still think metadata_auto_snapshot_interval should default to something like "6h" when db_engine = "lmdb". It’s a footgun that costs users their data.

If you run Garage on LMDB, stop now and check:

1. Is metadata_auto_snapshot_interval set? If not, add it to your garage.toml and restart.
2. Are the snapshots stored somewhere with redundancy? Set metadata_snapshots_dir to a path on your RAID.
3. Do you have Btrfs/ZFS snapshots on the metadata directory? If you use snapper, add a config for it.
4. Can you survive losing a single node? If not, add a second node and bump replication_factor.

The cost of these precautions is near-zero. The cost of skipping them is an evening of recovery work and a permanent knot in your stomach when you see a thunderstorm approaching.

This is not pro-grade hardware in a datacenter with redundant power. It’s a home server plugged into a wall outlet. Failures are not just possible. They’re expected. If your data matters, act like it.