Skip to main content

Backup and Restore

Fabric-X state lives in two places on the host: the per-node data directories (orderer ledger and WAL, committer ledger, sidecar MSP/TLS) and the postgres container's PGDATA. Both are captured by ChainLaunch's restic-based backup provider when configured.

This page covers what's snapshotted, what's not, and how to restore.

What gets backed up​

ChainLaunch's restic provider snapshots the configured data path (~/Library/Application Support/chainlaunch/ on macOS, /var/lib/chainlaunch/ or your custom dataPath on Linux) plus a hot copy of the SQLite control-plane database.

For Fabric-X, the data path contains:

PathContents
${dataPath}/fabricx-orderers/<group-slug>/Per-role MSP, TLS, genesis, ledger, store, and consensus WAL for the orderer group
${dataPath}/fabricx-committers/<group-slug>/Per-role MSP, TLS, genesis, ledger, and config for the committer
${dataPath}/services/postgres/<container-name>/Postgres PGDATA for the managed postgres container

The control-plane SQLite database is copied to ${dataPath}/dbs/chainlaunch-<timestamp>.db immediately before the restic snapshot starts, so it's captured in a consistent state along with the on-disk material.

Why postgres needs a bind mount​

Earlier versions of ChainLaunch ran the managed postgres without a host bind mount — the container's PGDATA lived only in its writable layer. Restic snapshots of the data path silently missed all coordinator and query-service state. Removing the container (upgrade, host migration, docker prune) lost the data.

Current ChainLaunch versions bind-mount PGDATA to ${dataPath}/services/postgres/<container-name>/, so the same restic snapshot that captures orderer and committer data also captures every committer's postgres state. No backup configuration changes are required to pick this up — the bind mount is created automatically.

If you upgraded an existing install across the bind-mount change, the postgres container will start against an empty PGDATA the first time it comes up after upgrade and will run the upstream image's initdb — meaning state from before the upgrade is not in the bind mount and will not be in subsequent backups. The pre-upgrade state lives in the now-orphaned container's writable layer until that container is removed. Plan a pg_dump from the running pre-upgrade container before the upgrade if you need that data.

What's not backed up​

  • token-sdk-x application state. ChainLaunch doesn't manage token-sdk-x; app-side keys, configs, and DBs are out of scope.
  • Docker images. Restoring assumes the same fabric-x-orderer and fabric-x-committer image tags are pullable. Pin and mirror your images separately if you need offline recovery.
  • External postgres. If a committer is configured against an external postgres (postgresHost pointing off-host), ChainLaunch doesn't snapshot it. Use the external database's native backup tooling.

Configuring a backup target​

Backup targets are configured via the Backups section of the web UI or via POST /api/v1/backups/targets. Today only the restic (S3-backed) provider is production-ready; EBS_SNAPSHOT and VMWARE_SNAPSHOT providers exist but are deployment-environment-specific.

Restic provider config (S3):

{
  "type": "restic",
  "name": "fabricx-prod",
  "config": {
    "endpoint": "s3.us-east-1.amazonaws.com",
    "bucket": "chainlaunch-backups",
    "prefix": "fabricx/prod",
    "accessKey": "AKIA...",
    "secretKey": "...",
    "encryptionPassword": "..."
  }
}

Once a target exists, you can:

  • Trigger an on-demand snapshot from the UI.
  • Schedule recurring snapshots via POST /api/v1/backups/schedules.
  • Verify a snapshot via the verify-backup endpoint (Fabric-peer-only today; see limitations).

Snapshot consistency​

ChainLaunch snapshots the hot data path and a SQLite copy without quiescing the running containers. For the Fabric-X subsystem this is acceptable because:

  • Orderer ledger and WAL are written via fsync-on-commit. A snapshot at any moment captures a recoverable state.
  • Committer postgres is captured two ways. First, before the snapshot starts ChainLaunch runs pg_dumpall inside every managed Postgres container (one per network) and writes a gzipped SQL dump to ${dataPath}/services/postgres/<container>/dumps/<UTC-timestamp>.sql.gz. Restic then snapshots the entire data path, so the snapshot contains both the live PGDATA directory (recoverable via Postgres WAL replay on restart) and a transactionally-consistent SQL dump that can be replayed with psql -f if WAL recovery isn't enough.
  • pg_dumpall failures don't abort the backup. If Docker is unreachable or pg_dumpall exits non-zero, the warning is logged and surfaced in the backup notification metadata, then the file-level snapshot proceeds. A partial backup with no SQL dumps is still better than no backup at all.

For applications that require a stronger guarantee (e.g. a known-clean end-of-day point), stop the affected node groups via the UI or POST /api/v1/node-groups/{id}/stop before triggering the snapshot.

Restoring​

Restoring is currently a manual procedure. The high-level sequence:

  1. Stop the ChainLaunch server.
  2. Restore the data path from the restic snapshot to the same on-disk location: 
    restic -r s3:s3.us-east-1.amazonaws.com/chainlaunch-backups/fabricx/prod \
      restore latest \
      --target /var/lib/chainlaunch
  3. Restore the SQLite control-plane database from ${dataPath}/dbs/chainlaunch-<timestamp>.db to the location ChainLaunch expects (default: ${dataPath}/chainlaunch.db).
  4. Start ChainLaunch. It will hydrate node groups from the SQLite DB.
  5. Start the node groups. The orderer ledger, committer ledger, and postgres PGDATA are all already on disk, so containers boot against the restored state.

Cross-host restores work as long as externalIp and the configured ports match — both are baked into the genesis block and the orderer group's TLS SAN list. If the new host has a different IP, you'll need to either:

  • Pre-rewrite externalIp in the SQLite DB before ChainLaunch starts, or
  • Use the per-network localDev flag for a same-host workstation restore.

A first-class restore endpoint is on the roadmap.

Verifying coverage​

A quick check that postgres state is actually under the data path:

ls ${dataPath}/services/postgres/
# fabricx-org1msp-mygroup-postgres/
# fabricx-org2msp-mygroup-postgres/
# ...

Each subdirectory contains a postgres PGDATA layout (base/, pg_wal/, etc.). If a committer's directory is missing here, the container is running against the writable layer instead of the bind mount — recreate the postgres service to fix it (see pre-upgrade state).

Limitations​

  • Verify-backup is Fabric-peer-only. The "verify" provider hook checks Fabric peer ledger integrity; there's no Fabric-X-specific verifier today. The snapshot itself is restic-verified end-to-end (restic's check command works against ChainLaunch backups), so corruption-of-snapshot is detected — but Fabric-X application-level integrity (e.g. block heights matching across parties) is not.
  • No first-class restore endpoint. Restore is operator-driven today.
  • Restic provider only for production. Other providers (EBS_SNAPSHOT, VMWARE_SNAPSHOT) are environment-specific and not as widely tested for Fabric-X.

See also​