Backups

Overview

PodWarden's built-in backup system creates incremental, deduplicated, encrypted backups of workload volumes using Restic. Backups run as Kubernetes Jobs or CronJobs inside the cluster — no external backup agent is required.

What gets backed up: persistent volume contents (PVC data). The backup engine mounts the same volumes as your workload and streams them to a storage connection you control.

Where backups go: any NFS or S3-compatible storage connection registered in PodWarden. One storage connection can serve as the backup target for many workloads. Each policy gets its own isolated Restic repository path.

Backup Modes

Hot mode

The backup Job mounts the PVC volumes while the workload is running. Add a pre-backup hook to run a command inside the pod before file copy starts:

mysqldump -u root -p$MYSQL_ROOT_PASSWORD --all-databases > /var/lib/mysql/dump.sql

The hook runs via kubectl exec. File backup starts only after the hook completes.

Cold mode

1. PodWarden scales the workload to 0 replicas
2. Restic backs up the volumes
3. Workload scales back to 1

Typical downtime is 20-60 seconds depending on backup storage speed.

PostgreSQL Backups

In addition to volume backups, PodWarden supports direct PostgreSQL database backups. Instead of mounting PVC files, a PostgreSQL backup runs pg_dump against the database and stores the dump file via Restic.

When to use

• Volume backup with pre-hook — best for most databases. The hook runs pg_dump inside the running pod, then Restic copies the dump file along with other volume data.
• PostgreSQL backup — use when the database is external to K8s (e.g. a managed database or a database on another host), or when you want a dedicated backup pipeline separate from volume data.

Creating a PostgreSQL Backup Policy

1. Go to Backups and click New backup policy
2. Set Backup type to postgres
3. Fill in the database connection details:
   • Host — database hostname or IP
   • Port — default 5432
   • Database — database name
   • User — PostgreSQL user
   • Password — stored as a Kubernetes Secret
4. Set the schedule and retention as usual
5. Select a storage target
6. Click Create policy

How it works

PodWarden creates a CronJob with two containers:

1. Init container (postgres:16-alpine) — runs pg_dump --format=custom to a shared temporary volume
2. Main container (restic/restic) — uploads the dump to the Restic repository

The custom format produces a single compressed file suitable for selective restore with pg_restore.

Creating a Backup Policy

1. Go to Backups in the sidebar
2. Click New backup policy
3. Select the deployed workload to back up
4. Choose backup mode: Hot or Cold
5. If hot mode: optionally enter a pre-backup hook command
6. Set the schedule (daily, weekly, or custom cron)
7. Select the storage target (a registered storage connection)
8. Configure retention
9. Click Create policy

After creating, click Apply to install the Kubernetes CronJob. Click Run now for an immediate backup.

Schedule

Retention

Retention rules are passed to Restic's forget command after each backup:

Restic applies them cumulatively — a snapshot is kept if it matches any rule.

Snapshots

Click any policy row to see its snapshots. Each shows the Restic snapshot ID, timestamp, size, status, and file counts.

Restic's incremental deduplication means subsequent backups of unchanged data are fast and store only deltas.

Restoring from a Snapshot

1. Click the restore button on a successful snapshot
2. Confirm the restore
3. PodWarden scales the workload to 0, runs restic restore, then scales back to 1

Restore takes 30-120 seconds plus data transfer time. Data written after the snapshot is overwritten.

Pre-backup Hooks

Shell commands run inside the running pod before Restic copies files. Hot mode only.

# MySQL
mysqldump -u root -p$MYSQL_ROOT_PASSWORD --all-databases > /var/lib/mysql/dump.sql
# PostgreSQL
pg_dumpall -U postgres > /var/lib/postgresql/data/dump.sql
# Redis
redis-cli BGSAVE && sleep 2

If the hook exits non-zero, the backup is aborted. Hub templates may pre-fill hooks via backup_hooks metadata.

Storage Requirements

Both NFS and S3-compatible storage connections are supported:

• NFS — on-premises or LAN backups, fast, no egress cost
• S3 — off-site backups, supports AWS S3, MinIO, SeaweedFS, Backblaze B2

Each policy creates its own Restic repository at a unique path:

backups/{cluster}/{workload}-{suffix}/

The unique suffix ensures that deleting and recreating a policy for the same workload produces a fresh Restic repository, avoiding password mismatches with leftover encrypted data. Repositories are initialized automatically with a random password stored as a Kubernetes Secret.

Compose Stack Backups

Compose stacks with multiple shared volumes are handled automatically:

• Per-PVC Jobs — each PVC is backed up by its own Kubernetes Job. Required because local-path volumes are pinned to specific nodes, and different PVCs may live on different nodes.
• Staggered schedules — CronJobs are staggered by 2 minutes (:00, :02, :04). The first job initializes the Restic repository; subsequent jobs wait for the repo before starting.
• Single snapshot — per-PVC jobs are grouped into one logical snapshot, marked successful only when all jobs complete.

No special configuration is needed. PodWarden discovers all PVCs and creates per-PVC CronJobs when you apply the policy.

Kubernetes Resources

Each backup policy creates:

• CronJob(s) pw-backup-{workload}-v0, -v1, etc. One per PVC with staggered schedules.
• Secret pw-backup-{suffix} — Restic password, generated once at policy creation.
• Secret for S3 credentials (if applicable) from the storage connection.
• ServiceAccount podwarden-backup with RBAC for Deployment reads and pod exec.

Jobs use --retry-lock for safe concurrent access to the Restic repository.

Troubleshooting

Stuck in running: Jobs missing from K8s after 15 minutes are marked failed. Click refresh to sync.

Pre-backup hook fails: Verify the command and that the pod has the required tools.

Restore fails: Check namespace and PVC names.

Password mismatch / ciphertext verification failed: The repo was initialized with a different password. Delete the policy, create a new one (fresh repo path and password), apply, and run a backup.