Configuration, deployment, commands, alerting, and log integration. Get your first backup running in under 10 minutes.
Get your first backup running in under 10 minutes.
# Extract the archive received after contract signing
cd backup-framework
pip install -r requirements.txt
Copy the example environment file and fill in your credentials:
cp .env.example .env
# Edit .env with your S3 and MongoDB credentials
Register your S3 bucket credentials once. The name is a local alias you choose.
python3 -m backup --register-bucket \
--name my-bucket \
--bucket prod-backups \
--access-key <ACCESS_KEY> \
--secret-key <SECRET_KEY> \
--region us-east-1
For a non-AWS endpoint (MinIO, Cloudflare R2, Wasabi…) add --endpoint https://your.endpoint.
Register a named job that binds a directory to a bucket.
python3 -m backup --register \
--name daily-backup \
--dir /path/to/your/data \
--bucket my-bucket
python3 -m backup --run --job daily-backup
# Check what was backed up python3 -m backup status --job daily-backup # Run a full audit (3 checks) python3 -m backup audit --job daily-backup
--dry-run on the first scan to preview what would be uploaded without making any changes.
| Dependency | Version | Notes |
|---|---|---|
| Python | 3.10+ | 3.12+ recommended |
| MongoDB | 4.0+ | Atlas or local; optional for S3-only restore |
| S3-compatible storage | Any | AWS S3, MinIO, Cloudflare R2, Wasabi… |
| RAM | 2 GB min | 4 GB+ recommended for large jobs |
| CPU | 2+ cores | For multi-threaded uploads |
All configuration is via environment variables (or a .env file). No credentials are ever hardcoded.
| Variable | Description |
|---|---|
| S3_BUCKET | Target S3 bucket name |
| S3_ACCESS_KEY | S3 access key (Fernet-encrypted at rest) |
| S3_SECRET_KEY | S3 secret key (Fernet-encrypted at rest) |
| S3_ENDPOINT_URL | Optional — for non-AWS S3-compatible endpoints |
| MONGO_URI | MongoDB connection string |
| MONGO_DB | MongoDB database name |
| FERNET_KEY | Key used to encrypt stored S3 credentials |
| LOG_FILE | Optional — write JSON logs to a rotating file |
| BANDWIDTH_LIMIT | Optional — e.g. 10MB/s, 500KB/s |
| WORKERS | Number of upload worker threads (default: 4) |
| COMPRESSION_LEVEL | gzip level 1–9, or 0 to disable |
| SECRETS_PROVIDER | Optional — aws, azure, gcp, ibm, vault, openbao |
S3 credentials are never stored in plaintext. Generate a key once and store it securely (or pull it from your secrets provider):
python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
Key rotation is supported via MultiFernet — see the full setup guide for the rotation procedure.
# From the project directory received after contract signing cp .env.example .env # fill in credentials docker compose up -d # Web dashboard at https://localhost # HTTP automatically redirects to HTTPS
[Unit] Description=Arkhivio daily backup [Service] Type=oneshot EnvironmentFile=/etc/arkhivio/.env ExecStart=/usr/bin/python3 -m backup scan upload audit [Install] WantedBy=multi-user.target
# /etc/cron.d/arkhivio — daily at 02:00 0 2 * * * backup-user /usr/bin/python3 -m backup scan upload audit --job daily
| Command | Description |
|---|---|
| scan | Walk directories, compute CRC32, detect changed files |
| upload | Upload new/modified files to S3 |
| audit | Run 3-check integrity verification |
| restore | Restore files from S3 (MongoDB-assisted) |
| restore-s3only | Restore directly from S3, no MongoDB required |
| status | Show last run status and metrics |
| history | List historical runs for a job |
| reconcile | Detect drift between MongoDB and S3 |
| delete | Remove a file from S3 and MongoDB |
| list | List backed-up files for a host/job |
| Flag | Description |
|---|---|
| --job NAME | Job name (used as MongoDB collection key) |
| --dir PATH [PATH…] | One or more directories to back up |
| --host NAME | Override hostname (useful for Kubernetes pods) |
| --dry-run | Preview changes without uploading |
| --workers N | Number of parallel upload threads |
| --throttle RATE | Bandwidth limit, e.g. 10MB/s |
A single job can back up multiple directories. All paths are tracked under one job, use one set of metrics, and upload in parallel.
python3 -m backup scan upload \
--job full-server-backup \
--dir /etc /var/www /home/deploy /opt/app
Global exclude patterns apply across all directories. Existing single-directory jobs require no changes — fully backward compatible.
Pod names are ephemeral and change on every restart. Use --host to set a stable
identity so all runs for a workload are grouped correctly in MongoDB and S3:
python3 -m backup scan upload audit \
--job k8s-backup \
--host my-deployment \
--dir /data
--host to the Deployment or StatefulSet name, not the pod name, so S3 paths stay stable across pod restarts.
Because all events are JSON, any log aggregator can generate alerts by filtering on event field values.
No special agent or plugin required.
| Event | Condition |
|---|---|
| upload_error | Any file backup failure |
| delete_s3_error | S3 delete failure |
| reconcile_missing_s3 | > 5 missing objects in 1 hour |
| Event | Condition |
|---|---|
| upload_slowdown | S3 rate-limiting > 10 times in 1 hour |
| scan_stat_error | stat() failures > 100 in 1 hour |
| secrets_load_failed | Cloud secrets provider unreachable |
| mongo_tls_disabled | TLS disabled for non-localhost MongoDB |
| (absence) | No upload_complete event in > 25 hours |
- alert: ArkhivioUploadError expr: | count_over_time( {app="arkhivio"} | json | event="upload_error" [5m]) > 0 for: 0m labels: severity: critical annotations: summary: Arkhivio upload failure detected
Route alerts to: Slack, Microsoft Teams, PagerDuty, OpsGenie, email, or SMS — using your existing alerting pipeline.
Set LOG_FILE=/var/log/arkhivio/backup.log to write to a rotating file
(50 MB per file, 10 files retained) in addition to stdout.
Both handlers emit identical newline-delimited JSON.
Key events emitted by the framework (all include ts, event, level, job, host):
| Event | Level | Description |
|---|---|---|
| job_started | info | Job begun; includes step, host |
| file_scanned | info | File fingerprinted; path, size, crc32, status |
| scan_complete | info | Summary: new, modified, unchanged, deleted counts |
| file_uploaded | info | S3 PUT succeeded; key, bytes, crc32 |
| upload_error | error | S3 PUT failed; path, error message |
| upload_slowdown | warn | S3 rate-limit received; will retry after backoff |
| upload_complete | info | Job summary: uploaded, skipped, errors, bytes |
| audit_check | info | Single check result: check name, status, details |
| audit_complete | info | All checks done: passed, total |
| file_restored | info | File downloaded and CRC32 verified |
| restore_complete | info | Summary: files, bytes, errors, crc32_verified |
| reconcile_missing_s3 | error | File marked synced but absent from S3 |
| secrets_load_failed | error | Cloud secrets provider unreachable |
No per-seat fees. No per-server royalties. No usage-based billing.
The source code is delivered when you sign the contract.