A five-stage pipeline from file scan to verified restore. Each stage is observable, independently auditable, and emits structured JSON events.
Start simple — the full detail is below.
MongoDB tracks state about the backup. S3 holds the actual backup data. This separation is intentional — the data is accessible independently of the metadata store.
{host}/{path}x-amz-meta-crc32files.json.gz inventory per run
scanner.py walks one or more source directories. For each file it records the
modification time, size, and a CRC32 checksum computed from file bytes.
By comparing against the last MongoDB snapshot, files are classified as new,
modified, unchanged, or deleted.
Only new and modified files proceed to the upload stage.
{"event":"file_scanned", "path":"/data/records.db","size":10485760, "crc32":"c0ffee12","status":"new"} {"event":"scan_complete", "new":8,"modified":0,"unchanged":142,"deleted":0}
uploader.py runs a configurable thread pool (default 4 workers) and uploads only the
files identified in the scan. Before each upload, files are optionally compressed with gzip
(levels 1–9, with a configurable size threshold — default 500 MB).
A token-bucket rate limiter enforces an aggregate bandwidth cap across all workers.
When S3 responds with SlowDown or TooManyRequests, the framework
waits 30 seconds and retries — up to 5 total attempts per file.
After each successful run, a compressed files.json.gz inventory is written to the bucket.
Each file lands at {hostname}/{absolute/path} — a path you can reconstruct by hand.
CRC32 and compression flag are stored in standard S3 object metadata
(x-amz-meta-crc32, x-amz-meta-compressed).
Because the path is deterministic and metadata is standard, you can inspect, download, or restore
any file using any S3-compatible client — aws s3, rclone, MinIO Console —
with no Arkhivio involvement required.
{host}/{/abs/path}x-amz-meta-crc32files.json.gz per runlatest.json in S3
audit.py performs three independent verification checks after each backup run.
Each check is separately observable in the log stream and can trigger separate alerts.
x-amz-meta-crc32 on the latest S3 object matches the CRC32 stored in MongoDBA reconciler can also detect structural drift — files marked as synced in MongoDB that are absent from S3, or S3 objects with no corresponding MongoDB record.
restore.py downloads files from S3 and verifies CRC32 on each before writing to disk.
Both full and selective (single file or subtree) restore are supported, with configurable
worker count and optional path remapping.
For disaster recovery, restore_onlys3.py works directly against the bucket —
no MongoDB connection required. It reads the files.json.gz inventory and
x-amz-meta-crc32 metadata from S3 itself, then downloads and verifies each file.
No Arkhivio application installation required for this recovery path.
Three deployment models supported out of the box. No agent to install. No daemon required.
The simplest model. Add a cron entry, point it at a directory, done. Structured JSON logs to stdout or a rotating log file.
# /etc/cron.d/arkhivio 0 2 * * * root python3 -m backup \ scan upload audit
Timer-based service with systemd journal integration.
Logs flow to journald with structured field support.
[Service] Type=oneshot EnvironmentFile=/etc/arkhivio/.env ExecStart=/usr/bin/python3 \ -m backup scan upload audit
Full Docker Compose stack with nginx reverse proxy, automatic HTTP→HTTPS redirect, and TLS termination. Web dashboard included.
$ docker compose up -d # Dashboard at https://localhost # HTTP redirects to HTTPS
No per-seat fees. No per-server royalties. No usage-based billing.
The source code is delivered when you sign the contract.
Follow the Quick Start and have your first backup running in under 10 minutes.