A developer hit 100% disk capacity on a Tuesday night while running pack_product.py, causing the script to die mid-write and corrupt the 39th product zip. The incident forced the deletion of 2 GB of raw render previews just to regain shell access, prompting a shift from ad-hoc backups to a structured archive routine. This approach prioritizes data integrity over storage efficiency, specifically designed for environments where the asset library exceeds local disk space.

Zip First, Copy Later

The author rejects expensive cloud syncs and larger disks in favor of zipping each finished product folder before it leaves the machine. A 30-vector pack of 22 MB compresses to roughly 18 MB, but the primary benefit is atomicity: a zip is a single file with one checksum. This ensures that if a USB copy or cloud sync fails halfway, the failure is immediate and obvious, rather than resulting in truncated JPGs discovered only after customer complaints.

Checksums Are Non-Negotiable

Simple copy commands like cp -r are insufficient because they can duplicate files while they are still being written. The author now mandates a SHA-256 checksum for every archive, using sha256sum _system/my-product.zip >> manifests/my-product.sha256. This practice uncovered two silent corruptions in a 856 MB library of 39 products. Without checksums, these errors would have remained invisible until a buyer encountered a broken file, making verification a critical, automated step rather than a manual hunt.

Optimizing for Low-Resource Environments

Running on a 1 vCPU, 2 GB RAM box without a GPU requires careful compression choices. The author uses zipfile with ZIP_DEFLATED for text-heavy assets but avoids deflating JPGs and PNGs, as these formats are already compressed and offer negligible savings. To prevent out-of-memory errors, the pipeline processes one archive at a time rather than fanning out parallel jobs. Memory usage is monitored via resource.getrusage(resource.RUSAGE_SELF).ru_maxrss, and gc.collect() is called between products to manage peak RSS.

The Restore Test

A backup is only valid if it can be restored. The author implements a monthly routine where a random zip is copied back, verified with sha256sum -c, and unzipped into a scratch folder. This two-minute process caught a failing USB drive during a scheduled test rather than during a customer refund. The strategy distinguishes between regenerable layers, which stay on the small disk, and finished packs, which are archived with strict integrity checks.

Key Takeaways

  • Zip finished products before copying to ensure atomic backups and easy corruption detection.
  • Generate SHA-256 checksums for every archive to catch silent failures during transfer.
  • Avoid parallel compression on low-resource servers to prevent memory thrashing and OOM kills.
  • Perform monthly random restore tests to validate backup integrity and hardware health.

The Bottom Line

Stop treating backups as an afterthought. Atomic zips, mandatory checksums, and monthly restore tests are the only way to guarantee your asset library survives a full disk or a failing drive.