If you are running large language models on Kubernetes and every pod restart triggers a fresh download of 10 to 30 GB weights, you are burning bandwidth and waiting 10 to 15 minutes for cold starts. That is not a minor inconvenience; it is a fundamental architectural failure in how you treat model artifacts. Suresh Kumar’s latest deep dive on DEV.to tackles this exact pain point: decoupling heavy model weights from the ephemeral lifecycle of pods using Persistent Volumes (PV) and PersistentVolumeClaims (PVC).

The Three-Layer Storage Abstraction

Kubernetes storage is not magic, but it is layered. You have the StorageClass, which dictates the provisioning mechanismβ€”whether that is a local path, AWS EBS, or Ceph RBD. Then you have the PersistentVolume, the actual storage resource provisioned by an admin or dynamically. Finally, the PersistentVolumeClaim is what developers actually request. It binds to a matching PV, and your pods reference the claim, not the volume directly. This abstraction is critical for MLOps because it allows you to swap storage backends without rewriting your deployment manifests.

Access Modes Matter for Model Integrity

Do not treat all storage access the same. Vector databases generally require ReadWriteOnce (RWO) modes because concurrent writes from multiple nodes can corrupt write-ahead-log segments. However, sharing model weights across dozens of stateless inference pods is a classic use case for ReadOnlyMany (ROX). Write the weights once to a shared network filesystem or local path, then mount them read-only in every pod. This prevents the thundering herd problem of every pod trying to download the same Llama-3 or Mistral checkpoint simultaneously. Note that NFS generally supports ROX, while plain block disks often do not, so check your backend capabilities before committing.

The k3d Local Storage Gotcha

For those testing locally with k3d, there is a critical trap. k3d nodes are Docker containers, so a path like /opt/ai/models only exists inside the container if you explicitly mount the host folder when creating the cluster. If you skip the --volume /opt/ai/models:/opt/ai/models@agent:* flag, your PV points to an empty directory inside the container, and your pods will fail to find the weights. Kumar’s example shows the correct manifest for a local PV with nodeAffinity, ensuring Kubernetes knows exactly which nodes hold the physical files.

Key Takeaways

  • Ephemeral pods cause 10-15 minute cold starts when downloading 10-30GB model weights on every restart.
  • Use ReadOnlyMany (ROX) access modes for sharing model weights across multiple stateless pods.
  • Vector databases require ReadWriteOnce (RWO) to prevent write-ahead-log corruption.
  • In k3d environments, you must explicitly mount host folders into agent containers to make local PVs work.
  • PersistentVolumeClaims (PVCs) allow developers to request storage without knowing the underlying provisioning details.

The Bottom Line

Stop treating model weights like ephemeral application data. By leveraging Persistent Volumes with ReadOnlyMany access modes, you eliminate redundant downloads and stabilize inference latency, turning cold starts from minutes into seconds.