Best practices
Local versus remote SSDs
Kubernetes provides you with an option of employing remote disks using dynamic provisioning or local storage which has to be preprovisioned.
Local storage provides great performance, but the data is not replicated and can be lost if the pod is moved to a different node for maintenance operations, or if a node fails. Remote storage has slightly lower performance but the data is resilient to failures.
The following table summarizes the tradeoffs of local vs remote storage:
| Feature | Local SSD storage | Remote SSD storage |
|---|---|---|
| Provision large disk capacity per node | Depends on cloud-provider | Yes |
| Disk storage resilient to failures or pod movement | No | Yes |
| Performance - latency | Lower | Higher |
| Performance - throughput | Higher | Lower |
| Typical cost characteristics | Lower | Higher |
| Kubernetes provisioning scheme | Pre-provisioned | Dynamic provisioning |
While local storage offers higher throughput and lower latency at a lower cost, it comes with significant limitations. Maintenance operations like cluster node pool upgrades, or rolling operations to upgrade software or set flags can cause a pod to lose its local copy and require a full remote bootstrap of all local tablets from its peers. This can be time consuming for large databases. Further, for large clusters, there is always a non-zero risk of another pod movement happening during these maintenance operations, which would cause data loss for a subset of tablets. Therefore, using local storage is not recommended for most production use cases.