Around 75% of VPS on XEON01-VHOST-EYG1 were affected by this outage.
Due to a software bug, a backup task for a large VPS was incorrectly written to the primary NVMe SSD pool instead of the designated backup storage. The backup file grew to approximately 4 TB, while the NVMe pool had only around 3 TB of free capacity available. This caused the storage pool to reach 100% utilization.
Once the primary storage was full, affected virtual machines were no longer able to write to disk. This resulted in widespread I/O errors, and VM crashes.
As soon as the issue was identified, we immediately removed the incorrectly created backup and began force-restarting all affected VPS instances to restore services as quickly as possible.
Preventive Measures
We have implemented safeguards to ensure this cannot happen again:
Backups can no longer be created on the primary NVMe storage pool under any circumstances.
Additional validation has been added to prevent incorrect backup targets.
We are rolling out improved disk usage monitoring and alerting across all host systems to ensure our team is notified before storage reaches critical levels.
We sincerely apologize for this incident.
If you are still experiencing any issues caused by this outage, please open a support ticket immediately so we can investigate and resolve it as quickly as possible.
