NovaCloud-Hosting - Notice history

Everything is running stable and as expected. Reverting A system update has fixed a bug, which has caused the issue.

We are sorry for the inconvenience caused by this incident and the high resolution time.

Thanks for your understanding and continous trust!

All VPS are active again. We are further monitoring the hostsystem to ensure stable operation.

We are continuing to work on a fix for this incident. All VMs should be starting within the next 15 minutes.

XEON03-VHOST seems to have a defective RAM module, which has caused the system reset as the server automatically disabled the affected RAM module. We will replace it in the coming days, which will require a planned maintenance.

All VPS, including Storage-VPS should now be back online. If you still experience issues, please immediately create a ticket on our dashboard and report it to our support team, so we can resolve it quickly.

We are sorry for any inconvenience caused by this incident.

Most VPS have been started, Storage-VPS on this hostsystem will take around 5-10 minutes until they are reachable again. Thanks for your patience.

XEON03-VHOST had temporarily lost power on both power feeds.

Around 75% of VPS on XEON01-VHOST-EYG1 were affected by this outage.

Due to a software bug, a backup task for a large VPS was incorrectly written to the primary NVMe SSD pool instead of the designated backup storage. The backup file grew to approximately 4 TB, while the NVMe pool had only around 3 TB of free capacity available. This caused the storage pool to reach 100% utilization.

Once the primary storage was full, affected virtual machines were no longer able to write to disk. This resulted in widespread I/O errors, and VM crashes.

As soon as the issue was identified, we immediately removed the incorrectly created backup and began force-restarting all affected VPS instances to restore services as quickly as possible.

Preventive Measures

We have implemented safeguards to ensure this cannot happen again:

• Backups can no longer be created on the primary NVMe storage pool under any circumstances.

• Additional validation has been added to prevent incorrect backup targets.

• We are rolling out improved disk usage monitoring and alerting across all host systems to ensure our team is notified before storage reaches critical levels.

We sincerely apologize for this incident.
If you are still experiencing any issues caused by this outage, please open a support ticket immediately so we can investigate and resolve it as quickly as possible.

All VPS have been started. In case your VPS is not working within the next 5 minutes, please create a ticket.

We have found the underlying issue and have fixed the core problem. All VPS should be starting up in 5-10 minutes.

Due to a DDoS Attack bypassing our Anti-DDoS Filters, the NIC of EPYC01-VHOST was not able to handle the amount of packets, which caused in packetloss and decreased network performance. Unfortunately our monitoring did not immediately report any issues to us. At 10:30 AM GMT+2, our team noticed packetloss on EPYC01-VHOST.

At 10:45 AM GMT+2, our team implemented changes to mitigate the issue. We increased ring-buffers on the NIC, increased the max conntrack and contacted our Upstream to mitigate the Attack. After the changes were made, we saw restoration of normal traffic levels and packetloss went back to 0%.

We are suspecting that the DDoS Attack was able to bypass the Anti-DDoS Filters, since some of the traffic was originating from an internal DDoS.

To prevent this issue from occuring in the future, we will apply these changes to all Hostsystems.

Traffic levels have returned to normal and we are monitoring the situation.