NovaCloud-Hosting - Notice history

On the morning of April 23, a brief utility power failure occurred at the EYG1 data center. One of the Uninterruptible Power Supply (UPS) units failed to engage, causing a total loss of power on one of the power feeds. Servers equipped with only a single Power Supply Unit (PSU) connected to this feed experienced an immediate powerloss. While most systems recovered automatically, the host RYZEN-07-VHOST-EYG1 suffered a secondary hardware issue that required manual intervention.

Root Cause Analysis

1. Power Feed Failure: An external power dip was not mitigated by a specific UPS system, leading to a localized blackout on one power feed.

2. Lack of Redundancy: Impacted Ryzen-based servers utilized non-redundant power configurations. Consequently, the loss of a single feed resulted in an immediate power loss for these machines.

3. Hardware Damage: The abrupt shutdown triggered a hardware-level fault on host RYZEN-07-VHOST-EYG1, preventing it from rebooting autonomously.

Incident Timeline

• 09:33 AM: Monitoring alerts triggered for multiple host systems at EYG1. Investigation initiated.

• 09:41 AM: Issue identified as a power feed failure. Most affected servers were already successfully restarted and entered monitoring status.

• 09:56 AM: All systems recovered except for RYZEN-07-VHOST-EYG1. An on-site technician was dispatched to address the hardware failure.

• 10:11 AM: Hardware repair completed. RYZEN-07-VHOST-EYG1 initiated startup, and all hosted VPS returned to online status shortly thereafter.

RYZEN-07-VHOST-EYG1 is now starting up. All VPS on it should be online within the next 5 minutes.

RYZEN-07-VHOST-EYG1 is currently still unreachable. A technitian has already been informed about this incident.

All affected servers have been started. It seems to be a power-related issue on one of the power feeds, which affected half the servers with non-redundant PSUs. Therefore only AMD Ryzen VPS in Netherlands were partially affected.

If your server is still unreachable, please create a support ticket so we can address the issue.

Everything is running stable and as expected. Reverting A system update has fixed a bug, which has caused the issue.

We are sorry for the inconvenience caused by this incident and the high resolution time.

Thanks for your understanding and continous trust!

All VPS are active again. We are further monitoring the hostsystem to ensure stable operation.

We are continuing to work on a fix for this incident. All VMs should be starting within the next 15 minutes.

XEON03-VHOST seems to have a defective RAM module, which has caused the system reset as the server automatically disabled the affected RAM module. We will replace it in the coming days, which will require a planned maintenance.

All VPS, including Storage-VPS should now be back online. If you still experience issues, please immediately create a ticket on our dashboard and report it to our support team, so we can resolve it quickly.

We are sorry for any inconvenience caused by this incident.

Most VPS have been started, Storage-VPS on this hostsystem will take around 5-10 minutes until they are reachable again. Thanks for your patience.

XEON03-VHOST had temporarily lost power on both power feeds.

Around 75% of VPS on XEON01-VHOST-EYG1 were affected by this outage.

Due to a software bug, a backup task for a large VPS was incorrectly written to the primary NVMe SSD pool instead of the designated backup storage. The backup file grew to approximately 4 TB, while the NVMe pool had only around 3 TB of free capacity available. This caused the storage pool to reach 100% utilization.

Once the primary storage was full, affected virtual machines were no longer able to write to disk. This resulted in widespread I/O errors, and VM crashes.

As soon as the issue was identified, we immediately removed the incorrectly created backup and began force-restarting all affected VPS instances to restore services as quickly as possible.

Preventive Measures

We have implemented safeguards to ensure this cannot happen again:

• Backups can no longer be created on the primary NVMe storage pool under any circumstances.

• Additional validation has been added to prevent incorrect backup targets.

• We are rolling out improved disk usage monitoring and alerting across all host systems to ensure our team is notified before storage reaches critical levels.

We sincerely apologize for this incident.
If you are still experiencing any issues caused by this outage, please open a support ticket immediately so we can investigate and resolve it as quickly as possible.

All VPS have been started. In case your VPS is not working within the next 5 minutes, please create a ticket.

We have found the underlying issue and have fixed the core problem. All VPS should be starting up in 5-10 minutes.

Due to a DDoS Attack bypassing our Anti-DDoS Filters, the NIC of EPYC01-VHOST was not able to handle the amount of packets, which caused in packetloss and decreased network performance. Unfortunately our monitoring did not immediately report any issues to us. At 10:30 AM GMT+2, our team noticed packetloss on EPYC01-VHOST.

At 10:45 AM GMT+2, our team implemented changes to mitigate the issue. We increased ring-buffers on the NIC, increased the max conntrack and contacted our Upstream to mitigate the Attack. After the changes were made, we saw restoration of normal traffic levels and packetloss went back to 0%.

We are suspecting that the DDoS Attack was able to bypass the Anti-DDoS Filters, since some of the traffic was originating from an internal DDoS.

To prevent this issue from occuring in the future, we will apply these changes to all Hostsystems.