NovaCloud-Hosting - Multiple Servers at EYG1 location unreachable – Incident details

On the morning of April 23, a brief utility power failure occurred at the EYG1 data center. One of the Uninterruptible Power Supply (UPS) units failed to engage, causing a total loss of power on one of the power feeds. Servers equipped with only a single Power Supply Unit (PSU) connected to this feed experienced an immediate powerloss. While most systems recovered automatically, the host RYZEN-07-VHOST-EYG1 suffered a secondary hardware issue that required manual intervention.

Root Cause Analysis

1. Power Feed Failure: An external power dip was not mitigated by a specific UPS system, leading to a localized blackout on one power feed.

2. Lack of Redundancy: Impacted Ryzen-based servers utilized non-redundant power configurations. Consequently, the loss of a single feed resulted in an immediate power loss for these machines.

3. Hardware Damage: The abrupt shutdown triggered a hardware-level fault on host RYZEN-07-VHOST-EYG1, preventing it from rebooting autonomously.

Incident Timeline

• 09:33 AM: Monitoring alerts triggered for multiple host systems at EYG1. Investigation initiated.

• 09:41 AM: Issue identified as a power feed failure. Most affected servers were already successfully restarted and entered monitoring status.

• 09:56 AM: All systems recovered except for RYZEN-07-VHOST-EYG1. An on-site technician was dispatched to address the hardware failure.

• 10:11 AM: Hardware repair completed. RYZEN-07-VHOST-EYG1 initiated startup, and all hosted VPS returned to online status shortly thereafter.

RYZEN-07-VHOST-EYG1 is now starting up. All VPS on it should be online within the next 5 minutes.

RYZEN-07-VHOST-EYG1 is currently still unreachable. A technitian has already been informed about this incident.

All affected servers have been started. It seems to be a power-related issue on one of the power feeds, which affected half the servers with non-redundant PSUs. Therefore only AMD Ryzen VPS in Netherlands were partially affected.

If your server is still unreachable, please create a support ticket so we can address the issue.