Postmortem: AMD Ryzen vHosts Network Connectivity Issues

Incident Start: 31 August 2025, 9:00 PM CEST
Incident End: 31 August 2025, 9:22 PM CEST
Duration: Approximately 22 minutes
Impact: Major outage on Ryzen vHosts, particularly RYZEN-05 and RYZEN-08

Summary

On 31 August 2025, a router misconfiguration caused a significant network outage affecting multiple Ryzen-based vHosts and services. The issue began at 9:00 PM CEST and was fully resolved by 9:22 PM CEST.

Impact

• RYZEN-05 and RYZEN-08 experienced full network loss for about 15 minutes.

• All other Ryzen vHosts were briefly disconnected for approximately 10 seconds during the network recovery process.

• Affected services included:

  • DB01 - Game-Cloud Infrastructure

  • Ryzen Game-Server Nodes (EYG1)

  • AMD Ryzen/High-End Root servers

All customers with services on RYZEN-05 and RYZEN-08 are eligible to receive 2 days of additional runtime credit. Please open a support ticket to request this credit.

Root Cause

A router misconfiguration combined with an automation error caused connectivity issues on RYZEN-05 and RYZEN-08. Full restoration required reapplying routing configuration across all nodes, resulting in a brief interruption to all Ryzen vHosts.

Resolution

• A temporary automation fix was deployed to prevent recurrence of this specific issue.

• Manual configuration changes restored full network connectivity by 9:22 PM CEST.

Preventive Actions

1. Implement per-node VM connectivity monitoring for faster detection and resolution.

2. Improve automation workflows to avoid similar misconfigurations.

3. Add validation checks to prevent faulty network configurations from being deployed.

We apologize for the disruption and appreciate your patience as we implement additional safeguards to strengthen our network stability.

Summary,

Between 20:29 and 20:32 GMT+2, several systems became temporarily unreachable due to a power distribution issue at the datacenter. The incident impacted specific power feeds and affected servers equipped with only a single power supply unit (PSU).

Affected Systems,

• Virtual Servers: RYZEN-VHOST 01, 03, 04, 05,

• Dedicated Servers: A small number of systems with single-PSU configurations,

• Unaffected: The majority of our infrastructure remained fully operational, including all dual-PSU systems,

Root Cause,

The incident was caused by a fault in one of the datacenter’s internal power subdistribution units. This resulted in a brief power loss on a specific feed.

Resolution,

Power was automatically restored by the datacenter within less than a minute. All affected systems came back online shortly afterward. We have confirmed that no data loss occurred.

We are sorry for any inconvenience caused by this incident. Please open a ticket immediately if you still experience an outage, so we can address the issue.

There was a powerloss on one of the power feeds. It seems that the power has been restored within a few minutes and we are awaiting a statement from the datacenter to what has been affected and why.

All affected Dedicated Servers and Root-Servers have already been started.

Summary

On July 17th, 2025, starting around 05:30 GMT+2, we experienced a partial connectivity issue affecting traffic to the endpoint 5.83.150.1. A portion of external networks was unable to reach some prefixes including 5.83.150.0/24, while others remained fully functional.

Initial observations pointed towards a potential BGP routing problem, but external route propagation appeared fully intact across all monitors.

Root Cause

The issue was caused by a malfunctioning SmartMitigate DDoS filtering node within the upstream provider's infrastructure. The affected host had stopped forwarding traffic.

As a result, incoming traffic that was routed through this specific mitigation node was effectively dropped, leading to asymmetric reachability depending on which SmartMitigate host was assigned.

This incident was not related to our own BGP announcements or configurations.

Resolution

The upstream provider identified the faulty SmartMitigate host and removed it from operation. Immediately afterward, traffic flows normalized and full connectivity to the affected prefix was restored. This occurred around 09:20 GMT+2.

Impact Overview

• Affected Prefix: 5.83.150.0/24

• Root Cause: Faulty SmartMitigate (DDoS filter) node

• BGP Status: Unaffected – all routes remained correctly announced and visible

• Duration: ~3 hours 50 minutes

• Impact: Partial traffic loss (limited to traffic routed via the affected SmartMitigate node)

Current Status

Connectivity has been fully restored. The root cause has been addressed at the upstream provider level. No further disruptions are expected related to this issue.

The issue has been identifies and is caused by the Dark Fiber connection between Skylink and Amsterdam, which is experiencing a failure. The traffic is mostly being re-routed via Frankfurt, but some routes are still going via Amsterdam, causing some packetloss and higher latency.

The network connectivity is mostly restored to normal latencies and with less packetloss. Our Upstream provider is currently working on a solution for this issue.