Since the workaround has been applied, the situation is stable, however, this involves our DDoS protection being degraded while we work on a permanent solution. IPv6 was entirely unaffected.

We have successfully implemented a fix and the Game-Panel is operational.

We have identified the root cause and will be using this maintenance to improve some other problems.

ETA: 30-60 Minutes

RYZEN-01-GH has successfully started and running stable.

A software bug has prevented us from starting the Game-Host. We have managed to fix the bug and RYZEN-01-GH is already starting. We expect all Game-Servers to be online within the next 5-10 minutes.

Ryzen-04 has somehow gone offline again. We are investigating the issue.

The PDU has been replaced and Ryzen-04 is online. All Root-Servers are online aswell.

The PDU is being replaced at the moment.

We have also fixed an issue in our Ryzen Cluster, which was caused by the outage. Altough it hasn't affected any customers.

The Fuse has blown again. It seems that the PDU is defective. A technician is taking further investigations.

The backup is back in place, and the host system is already booting. The root servers should be online in 5–10 minutes.

It looks like a fuse has blown. A technician has already been informed and is looking into the problem more closely.

We are sorry for any inconvenience caused by this incident.

Our ISP has implemented improved Filter rules, which should improve the network performance when creating/restoring backups.

the problem will continue to be circumvented with the static ndp entry as it has been for the last few months. no further problems are to be expected. we will intensively test and fix the issue in the background. All affected customers who have either IPv6 endpoints OR IPv6 addresses are entitled to a 3-day extension of the runtime by means of a ticket. We apologize for the problems, this is not our claim.

the problem has reappeared. we have again added a permanent NDP entry which had also caused problems before. this should keep the problem stable for the time being. compensation in the form of runtime for affected customers is being clarified internally.

it seems to be stable again without any changes on our part. We have contacted the ISP because it has often attracted negative attention.

We are continuing to work on a fix for this incident. NDP is flapping.

The problem could be traced back to a strange problem with NDP which we will analyze further. the main instance is now back in operation and we implemented a fix for it and checking the stability now.

The problem occurs specifically only on our main instance (1). we maintain this instance and try to fix the problem. until then the traffic runs via the 2nd backup instance without interruption.

We (seems like that) found the cause and are implementing a fix. after each reload of the network config, a problem arose whereby the routers no longer had a functioning ipv6 for several minutes

This incident has been resolved.

Update from ISP:

„On 23.11.2024 at 23:12 (CET), our network experienced a sophisticated attack directly targeting our edge routing infrastructure. Due to the complexity of the attack, initial debugging efforts mistakenly concluded that we were dealing with a severe hardware failure within our edge routing equipment and that redundancy mechanisms were not functioning as expected. After manually triggering failovers to standby hardware in an attempt to rule out defective components without success, we intentionally disconnected our network from the internet to isolate and restore components one by one. This approach proved effective, with services beginning to recover at 23:41. IPv6 connectivity was restored quickly, while the last IPv4 prefixes came back online at 23:55. Unfortunately, the attackers quickly adjusted their methods, temporarily bypassing the newly implemented filters. As a result, another brief IPv4 outage occurred between 00:22 and 00:35, while IPv6 connectivity remained stable since the first fix was implemented.

Since the incident occurred six hours ago, our team has been working through the night in collaboration with our upstream providers to implement permanent fixes against these new attack vectors. Although the network has faced further attacks in the meantime, it has remained stable, as our solutions have proven effective. Nevertheless, we remain on high alert over the coming hours and days to respond swiftly to any potential new attack patterns.

This incident marks the first outage of our entire edge routing infrastructure, impacting every single dataforest customer, including IP transit services across all datacenters, since the launch of AS58212 nearly five years ago. Such an outage falls far short of the standards we set for ourselves, and we deeply apologize for the disruption caused to our customers. It is particularly frustrating that this attack succeeded despite our daily efforts to mitigate hundreds of similar attacks unnoticed. In this case, the attack managed to overwhelm our routers due to insufficient filtering against this unprecedented level of complexity.

Please note that this incident was purely a reachability issue. There was no power outage, no hacking attempt, and no data breach.“

Our ISP has updated the incident:

"Another fix has been implemented a few minutes ago and operation is stable since then. We are working on a permanent solution to avoid further outages. If any occur, we will report here."

Our ISP has deescalated the incident.

"We have implemented a fix. Network is stable for about five minutes now, IPv6 was not affected as much as IPv4 as remained available during the issue most of the time while IPv4 was affected heaviliy, leading to massive packet loss for a longer period of time. A post-mortem will follow tomorrow. Please rest assured we are 24/7 monitoring the status of our edge routing devices and will take action immediately if needed."

We are sorry for the inconvenience caused by this rare accident.

The network is online at the moment, but we are waiting for the confirmation of our ISP.

Update from ISP:
"We identified the issue and work on a solution"

Update from ISP:

"We currently experience a outage in our edge routing, and investigating the situation."