NovaCloud-Hosting - EPYC01-VHOST Packet loss – Incident details

EPYC01-VHOST Packet loss

Resolved
Degraded performance
Started about 2 months agoLasted about 1 hour

Affected

SkyLink Data Center

Operational from 9:00 AM to 10:22 AM

VPS Hypervisor EYG1

Operational from 9:00 AM to 10:22 AM

EPYC-01-VHOST

Operational from 9:00 AM to 10:22 AM

Updates
  • Postmortem
    Postmortem

    Due to a DDoS Attack bypassing our Anti-DDoS Filters, the NIC of EPYC01-VHOST was not able to handle the amount of packets, which caused in packetloss and decreased network performance. Unfortunately our monitoring did not immediately report any issues to us. At 10:30 AM GMT+2, our team noticed packetloss on EPYC01-VHOST.

    At 10:45 AM GMT+2, our team implemented changes to mitigate the issue. We increased ring-buffers on the NIC, increased the max conntrack and contacted our Upstream to mitigate the Attack. After the changes were made, we saw restoration of normal traffic levels and packetloss went back to 0%.

    We are suspecting that the DDoS Attack was able to bypass the Anti-DDoS Filters, since some of the traffic was originating from an internal DDoS.

    To prevent this issue from occuring in the future, we will apply these changes to all Hostsystems.

  • Resolved
    Resolved
    This incident has been resolved.
  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Investigating
    Investigating

    EPYC01-VHOST is currently experiencing high packetloss (10-20%) and degraded network performance. We are currently investigating this incident.