NovaCloud-Hosting - FFM2 outage – Incident details

All systems operational

FFM2 outage

Resolved
Major outage
Started 26 days agoLasted 19 days

Affected

NBG Datacenter

Major outage from 8:05 AM to 10:28 AM, Operational from 10:28 AM to 3:43 PM, Partial outage from 3:43 PM to 5:48 PM, Operational from 3:43 PM to 2:00 PM

VPS Hypervisor FFM2

Major outage from 8:05 AM to 10:28 AM, Operational from 10:28 AM to 3:43 PM, Partial outage from 3:43 PM to 5:48 PM, Operational from 3:43 PM to 2:00 PM

RYZEN-01-VHOST-DE Gen-3

Major outage from 8:05 AM to 10:28 AM, Operational from 10:28 AM to 2:00 PM

RYZEN-02-VHOST-DE Gen-3

Major outage from 8:05 AM to 10:28 AM, Operational from 10:28 AM to 3:43 PM, Partial outage from 3:43 PM to 5:48 PM, Operational from 5:48 PM to 2:00 PM

RYZEN-03-VHOST-DE Gen-3

Major outage from 8:05 AM to 10:28 AM, Operational from 10:28 AM to 2:00 PM

RYZEN-04-VHOST-DE Gen-3

Major outage from 8:05 AM to 10:28 AM, Operational from 10:28 AM to 2:00 PM

Updates
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    All IP-Transit and High-End VPS DE services are currently fully reachable.

    We are in contact to our infrastructure provider at the FFM2 location to find a solution for our Dedicated Server and Standard VPS DE customers. If you have a Standard VPS DE, you can contact our team at support@novacloud-hosting.com

  • Update
    UTC
    Update

    PletX seems to be blocking GRE packets from multiple source IPs. If you experience any issues with your IP-Transit services, please send us a ticket to support@novacloud-hosting.com, so we can gather more information.

    We are currently waiting for a solution from our Upstream's NOC.

    High-End VPS DE are also partially affected.

  • Monitoring
    UTC
    Monitoring

    All GRE, GRETAP and VXLAN Tunnels are now reachable again. Our team is still working on restoring Wireguard Tunnel connectivity.

    The issue was caused by false filtering of PletX, which dropped TCP connections.

  • Update
    UTC
    Update

    IP-Transit and High-End VPS DE are now online. Regarding Dedicated Servers and Standard VPS DE, we are currently waiting for a resolution from our infrastructure provider at the NTT-FRA1 datacenter.

    Some customers are still reporting issues with TCP connections or Tunnels not working.

  • Identified
    UTC
    Identified

    Since we currently don't have an ETA for until FFM2 is normally available again, we are currently working on restoring the IP-Transit services at another location.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.