NovaCloud-Hosting - XEON-01-VHOST HDD-Pool inaccessible – Incident details

All systems operational

XEON-01-VHOST HDD-Pool inaccessible

Resolved
Major outage
Started 7 months agoLasted 16 days

Affected

SkyLink Data Center

Degraded performance from 11:15 AM to 1:40 PM, Operational from 11:15 AM to 11:35 PM, Degraded performance from 11:35 PM to 11:41 PM, Operational from 11:35 PM to 11:41 PM, Partial outage from 11:41 PM to 12:26 AM, Operational from 12:26 AM to 10:37 PM

Root-Server Vhosts EYG1

Degraded performance from 11:15 AM to 1:40 PM, Operational from 1:40 PM to 11:35 PM, Degraded performance from 11:35 PM to 11:41 PM, Partial outage from 11:41 PM to 12:26 AM, Operational from 12:26 AM to 10:37 PM

XEON-01-VHOST

Degraded performance from 11:15 AM to 1:40 PM, Operational from 1:40 PM to 11:35 PM, Degraded performance from 11:35 PM to 11:41 PM, Partial outage from 11:41 PM to 12:26 AM, Operational from 12:26 AM to 10:37 PM

Game-Server Nodes EYG1

Operational from 11:15 AM to 11:41 PM, Partial outage from 11:41 PM to 12:26 AM, Operational from 12:26 AM to 10:37 PM

Generic-01

Operational from 11:15 AM to 11:41 PM, Partial outage from 11:41 PM to 12:26 AM, Operational from 12:26 AM to 10:37 PM

Proxmox-Backup-Servers

Operational from 11:15 AM to 11:35 PM, Major outage from 11:35 PM to 12:26 AM, Operational from 12:26 AM to 10:37 PM

Updates
  • Resolved
    Resolved

    We have completed the maintenance and resolved the issue with a more reliable Storage-Layout.

    The setup is tested to ensure that this issue won't occur in the future. Thanks for your understanding.

    All affected customers (XEON Root-Servers + Standard Game-Servers + Storage Root-Servers) can open a ticket to request a free service extension of 5 days.

  • Monitoring
    Monitoring
    We implemented a fix and are currently monitoring the result.
  • Identified
    Identified

    We have made some changes and are currently rebooting the hostsystem in order to apply the changes.

    Additionally, the NVMe cache will be disabled to test whether this issue is caused by the caching method we are currently using.

  • Investigating
    Investigating

    Any IO operations on the HDD pool are not possible again. We are currently investigating the issue and trying to get the HDD pool operational.

  • Monitoring
    Monitoring

    A routine Raid-Check resulted in a bug which locked up read/write operations.

    We have managed to restore normal operation, but with degraded speed of the HDD-Pool in order to prevent another issue. As soon as we have found a solution, full speed will be restored.

    MAINTENANCE: https://novacloud.instatus.com/cm9e9zh6o00nl79u2h3ibnqhn

  • Investigating
    Investigating

    Some servers cannot be started and the Disk-IO of the HDD-pool is very low. We are currently investigating the cause for this issue.