Scheduled for 25 January, 2025 at 11:00 AM – 26 January, 2025 at 1:55 AM
Affects
Maincubes-FRA01
Under maintenance from 11:00 AM to 1:55 AM
IP-Transit Router
Under maintenance from 11:00 AM to 1:55 AM
Router-01
Under maintenance from 11:00 AM to 1:55 AM
Router-02
Under maintenance from 11:00 AM to 1:55 AM
Update
26 January, 2025 at 1:55 AMCompleted
26 January, 2025 at 1:55 AMAfter several days of intensive debugging and testing, we have come up with a rather ingenious workaround for the Wireguard-Bug. this means that no configuration changes are necessary on the customer side and the problem of wireguard simply changing the endpoint ip is finally history.
thanks for your patience.
Completed
25 January, 2025 at 8:33 PMCompleted
25 January, 2025 at 8:33 PMThe tunnels have been running mostly stable again for some time and no further outages are to be expected.
The interventions resulted in a downtime of approx. 38 minutes for IPv6 traffic, approx. 32 minutes for IPv4 traffic and approx. 1 hour for wireguard in general.
VRRP is now stable again. In order to completely fix the Wireguard error, further measures are necessary on the customer side, they will be informed about this (it only affects a very small group of customers).
As a result of the downtime, each customer has been allocated 3 days extra time, yes, this has already been done.
Update
25 January, 2025 at 12:58 PMIn progress
25 January, 2025 at 12:58 PMit is currently stable, but not everything is implemented yet, but we will not make any other changes for the time being, as filters are unexpectedly in our way.
Update
25 January, 2025 at 12:11 PMIn progress
25 January, 2025 at 12:11 PMthe spin seems to be out, arp was filtered incorrectly. now only the persistence is checked.
Update
25 January, 2025 at 12:05 PMIn progress
25 January, 2025 at 12:05 PMUnfortunately VRRP is still a bit bitchy, but it should be fixed soon.
In progress
25 January, 2025 at 11:01 AMIn progress
25 January, 2025 at 11:01 AMWe start with the implementation.
Planned
25 January, 2025 at 11:00 AMPlanned
25 January, 2025 at 11:00 AMIn the last 24+ hours we have noticed an error that VRRP is briefly interrupted during configuration reloading. this generates minimal packetloss.
At the same time we are fixing a bug where some wireguard clients simply change the endpoint (which was set statically) to the secondary address of the routing servers when a VRRP “BACKUP” event occurs (e.g. because we need to migrate load or because a router fails or currently because of a config reload, which we also fix see above). This is quite rare but will be fixed now.
This update is quite important to achieve a far-reaching stability and therefore planned for the short term.
We apologize for the inconvenience, but we want to maximize uptime and stability for all with this.
Short interruptions are to be expected during implementation.
