MXroute - Notice history

Lucy is back online. A full account of events should follow a bit later.

There's an admin dream team working on this box right now with us.

We are continuing to work on a fix for this incident.

With activity split between getting the production server to boot it's OS, and deploying a replacement server, suddenly JetBackup doesn't work on a new server. So we opened a ticket with JetBackup. Whatever supernatural entity is responsible for this incredible sequence of events, even you have to sleep eventually.

We're still working on the OS. During this process, we've been preparing a replacement server to restore backups to, as we'd like that process to be moving along just in case we need it. If it's time wasted, and we sure hope it is, it was a worthy cause.

By all accounts we should either be online or right on the edge of being online. But the OS is still giving us hell. You can vent in tickets if you want, and we get it, but you won't speed this up. If we had a magic button we'd have pressed it already.

Hardware replaced, now it's just a matter of getting the OS to deal with the hardware changes and boot properly.

Still working on this. Note there is a reason it says 'Operational' above and it's too long to explain here, but no service on the lucy.mxrouting.net server is online until the text of these updates says otherwise.

RAID controller replacement didn't help. Was it replaced wrong? Is it the riser card? Stay tuned to find out.

We are continuing to work with remote hands on the raid controller replacement.

Frustration is understood. We're waiting to hear back from remote hands at the datacenter. Here are some facts:

1. No inbound email will be missed, it will flow in after the server is back online.
2. Many reasonable redundancies were in place here. It is not common for raid controllers to suddenly fail.
3. This server is actively backed up, no matter what else happens.

We expect to pop in the new controller, import the raid, and work fine under the expectation that this hardware configuration is perfect. This being a statistical anomaly alone, we don't expect a repeat event.

The server is not detecting the RAID controller. We're going to swap out the controller with the spare.

The Lucy server is having trouble booting, so that's interesting. We're working on it.

This incident has been resolved.

Routing issue in Ashburn may be causing some issues. Understanding scope and reviewing mitigation options.

There was an issue with an upstream network provider. This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

There is currently a network issue at the datacenter, working to resolve.

This incident has been resolved.

There appears to be no available on-site staff to resolve the current issue with the longhorn.mxrouting.net server. The server may be offline for some time until we are able to get someone on-site to resolve the issue. Migration of this server to a new facility, to prevent this issue, will begin a few hours after that resolution.

We are continuing to work on a fix for this incident.

This incident has been resolved.

Intermittent packet loss over some routes may indicate unmitigated DDOS attack, continuing to investigate.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.