MXroute - Notice history

Support experiencing degraded performance

Billing Panel - Operational

100% - uptime
Jul 2023 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2023
Aug 2023
Sep 2023

DirectAdmin Panel - Operational

100% - uptime
Jul 2023 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2023
Aug 2023
Sep 2023

IMAP - Operational

100% - uptime
Jul 2023 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2023
Aug 2023
Sep 2023

SMTP - Operational

100% - uptime
Jul 2023 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2023
Aug 2023
Sep 2023

Webmail - Operational

100% - uptime
Jul 2023 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2023
Aug 2023
Sep 2023

Support - Degraded performance

100% - uptime
Jul 2023 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2023
Aug 2023
Sep 2023

Notice history

Sep 2023

Lucy server
  • Resolved
    Resolved

    Lucy is back online. A full account of events should follow a bit later.

  • Update
    Update

    There's an admin dream team working on this box right now with us.

  • Update
    Update

    We are continuing to work on a fix for this incident.

  • Update
    Update

    With activity split between getting the production server to boot it's OS, and deploying a replacement server, suddenly JetBackup doesn't work on a new server. So we opened a ticket with JetBackup. Whatever supernatural entity is responsible for this incredible sequence of events, even you have to sleep eventually.

  • Update
    Update

    We're still working on the OS. During this process, we've been preparing a replacement server to restore backups to, as we'd like that process to be moving along just in case we need it. If it's time wasted, and we sure hope it is, it was a worthy cause.

  • Update
    Update

    By all accounts we should either be online or right on the edge of being online. But the OS is still giving us hell. You can vent in tickets if you want, and we get it, but you won't speed this up. If we had a magic button we'd have pressed it already.

  • Update
    Update

    Hardware replaced, now it's just a matter of getting the OS to deal with the hardware changes and boot properly.

  • Update
    Update

    Still working on this. Note there is a reason it says 'Operational' above and it's too long to explain here, but no service on the lucy.mxrouting.net server is online until the text of these updates says otherwise.

  • Update
    Update

    RAID controller replacement didn't help. Was it replaced wrong? Is it the riser card? Stay tuned to find out.

  • Update
    Update

    We are continuing to work with remote hands on the raid controller replacement.

  • Update
    Update

    Frustration is understood. We're waiting to hear back from remote hands at the datacenter. Here are some facts:

    1. No inbound email will be missed, it will flow in after the server is back online.
    2. Many reasonable redundancies were in place here. It is not common for raid controllers to suddenly fail.
    3. This server is actively backed up, no matter what else happens.

    We expect to pop in the new controller, import the raid, and work fine under the expectation that this hardware configuration is perfect. This being a statistical anomaly alone, we don't expect a repeat event.

  • Update
    Update

    The server is not detecting the RAID controller. We're going to swap out the controller with the spare.

  • Identified
    Identified

    The Lucy server is having trouble booting, so that's interesting. We're working on it.

  • Investigating
    Investigating

    We are currently investigating this incident.

Possible routing issues
  • Resolved
    Resolved

    This incident has been resolved.

  • Investigating
    Investigating

    Routing issue in Ashburn may be causing some issues. Understanding scope and reviewing mitigation options.

  • Resolved
    Resolved

    There was an issue with an upstream network provider. This incident has been resolved.

  • Investigating
    Investigating

    Some users may be seeing routing issues to some of our servers. While we have not been able to replicate it ourselves, we are seeing reports of outages which suggests a potential upstream issue that may be more significant to a wider portion of the internet than just us.

aus.mxroute.com
  • Resolved
    Resolved

    This incident has been resolved.

  • Investigating
    Investigating

    We are currently investigating this incident.

aus.mxroute.com server
  • Resolved
    Resolved

    This incident has been resolved.

  • Identified
    Identified

    There is currently a network issue at the datacenter, working to resolve.

  • Investigating
    Investigating

    We are currently investigating this incident.

Longhorn Server
  • Resolved
    Resolved

    This incident has been resolved.

  • Update
    Update

    There appears to be no available on-site staff to resolve the current issue with the longhorn.mxrouting.net server. The server may be offline for some time until we are able to get someone on-site to resolve the issue. Migration of this server to a new facility, to prevent this issue, will begin a few hours after that resolution.

  • Identified
    Identified

    We are continuing to work on a fix for this incident.

  • Investigating
    Investigating

    We are currently investigating this incident.

Aug 2023

aus.mxroute.com server
  • Resolved
    Resolved

    Hardware migration complete. Crossbox will be reinstalled soon, but everything else is working. Remember that Crossbox is only one webmail option, others are at https://aus.mxroute.com:2096

  • Update
    Update

    Hardware migration will most likely take place within the next 4 hours. Users may see a few rejected connections during the transition. This should clear up on it's own in a very short amount of time.

  • Update
    Update

    We'll be pulling the trigger on hardware migration for the Aus server tonight. No downtime, unlikely customers will notice anything other than a brief hiccup. The hiccup in question is expected to be less troubling than your cell phone switching towers with a rough handshake.

  • Update
    Update

    While the aus server has been online since our last update, we're not confident in it's ability to stay online. We are migrating the system to new hardware right now. Steps have been taken to ensure data integrity in the event of a hardware failure, should it occur prior to completion of our migration.

  • Update
    Update

    The Aus server is online. This shouldn't be celebrated yet. Here's to copying data as fast as physically possible before a degraded RAID1 array which failed to boot multiple times says it's had enough.

  • Update
    Update

    The RAID controller is still reporting a fault, but we have more eyes on the issue on site than we've had in the previous hours.

  • Update
    Update

    We are continuing to work on a fix for this incident.

  • Update
    Update

    We are continuing to work on a fix for this incident.

  • Update
    Update

    We are continuing to work on a fix for this incident.

  • Update
    Update

    We are continuing to work on a fix for this incident. The RAID controller has failed, and we're waiting for on-site staff to further investigate possible solutions with available hardware.

  • Update
    Update

    We are continuing to work on a fix for this incident.

  • Identified
    Identified

    There appears to be a hardware issue with the aus.mxroute.com server. We are working with on-site staff to determine the scope and best path to resolution.

  • Investigating
    Investigating

    We are currently investigating this incident.

Path.net just shit the bed
  • Resolved
    Resolved

    This incident has been resolved.

  • Identified
    Identified

    We are continuing to work on a fix for this incident. Some updates here: https://status.path.net/incidents/50t7b6ts412f

  • Investigating
    Investigating

    We are currently investigating this incident. It looks like path.net just took down a good portion of the internet.

Acadia Server
  • Resolved
    Resolved

    This incident has been resolved.

  • Identified
    Identified

    Intermittent packet loss over some routes may indicate unmitigated DDOS attack, continuing to investigate.

  • Investigating
    Investigating

    We are currently investigating this incident.

Longhorn Server
  • Resolved
    Resolved

    This incident has been resolved.

  • Investigating
    Investigating

    We are currently investigating this incident. No email will be lost due to their event.

Jul 2023

Spam filters on Arrow server
  • Resolved
    Resolved

    This incident has been resolved.

  • Identified
    Identified

    Spam filters on the arrow.mxrouting.net server are not performing as desired after upgrading to the new system. More spam is coming in than is desirable. We are working on this.

Arrow Spam Filtering
  • Completed
    July 17, 2023 at 7:05 PM
    Completed
    July 17, 2023 at 7:05 PM

    Spam filtering on the Arrow server should be better and improved. An email will be sent to users on the Arrow server explaining further.

  • In progress
    July 17, 2023 at 6:48 PM
    In progress
    July 17, 2023 at 6:48 PM

    Spam filtering might be a bit "wonky" for a while as we perfect these improvements, but the worst result should be receiving an email that you prefer not to accept. Please do not open support tickets about spam received that shouldn't have been until this maintenance is closed.

  • Planned
    July 17, 2023 at 6:24 PM
    Planned
    July 17, 2023 at 6:24 PM

    We are expecting a temporary increase in inbound spam for users on the arrow.mxrouting.net server while we work on improvements to it's spam filtering.

Witcher Server
  • Resolved
    Resolved

    This incident has been resolved.

  • Investigating
    Investigating

    We are currently investigating an issue with the witcher.mxrouting.net server.

Jul 2023 to Sep 2023

Next