MXroute - Notice history

Support experiencing degraded performance

Billing Panel - Operational

100% - uptime
Jul 2024 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024

DirectAdmin Panel - Operational

100% - uptime
Jul 2024 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024

IMAP - Operational

100% - uptime
Jul 2024 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024

SMTP - Operational

100% - uptime
Jul 2024 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024

Webmail - Operational

100% - uptime
Jul 2024 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024

Support - Degraded performance

100% - uptime
Jul 2024 · 100.0%Aug · 100.0%Sep · 100.0%

Previous page

Next page

Jul 2024
Aug 2024
Sep 2024

Notice history

Sep 2024

DirectAdmin Temporarily Disabled on Safari server
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    We were seeing webmail outages up to 5 minutes at a time on the Safari server yesterday, increasing in frequency throughout the day. The decision was made to move the Apache includes to the SSD array currently housing MySQL, to speed up config reloads. For us that means moving the DA user data. This is proving to take longer than predicted, there's a strange blocker to determining progress other than "done" or "not done."

    Giving an ETA for DirectAdmin to be back online would be irresponsible. Best course of action is see this through. No email access or delivery is harmed by this. This is only administration control panel access.

  • Identified
    UTC
    Identified

    We are performing emergency maintenance that does not impact any email service on the safari.mxrouting.net server. Part of this process involves temporarily disabling DirectAdmin. We will keep you posted.

safari.mxrouting.net hardware upgrade
  • Completed
    September 13, 2024 at 3:05 AMUTC
    Completed
    September 13, 2024 at 3:05 AMUTC
    Maintenance has completed successfully.
  • Update
    September 13, 2024 at 2:17 AMUTC
    Update
    September 13, 2024 at 2:17 AMUTC

    Server has been back up for 20 minutes but now we're moving MySQL. This means your email clients are working fine, webmail clients like webmail.mxroute.com are working fine, but Crossbox and Roundcube on the Safari server itself will be back up once we finish moving MySQL data to the new SSD array.

  • Update
    September 13, 2024 at 1:16 AMUTC
    Update
    September 13, 2024 at 1:16 AMUTC

    Services taken down for power down.

  • Update
    September 13, 2024 at 1:06 AMUTC
    Update
    September 13, 2024 at 1:06 AMUTC

    Shutting down MySQL

  • In progress
    September 13, 2024 at 12:00 AMUTC
    In progress
    September 13, 2024 at 12:00 AMUTC
    Maintenance is now in progress
  • Planned
    September 13, 2024 at 12:00 AMUTC
    Planned
    September 13, 2024 at 12:00 AMUTC

    The safari.mxrouting.net server will be going down sometime between 7PM and 11PM US/Central time today for the addition of two SSD drives that will be used to migrate the MySQL installation off of the same array housing the rest of customer data. This is intended to address the root cause of a performance issue that had been reported a while back. While we've been fine on performance for some time, that event did teach us what was necessary to prevent it from occurring again in the future, and this is it. We do not expect the server to be down during the 4 hour window that we've specified. We expect it to occur sometime during that window. Unfortunately we are not able to schedule this to a more specific time or to a further date in the future, today is when the stars aligned.

Packet loss over a few routes to Ashburn
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Monitoring
    UTC
    Monitoring

    From Jeff. The man, myth, legend:

    "Arelion/Telia is having an outage caused by an equipment failure in Atlanta that is impacting East/West traffic in the US. Arelion plans to replace the equipment shortly (they said at about midnight Eastern time). We've turned off Arelion in both Ashburn and LA temporarily until the issue is resolved."

  • Investigating
    UTC
    Investigating

    Some users are reporting high packet loss when connecting to servers in our Ashburn location. A check from ping.pe shows only a few routes impacted, most can reach it just fine. Looks to have a correlation with GTT, unclear. Investigating.

Crossbox down for users on wednesday.mxrouting.net server
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Identified
    UTC
    Identified

    The application reports that the license is expired for this server. It isn't. We've done everything we can, it's in the hands of their developers as of right now. You can use any other email client or webmail client in the meantime, webmail.mxroute.com is a popular choice.

Aug 2024

Restarting Crossbox
  • Completed
    August 31, 2024 at 8:21 PMUTC
    Completed
    August 31, 2024 at 8:21 PMUTC

    Done.

  • In progress
    August 31, 2024 at 8:08 PMUTC
    In progress
    August 31, 2024 at 8:08 PMUTC

    We are restarting instances of the Crossbox webmail application across the fleet.

Echo server reboot
  • Completed
    August 27, 2024 at 8:46 PMUTC
    Completed
    August 27, 2024 at 8:46 PMUTC

    Been back up for a while, forgot to close status.

  • In progress
    August 27, 2024 at 8:30 PMUTC
    In progress
    August 27, 2024 at 8:30 PMUTC
    Maintenance is now in progress
  • Planned
    August 27, 2024 at 8:30 PMUTC
    Planned
    August 27, 2024 at 8:30 PMUTC

    The echo.mxrouting.net server will be rebooted at 8:30 PM GMT on August 27th, 2024. The expectation is that this takes about 5-10 minutes and then everything is back up and running. Secondary plans will be prepared by that time in the case of failure to come back online.

Outbound email delivery delays
  • Resolved
    UTC
    Resolved

    This incident has been resolved. Postmortem:

    In certain events this config block caused exim to give an empty HELO to postfix:

    helo_data = ${if eq{$sender_address_domain}{}\

    {${lookup{${lc:${domain:$h_from:}}}lsearch,ret=key{/etc/virtual/domainowners}{$value}}}\

    {${if exists{/etc/virtual/helo_data}{${lookup{$sending_ip_address}iplsearch{/etc/virtual/helo_data}{$value}{$primary_hostname}}}{$primary_hostname}}}\

    }

    That config block has been updated:

    helo_data = ${if exists{/etc/virtual/helo_data}{${lookup{$sending_ip_address}iplsearch{/etc/virtual/helo_data}{$value}{$primary_hostname}}}{$primary_hostname}}

  • Identified
    UTC
    Identified

    We're seeing some intermittent outbound email delivery delays due to a strange issue with postfix. Currently investigating the problem a bit deeper with an increased log level to see if we can shake loose some more details.

Jul 2024

friday.mxlogin.com
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Investigating
    UTC
    Investigating

    The Friday server experienced a load exceeding 500. The system was rebooted to try to bring down the load to troubleshoot the cause. As of this posting, the server has not come back online.

banshee.mxlogin.com
Ocean cPanel Server Unresponsive
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.
Crossbox on our cPanel servers
  • Resolved
    UTC
    Resolved
    This incident has been resolved.
  • Update
    UTC
    Update

    MySQL is working fine now on Ocean, Aus, Acadia, and Friday cPanel servers. The Banshee server is the only one remaining to be repaired. This one may take more time.

  • Update
    UTC
    Update

    The problem appears to be spreading in our cPanel servers. This is not a Crossbox problem. This is a cPanel problem which impacts Crossbox. It seems that cPanel pushed a bad MySQL update which broke a wealth of servers, not just ours. We're evaluating and testing options, we don't want to risk causing larger issues as this impact is not what we consider to be service breaking.

  • Identified
    UTC
    Identified

    Currently users are unable to access Crossbox (or anything else which uses MySQL) on these cPanel servers: Banshee, Aus, Friday, Ocean. This is specific to our cPanel servers, we are not seeing any similar behavior on our DirectAdmin servers. It may take some time to get a handle on this, it's quite odd. As it is impacting a small number of customers this status page will continue to say that everything is operational, as we do not want to give the rest of our customers a reason to not open a support ticket if they are experiencing an issue with any other webmail.

Jul 2024 to Sep 2024

Next