WEB05 Failure
- Resolved ·
All known sites are now restored.
- Monitoring ·
Nearly all websites have been restored. Some things such as Redis or Litespeed caching plugins may be missing, however, those are minor issues and can be rectified (if required) at a future time.
There are several sites (currently 6) that were affected by a further issue where DNS records were not contained in the Enhance backups. This matter is being investigated by the Enhance team, and those sites remain OFFLINE at this moment.
- Identified ·
Site restoration continues. In each case Enhance backups have come to the rescue, however, the full disaster recovery provided for downed servers in Enhance did not work as expected. Please do not open tickets for down sites at this juncture, we are aware of all sites that are currently still not working properly, and are methodically working through the process.
- Identified ·
Sites are being restored. At this juncture there is a manual step in the disaster recovery where our team needs to touch each web site. We have notified and been working directly with Enhance support about that particular issue. Sites will come online in a strictly alphabetical order.
- Identified ·
Restorations are under way, however, database restorations are not properly working. We are working with the Enhance team to pin this DR issue down.
- Identified ·
web05 has been rebuilt.
web06 has been rebuilt
Restoration of sites will begin via disaster recover process at approximately 9:00pm Eastern.This will not guarantee any specific site will be online shortly thereafter. The process is expected to take some time, so there will be no ETA until restoration is complete and our team begins going through sites to verify them online, or not.
- Investigating ·
There has been an unknown failure on our WEB05 server.
There is no ETA on resolution.