Emergency Reboot Cloud Nodes

Minor Resolved Started Aug 28, 2018 10:47 UTC · resolved Sep 6, 2018 15:19 UTC (9 days)

Resolved · Sep 6, 2018 15:19 UTC
Latest Spectre, Meltdown & ForeShadow Speculative Bugs are patched across 8Dweb cloud systems. The ForeShadow patches have increased CPU load a bit, but all nodes/hosts are running well within parameters.
Monitoring · Sep 1, 2018 12:57 UTC
All IAD3 NCP-ng Nodes Patched & Online
Monitoring · Aug 31, 2018 17:03 UTC
Most Nodes patched and online.
Identified · Aug 31, 2018 16:18 UTC
IAD3 Cloud System is about to be affected by an emergency reboot of all nodes. This **will** affect all hosts/nodes and **all** VM's and cloud servers running from IAD3 data center
This is considered emergency maintenance and as such you will need to follow this thread for updates.
Monitoring · Aug 28, 2018 14:18 UTC
This issue will remain open in Monitoring Mode for 24 hours.
Monitoring · Aug 28, 2018 12:34 UTC
8Dweb has identified a potential issue with a single Centos/KVM node in IAD3. This node has been removed from our primary cloud system for further evaluation. This node is dedicated to a single customer site at this time, so most sites/sentryservers will remain unaffected
Monitoring · Aug 28, 2018 10:58 UTC
As 8Dweb continues to deal with the Intel Speculative flaws, we will be **extremely** pro-active in patching both Guest VM Operating Systems as well as Hyper-visor (Host) Operating Systems in our public facing cloud infrastructure.
We will attempt to keep reboots outside normal business hours, however, if our team deems a vulnerability to be critical, we will notify of Emergency updates and restarts on this site.
We consider the patches to our cloud infrastructure applied this morning of just such a critical nature
Identified · Aug 28, 2018 10:47 UTC
There will be an emergency reboot of certain cloud nodes in IAD3 and IAD4 this morning to mitigate certain Vendor flaws.