Console is back up and data packet transfer has returned to normal operations. Thanks for your patience and apologies for any inconvenience caused.
Posted Nov 17, 2021 - 18:17 UTC
Update
Validator release v1.5.5 is available. Validator operators should immediately update to this latest release to remove the bottleneck and continue normal block production.
Posted Nov 17, 2021 - 00:23 UTC
Update
The core developers are preparing Validator v1.5.x. Once released Validator operators in the Consensus Group should update to this new release without delay. This release includes a number of performance improvements and is designed to allow Validators to consume and manage the large block causing the bottleneck and resume normal block production.
Posted Nov 16, 2021 - 21:34 UTC
Update
The core developers are finalizing performance update fixes and preparing a number of releases for validators, miners, and an updated snapshot for Hotspots. These performance updates will continue to benefit the chain after normal operations return. For example, during tests a Validator went from using 600% CPU utilization to 6%.
Posted Nov 16, 2021 - 18:51 UTC
Update
The core developers continue to work on additional performance improvements for an upcoming release that will require Validators in the Consensus Group to take on. Once the release is ready any operator with a Validator in the Consensus Group please update without delay.
Posted Nov 16, 2021 - 15:52 UTC
Update
After letting the Consensus Group try to figure out how to produce the block for a few hours, the core developers have started their day looking at additional performance adjustments that may help the Consensus Group come together and restart the chain. Anyone with a validator in the Consensus Group, please prepare for another tagged release. The entire network appreciates your diligence.
Posted Nov 16, 2021 - 14:51 UTC
Update
The core developers have released Validator v1.5.0 and are now waiting on Validators in the Consensus Group to upgrade, come online, and start producing blocks. This process usually takes a little time as the Consensus Group starts to connect to each other and maintain connections for passing signatures. This may take a few hours before block production resumes.
In the meantime, some of the core team will be going offline while others stay online to monitor block production. We have tagged an updated version of `blockchain-node`, `blockchain-etl`, and `router`. Please upgrade to these latest versions as soon as possible as once block production resumes, they will be able to start syncing again.
Again, Hotspots will need to wait on a chain-blessed snapshot before we can issue a new firmware release. It'll likely be about 5 hours after block production resumes. Until then, PoC activity and Data Transfer will remain affected.
Thank you again for your patience.
Posted Nov 16, 2021 - 09:26 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Nov 16, 2021 - 08:14 UTC
Update
A Validator update (v1.5.0) has been released and anyone with Validators in the Consensus Group should update ASAP. Once this release has been picked up by a sufficient number of Consensus Group members and block production returns to normal, we’ll communicate with the rest of the Validator community to update to this release as well. Hotspots may continue to be offline after block production resumes and we will need to cut an emergency release for Hotspot firmware after a snapshot has been agreed upon by the Consensus Group.
Posted Nov 16, 2021 - 07:33 UTC
Update
The team is continuing to finalize the update to return the blockchain to normal operations. Stay tuned.