Block production has resumed and we have issued a new Hotspot firmware release that should help Hotspots absorb the Rescue Block and continue syncing. We will continue to monitor the network over the next several hours and follow up with any updates necessary to help older Hotspots or other blockchain components.
Postmortem to follow this coming week.
Posted Nov 21, 2021 - 12:40 UTC
Monitoring
Block production has resumed. The core developers and validator operators are continuing to monitor the chain.
Posted Nov 21, 2021 - 08:51 UTC
Update
After several hours of monitoring the Consensus Group, the Decentralized Wireless Alliance (DeWi) in conjunction with several core developers will attempt to end the current chain halt with a Rescue Block, All funds remain safe through this transaction and once block production resumes, all participants will be able to inspect the chain for the status of in-flight transactions.
The Rescue Block changes the value of the `sc_grace_blocks` chain variable (to `1`) and elects a new Consensus Group using existing members of the current group and a few new connectable, upgraded (v1.5.12) Validators. The variable change should force the remaining open State Channels from the Discovery Mode Router closed. Block production should then resume since Discovery Mode has been taken offline until we can solve the underlying issues that was the root cause.
Posted Nov 21, 2021 - 06:52 UTC
Identified
The core developers continue to monitor the Consensus Group as it struggles to produce block 1105523. At this time, all transactions, Hotspot, and PoC activity continues to be halted until block production resumes.
If you are operating a Validator and haven't upgraded to v1.5.12, please do so, ASAP.
Posted Nov 21, 2021 - 05:04 UTC
Investigating
Core developers are back to monitoring the Consensus Group which is struggling to produce block 1105523. The last produced block was around 50 minutes ago at height 1105522.
Posted Nov 21, 2021 - 03:36 UTC
This incident affected: User Apps (Blockchain, API) and Consensus Groups, Proof-of-Coverage.