Blockchain halted - Resolved
Incident Report for Helium
This incident has been resolved.
Posted Nov 17, 2021 - 20:38 UTC
We are continuing to monitor for any further issues.
Posted Nov 17, 2021 - 19:36 UTC
The core developers are tagging the 2021.11.16.3 miner firmware release as an emergency release. We will be issuing the GA imminently and have also informed major manufacturers to update their fleets.

We hope that in the coming hours and days, things will start to look a little more normal around here. In the meantime, however, expect lots of fluctuations in the PoC rate and rewards, including incredibly lucky rewards when there is limited PoC activity within the epoch.
Posted Nov 17, 2021 - 08:13 UTC
Thanks to the efforts of the Validator community, the Consensus Group has started producing blocks again. We will still need to continue to work on ETL/API nodes and Hotspots before PoC activity and all transactions are resumed but the network has made significant progress.
Posted Nov 17, 2021 - 02:35 UTC
Validator release v1.5.5 is available. Validator operators should immediately update to this latest release to remove the bottleneck and continue normal block production.
Posted Nov 17, 2021 - 00:22 UTC
The core developers are preparing Validator v1.5.x. Once released Validator operators in the Consensus Group should update to this new release without delay. This release includes a number of performance improvements and is designed to allow Validators to consume and manage the large block causing the bottleneck and resume normal block production.
Posted Nov 16, 2021 - 21:33 UTC
The core developers are finalizing performance update fixes and preparing a number of releases for validators, miners, and an updated snapshot for Hotspots. These performance updates will continue to benefit the chain after normal operations return. For example, during tests a Validator went from using 600% CPU utilization to 6%.
Posted Nov 16, 2021 - 18:50 UTC
The core developers continue to work on additional performance improvements for an upcoming release that will require Validators in the Consensus Group to take on. Once the release is ready any operator with a Validator in the Consensus Group please update without delay.
Posted Nov 16, 2021 - 15:51 UTC
After letting the Consensus Group try to figure out how to produce the block for a few hours, some of the core developers have started their day looking at additional performance adjustments that may help the group come together and restart the chain. Anyone with a Validator in the Consensus Group, please prepare for another tagged release. The entire network appreciates your diligence.
Posted Nov 16, 2021 - 14:42 UTC
The core developers have released Validator v1.5.0 and are now waiting on Validators in the Consensus Group to upgrade, come online, and start producing blocks. This process usually takes a little time as the Consensus Group starts to connect to each other and maintain connections for passing signatures. This may take a few hours before block production resumes.

In the meantime, some of the core team will be going offline while others stay online to monitor block production. We have tagged an updated version of `blockchain-node`, `blockchain-etl`, and `router`. Please upgrade to these latest versions as soon as possible as once block production resumes, they will be able to start syncing again.

Again, Hotspots will need to wait on a chain-blessed snapshot before we can issue a new firmware release. It'll likely be about 5 hours after block production resumes. Until then, PoC activity and Data Transfer will remain affected.

Thank you again for your patience.
Posted Nov 16, 2021 - 09:26 UTC
A fix has been implemented and we are monitoring the results.
Posted Nov 16, 2021 - 08:13 UTC
A Validator update (v1.5.0) has been released and anyone with Validators in the Consensus Group should update ASAP. Once this release has been picked up by a sufficient number of Consensus Group members and block production returns to normal, we’ll communicate with the rest of the Validator community to update to this release as well. Hotspots may continue to be offline after block production resumes and we will need to cut an emergency release for Hotspot firmware after a snapshot has been agreed upon by the Consensus Group.
Posted Nov 16, 2021 - 07:33 UTC
The team is continuing to finalize the update to return the blockchain to normal operations. Stay tuned.
Posted Nov 16, 2021 - 05:42 UTC
Great progress by the team. A number of performance fixes have been tested and are preparing to be pushed.

The team will deploy an update that includes these fixes to move past the megablock and allow validators to resume normal block production.
Posted Nov 16, 2021 - 04:14 UTC
The team is continuing to make progress across a number of areas including addressing the megablock that is causing the bottleneck, fixing the boot looping caused by the megablock, and pushing an update to increase the speed to close state channels.

At this time blockchain activity remains halted.
Posted Nov 16, 2021 - 02:55 UTC
The router fix has been pushed to the various Helium Console/Router instances. This fix will prevent the reoccurrence of this issue.

The team is still testing various fixes, however, the blockchain status has not yet changed and blockchain activity is still halted.
Posted Nov 16, 2021 - 01:47 UTC
A very large block has been made with additional blocks slowly getting formed. The team is testing a few patches and to prevent this from reoccurring, the update to other router instances is ready for deployment this week. A validator release is also being prepared. Unfortunately, all blockchain activity (e.g., proof of coverage, payments, hotspot assertions, transfers, etc.) is still effectively halted until blocks are back to normal production. Stay tuned.
Posted Nov 16, 2021 - 00:38 UTC
The blockchain has halted due to a bottleneck related to large number of transactions. This outage impacts PoCs, challenges, etc. The team has identified the issue and is working on a fix.
Posted Nov 15, 2021 - 22:53 UTC
This incident affected: Consensus Groups and Proof-of-Coverage.