We have moved towards operational status on all apps that introspect the blockchain including the API, Explorer, DeWi Onboarding Service, and the Helium Hotspot App.
Posted Nov 17, 2021 - 08:15 UTC
Update
Validator release v1.5.5 is available. Validator operators should immediately update to this latest release to remove the bottleneck and continue normal block production.
Posted Nov 17, 2021 - 00:22 UTC
Update
The core developers are preparing Validator v1.5.x. Once released Validator operators in the Consensus Group should update to this new release without delay. This release includes a number of performance improvements and is designed to allow Validators to consume and manage the large block causing the bottleneck and resume normal block production.
Posted Nov 16, 2021 - 21:33 UTC
Update
The core developers are finalizing performance update fixes and preparing a number of releases for validators, miners, and an updated snapshot for Hotspots. These performance updates will continue to benefit the chain after normal operations return. For example, during tests a Validator went from using 600% CPU utilization to 6%.
Posted Nov 16, 2021 - 18:50 UTC
Update
The core developers continue to work on additional performance improvements for an upcoming release that will require Validators in the Consensus Group to take on. Once the release is ready any operator with a Validator in the Consensus Group please update without delay.
Posted Nov 16, 2021 - 15:51 UTC
Update
After letting the Consensus Group try to figure out how to produce the block for a few hours, the core developers have started their day looking at additional performance adjustments that may help the Consensus Group come together and restart the chain. Anyone with a validator in the Consensus Group, please prepare for another tagged release. The entire network appreciates your diligence.
Posted Nov 16, 2021 - 14:50 UTC
Update
The core developers have released Validator v1.5.0 and are now waiting on Validators in the Consensus Group to upgrade, come online, and start producing blocks. This process usually takes a little time as the Consensus Group starts to connect to each other and maintain connections for passing signatures. This may take a few hours before block production resumes.
In the meantime, some of the core team will be going offline while others stay online to monitor block production. We have tagged an updated version of `blockchain-node`, `blockchain-etl`, and `router`. Please upgrade to these latest versions as soon as possible as once block production resumes, they will be able to start syncing again.
Again, Hotspots will need to wait on a chain-blessed snapshot before we can issue a new firmware release. It'll likely be about 5 hours after block production resumes. Until then, PoC activity and Data Transfer will remain affected.
Thank you again for your patience.
Posted Nov 16, 2021 - 09:25 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Nov 16, 2021 - 08:13 UTC
Update
A Validator update (v1.5.0) has been released and anyone with Validators in the Consensus Group should update ASAP. Once this release has been picked up by a sufficient number of Consensus Group members and block production returns to normal, we’ll communicate with the rest of the Validator community to update to this release as well. Hotspots may continue to be offline after block production resumes and we will need to cut an emergency release for Hotspot firmware after a snapshot has been agreed upon by the Consensus Group.
Posted Nov 16, 2021 - 07:32 UTC
Update
The team is continuing to finalize the update to return the blockchain to normal operations. Stay tuned.
Posted Nov 16, 2021 - 05:41 UTC
Update
While the blockchain is halted the team is putting the API into maintenance mode. The API can still be used to check pending transactions, however, all new transactions will fail including those submitted from the wallet.
Posted Nov 16, 2021 - 03:30 UTC
Update
The team is continuing to make progress across a number of areas including addressing the megablock that is causing the bottleneck, fixing the boot looping caused by the megablock, and pushing an update to increase the speed to close state channels.
At this time blockchain activity remains halted.
Posted Nov 16, 2021 - 02:54 UTC
Update
The router fix has been pushed to the various Helium Console/Router instances. This fix will prevent the reoccurrence of this issue.
The team is still testing various fixes, however, the blockchain status has not yet changed and blockchain activity is still halted.
Posted Nov 16, 2021 - 01:46 UTC
Update
A very large block has been made with additional blocks slowly getting formed. The team is testing a few patches and to prevent this from reoccurring, the update to other router instances is ready for deployment this week. A validator release is also being prepared. Unfortunately, all blockchain activity (e.g., proof of coverage, payments, hotspot assertions, transfers, etc.) is still effectively halted until blocks are back to normal production. Stay tuned.
Posted Nov 16, 2021 - 00:39 UTC
Identified
The blockchain has halted due to a bottleneck related to large number of transactions. This outage impacts PoCs, challenges, etc. The team has identified the issue and is working on a fix.
Posted Nov 15, 2021 - 22:54 UTC
This incident affected: User Apps (API) and Onboarding Service, Explorer, Helium App.