Validator release v1.5.5 is available. Validator operators should immediately update to this latest release to remove the bottleneck and continue normal block production.
Posted Nov 17, 2021 - 00:22 UTC
Update
The core developers are preparing Validator v1.5.x. Once released Validator operators in the Consensus Group should update to this new release without delay. This release includes a number of performance improvements and is designed to allow Validators to consume and manage the large block causing the bottleneck and resume normal block production.
Posted Nov 16, 2021 - 21:33 UTC
Update
The core developers are finalizing performance update fixes and preparing a number of releases for validators, miners, and an updated snapshot for Hotspots. These performance updates will continue to benefit the chain after normal operations return. For example, during tests, a Validator went from using 600% CPU utilization to 6%.
Posted Nov 16, 2021 - 18:50 UTC
Update
The core developers continue to work on additional performance improvements for an upcoming release that will require Validators in the Consensus Group to take on. Once the release is ready any operator with a Validator in the Consensus Group please update without delay.
Posted Nov 16, 2021 - 15:51 UTC
Update
After letting the Consensus Group try to figure out how to produce the block for a few hours, the core developers have started their day looking at additional performance adjustments that may help the Consensus Group come together and restart the chain. Anyone with a validator in the Consensus Group, please prepare for another tagged release. The entire network appreciates your diligence.
Posted Nov 16, 2021 - 14:50 UTC
Update
The core developers have released Validator v1.5.0 and are now waiting on Validators in the Consensus Group to upgrade, come online, and start producing blocks. This process usually takes a little time as the Consensus Group starts to connect to each other and maintain connections for passing signatures. This may take a few hours before block production resumes.
In the meantime, some of the core team will be going offline while others stay online to monitor block production. We have tagged an updated version of `blockchain-node`, `blockchain-etl`, and `router`. Please upgrade to these latest versions as soon as possible as once block production resumes, they will be able to start syncing again.
Again, Hotspots will need to wait on a chain-blessed snapshot before we can issue a new firmware release. It'll likely be about 5 hours after block production resumes. Until then, PoC activity and Data Transfer will remain affected.
Thank you again for your patience.
Posted Nov 16, 2021 - 09:25 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Nov 16, 2021 - 08:13 UTC
Update
A Validator update (v1.5.0) has been released and anyone with Validators in the Consensus Group should update ASAP. Once this release has been picked up by a sufficient number of Consensus Group members and block production returns to normal, we’ll communicate with the rest of the Validator community to update to this release as well. Hotspots may continue to be offline after block production resumes and we will need to cut an emergency release for Hotspot firmware after a snapshot has been agreed upon by the Consensus Group.
Posted Nov 16, 2021 - 07:33 UTC
Update
The team is continuing to finalize the update to return the blockchain to normal operations. Stay tuned.
Posted Nov 16, 2021 - 05:42 UTC
Update
Great progress by the team. A number of performance fixes have been tested and are preparing to be pushed.
The team will deploy an update that includes these fixes to move past the megablock and allow validators to resume normal block production.
Posted Nov 16, 2021 - 04:14 UTC
Update
The team is continuing to make progress across a number of areas including addressing the megablock that is causing the bottleneck, fixing the boot looping caused by the megablock, and pushing an update to increase the speed to close state channels.
At this time blockchain activity remains halted.
Posted Nov 16, 2021 - 02:55 UTC
Update
The router fix has been pushed to the various Helium Console/Router instances. This fix will prevent the reoccurrence of this issue.
The team is still testing various fixes, however, the blockchain status has not yet changed and blockchain activity is still halted.
Posted Nov 16, 2021 - 01:46 UTC
Update
A very large block has been made with additional blocks slowly getting formed. The team is testing a few patches and to prevent this from reoccurring, the update to other router instances is ready for deployment this week. A validator release is also being prepared. Unfortunately, all blockchain activity (e.g., proof of coverage, payments, hotspot assertions, transfers, etc.) is still effectively halted until blocks are back to normal production. Stay tuned.
Posted Nov 16, 2021 - 00:38 UTC
Update
The blockchain has halted due to a bottleneck related to large number of transactions. This outage impacts PoCs, challenges, etc. The team has identified the issue and is working on a fix.
Posted Nov 15, 2021 - 22:49 UTC
Identified
The additional loading of a recently released chainvar caused a router instance to propagate many stale state channels which resulted in a large number of duplicated transactions.
This large number of duplicated transactions has caused a bottleneck resulting in longer than normal block times.
An out of band fix was applied to the router to prevent a reoccurrence and this patch will be released to other router instances and available to open source routers this week.
The team is working on ways to remove the bottleneck.
Posted Nov 15, 2021 - 22:05 UTC
Investigating
The blockchain is currently experiencing long block times. We have identified the root cause and the team is currently working on a fix to resolve and prevent future occurrences.