The chain has been consistently creating blocks for the last 3 hours. The team will continue to closely monitor but believe the issue has been resolved.
Posted Feb 07, 2022 - 17:28 UTC
Monitoring
Validators have resumed block production and the chain is back to normal operations. Pending transactions have been re-enabled and already cleared.
A large block was created and the skip of the halting block included wiping the transaction buffer and replacing with a chain var to temporarily reduce the state channel grace period to mitigate the issue. However, state channel closes will be negatively impacted including rewards for packet transfer and high overages.
Another chain var will be issued to return the state channel back to normal operations.
The team is performing further investigation to determine root cause.
Posted Feb 07, 2022 - 15:05 UTC
Identified
Rather than issuing a Rescue Block, the team would like all Validators currently in the Consensus Group to come online and issue a command on their Validator. Please join us in Discord (https://discord.gg/helium) on the #validator channel for more instructions.
Posted Feb 07, 2022 - 10:55 UTC
Update
The Validator community continues to monitor the Consensus Group as it tries to agree upon a block. Once this is completed, the core developers will prepare a chain variable to address the State Channel issue that may be affecting the blockchain. Although we are prepared to issue a Rescue block, we are hoping to avoid doing this as Data Transfer continues to be unaffected.
Other transactions (including Hotspot onboarding, Payments, and Proof of Coverage) remain degraded at this time.
Posted Feb 07, 2022 - 09:08 UTC
Update
We are continuing to monitor the Consensus Group. It is making slow progress but has not produced a block yet.
Posted Feb 07, 2022 - 07:48 UTC
Update
The Consensus Group is still struggling to produce the next block with transactions and, as a result, the chain remains halted.
Data Transfer continues to be unaffected but all other blockchain activity is delayed, including transaction processing.
Posted Feb 07, 2022 - 06:31 UTC
Investigating
Helium Validator operators and core developers are investigating a potential chain halt @ block 1215286.
The Consensus Group has not produced a block in about 35 minutes. More information to come as we uncover more details. Validator community, please come online if you have validators currently in the Consensus Group.
Proof of Coverage may be degraded, and other transactions may be queued until the Consensus Group resumes block production. Data Transfer remains unaffected.
Posted Feb 07, 2022 - 05:09 UTC
This incident affected: User Apps (Blockchain, API) and Consensus Groups, Proof-of-Coverage, Onboarding Service, Explorer, Helium App.