Chain Halt @ Block 1,589,496
Incident Report for Helium
Resolved
The chain halt is resolved. Enough of the consensus members have agreed on blocks and elected several times. We will continue to monitor.
Posted Oct 28, 2022 - 22:31 UTC
Update
The chain has progressed a bit further but we're not out of the woods. We've yet to have a new Consensus Group election so rewards continue to be delayed.

Proof-of-Coverage is degraded at this time, and pending transactions may take some time to clear.
Posted Oct 28, 2022 - 20:15 UTC
Update
The core team has a candidate Validator 1.15.8 release and will be available shortly. In the mean time, the current consensus group is having difficulty agreeing on a block and we are stuck again on 1,589,531.

A reminder to the rest of community - pending transactions will not clear until we get a new block. Please refrain from submitting duplicate transactions as they will all clear when the chain moves once again.
Posted Oct 28, 2022 - 18:06 UTC
Update
The Consensus Group has agreed on a block and we're moving onto block 1,589,497. We're not in the clear yet, as we need to see through the next consensus group election.

In the mean time, a Validator release 1.15.7 is available and in testing. This release includes a blessed snapshot and improvements to prevent Hotspot ddos. Run at your own risk as it's not been tagged GA yet.

Final reminder: Validator Operators, please close port 8080. We will let you know when to reopen it.
Posted Oct 28, 2022 - 17:04 UTC
Update
The core team is actively monitoring the Consensus Group to see if they can agree on a new block. At this time @Validator if you have a validator in Consensus, please join us in validator-ops and remember to close your port 8080.

The team is also preparing a new Validator release with the blessed snapshot and Hotspot ddos prevention.
Posted Oct 28, 2022 - 15:59 UTC
Investigating
The core developers and validator operators have noticed a chain halt at block 1,589,496. We are investigating the root cause.

At this time, all transactions on chain will be unconfirmed until Validators begin producing blocks again. Funds are safe but activity like Proof of Coverage, Hotspot onboarding and movement, Token Transfers, etc. are unavailable. Data Transfer is not affected.

At this time, all transactions on chain will be unconfirmed until Validators begin producing blocks again. Funds are safe but activity like Proof of Coverage, Hotspot onboarding and movement, Token Transfers, etc. are unavailable. Data Transfer is not affected.
Posted Oct 28, 2022 - 14:59 UTC
This incident affected: User Apps (Helium Wallet App, Blockchain, API) and Consensus Groups, Proof-of-Coverage, Helium Packet Router, Onboarding Service, Explorer, Helium App.