Chain Halt @ 1368539

Incident Report for Helium

Resolved

The blockchain has stabilized but consensus groups are still running longer than expected. The team will continue to monitor but we'll be clearing the chain halt at this time, since it continues to progress.
Posted May 26, 2022 - 04:02 UTC

Monitoring

Block production has resumed. The core developers and Validator operators will continue to monitor the chain. We plan on deferring the h3dex (HIP 54) and XOR filter fees (HIP 59) chain variables until tomorrow.
Posted May 25, 2022 - 22:18 UTC

Update

The core developers have asked all Validator operators with Validators in the current Consensus Group to come online to help with active management of the ongoing block production issue.

We appreciate the ongoing patience of Hotspot owners and other network participants
Posted May 25, 2022 - 22:01 UTC

Identified

Although the Consensus Group has produced block 1,368,539 we are still monitoring the chain as it processes this very large block (taking some validators 10+ minutes to validate) and moves forward to the next block. The core developers and validator operators continue to monitor the chain.
Posted May 25, 2022 - 21:09 UTC

Investigating

There is a big block that the core devs and validator operators are keeping an eye on. We did just issue a chain variable during the last epoch so it could be related to that but we will keep the community updated as we have more information

Nothing actionable at this time for Hotspot owners or other network participants. We will provide updates as they come.
Posted May 25, 2022 - 20:32 UTC
This incident affected: Consensus Groups, Proof-of-Coverage, and Onboarding Service.