The chain has resumed normal operations. The core dev team is working on creating a blessed snapshot for a small number of Validators that continue to be stuck at block height 1,587,339 or prior to 1,587,313.
If you're stuck on 1,587,313: - Nodes (ETL, routers, etc) - simply need to restart - Validators run the following command: miner eval 'blockchain_worker:sync().'
Posted Oct 27, 2022 - 05:26 UTC
Update
The core team is now trying to get the rest of the Validator fleet caught up to the new block height so it can gossip block information around and be eligible for a new consensus election.
Posted Oct 27, 2022 - 04:08 UTC
Update
A new consensus group has been assembled and there are now new blocks. We're now waiting to see if the other services including the ETL that serves data to the API (and explorer/apps) can catch up.
At this time, please refrain from submitting new transactions until we can give the go-ahead, they may take a while to clear still.
Posted Oct 27, 2022 - 03:16 UTC
Update
The core team has assembled a new candidate Consensus Group. After the new group is elected, the core team will issue a rescue block to get us past the halt.
For now, the chain is still halted. Funds are safe, data transfer is not affected.
Posted Oct 27, 2022 - 02:09 UTC
Identified
We believe we have the root cause and will likely need to solve it with a rescue block.
All Validator Operators should join us on Discord for further coordination. For the rest of the community, we remain halted at this time. Funds remain safe but transactions are still delayed until the halt is resolved.
Posted Oct 27, 2022 - 01:52 UTC
Investigating
The core developers and validators operators have noticed a chain halt where validators aren't producing blocks after 1,587,313. We are investigating the root cause.
At this time, all transactions on chain will be unconfirmed until Validators begin producing blocks again. Funds are safe but activity like Proof of Coverage, Hotspot onboarding and movement, Token Transfers, etc. are unavailable. Data Transfer is not affected.
Posted Oct 27, 2022 - 00:53 UTC
This incident affected: Consensus Groups, Proof-of-Coverage, Onboarding Service, Explorer, and Helium App.