The core developers have tagged a new 2021.11.22.0 Hotspot firmware release which includes a mandatory fix to unstick some Hotspot stuck syncing at 1,107,994 due to a nonce disagreement from loading a snapshot. You can read more about the release on the Engineering blog: https://engineering.helium.com/2021/11/22/blockchain-release-sync-emergency.html
Plan We have been beta testing 2021.11.22.0 since November 22, 2021 8:00 PM PT. Current ETA for GA is 9:15 PM PT, November 22, 2021.
For clarity - this new Hotspot release will unstick those Hotspots stuck on block 1,107,994. Makers have been notified of this new release and they will be pushing it out to their fleet when they are able to.
No action is required for Hotspot owners.
Posted Nov 23, 2021 - 14:55 UTC
Update
The team has finalized a fix and prepared a release for Hotspots 2021.11.22.0 and will be sending it out for a short beta testing cycle before deploying to Helium and RAK Hotspots, and tagging for Makers to update their fleet.
No action is needed for Hotspot owners at this time.
Posted Nov 23, 2021 - 04:10 UTC
Update
The core developers have not been able to find a small enough snapshot for the network to consume. Instead, the team will be issuing a new Hotspot firmware release and a Validator tag.
What happened: The affected Hotspots and Validators loaded a snapshot with a rescue block with a specific nonce. The rest of the blockchain that did not load this snapshot have a difference nonce. The disagreement in nonce is what is causing some Hotspots and Validators to stop syncing. This new release will realign nonce values across the network.
As a Hotspot owner, you’ll know you’re affected if your Hotspot is stuck at block 1,107,994. Best way to know this is if you pair with your Hotspot over bluetooth or have direct information through a Maker dashboard (and not through something like explorer that could be a bit delayed).
The team is currently testing the fix.
Your transactions are safe, HNT funds are secure, packets are still moving, PoC is unaffected for those synced to the chain, and Hotspots can still be added to the network.
Posted Nov 23, 2021 - 02:58 UTC
Identified
The core developers believe that a new snapshot can help Validators and Hotspots get past this sync issue. The team is actively testing snapshots to verify a suitable one.
Posted Nov 23, 2021 - 02:32 UTC
Update
The core developers have identified that the issue is related to a snapshot that included the rescue block and the recently deployed chain variable.
It appears this issue is only impacting miners that consumed this snapshot which is why the blockchain is continuing to process blocks. The team is continuing to verify this issue and start working on a fix.
Posted Nov 23, 2021 - 01:46 UTC
Update
Blocks are still being produced by the Consensus Group and Transactions and Proof of Coverage are still occurring on the network, albeit at a reduced rate. The core developers are trying to scope the impact of this issue.
Posted Nov 23, 2021 - 01:04 UTC
Investigating
The community has identified syncing issues @ block 1107995 potentially due to a chain variable that was issued a few moments ago. At this time, some Hotspots and Validators may be affected and the core developers are investigating.
More details to come.
Posted Nov 23, 2021 - 00:43 UTC
This incident affected: User Apps (Blockchain) and Proof-of-Coverage.