Proof-of-Coverage Rates and Stabilizing Rewards Update 9
At this point, the team is monitoring Proof-of-Coverage receipts and overall network health. There are no outstanding items for the team to address. Unless something goes wrong, there will be no more updates from me.
If you're interested in the metrics, visit https://dashboard.helium.com/. The green line represents Proof-of-Coverage receipts with witnesses (higher is good), the red line represents Proof-of-Coverage receipts with 0 witnesses (we want this as low as possible).
Thanks everyone for their continued patience 🎈
Posted May 14, 2022 - 00:28 UTC
Identified
Proof-of-Coverage Rates and Lower Rewards Update 8
The Hotspot release is now available. Makers will take this release and send the update to their fleet of Hotspots. It is up to each individual Hotspot manufacturer to send the update to your Hotspot.
This update allows Hotspots to disconnect from a poorly performing Validator and search for a new one if the connected validator is not keeping up with the chain for 15 blocks (approximately 15 minutes).
No action is required for Hotspot owners.
Posted May 13, 2022 - 22:17 UTC
Update
Proof-of-Coverage Rates and Lower Rewards Update 7
1) Validator release is out. We're seeing PoC receipts trending in the right direction (UpOnly) as Validator operators slowly take the update.
2) Hotspot release is now in testing.
No action is required for Hotspot owners.
Posted May 13, 2022 - 22:03 UTC
Update
Proof-of-Coverage Rates and Lower Rewards Update 6
1) A Validator release is now available and is mandatory. Validators are upgrading as we speak. It should fix the PoC receipt delivery that Hotspots have been tirelessly trying to send.
2) The Hotspot release is almost through reviews. Testing next. While this fixes connecting to a better Validator, the real fix lies in 1️⃣ with the Validator release.
The team will monitor 1️⃣ and proceed down the path of Light Hotspots and will not rollback at this time.
No action is required for Hotspot owners.
Posted May 13, 2022 - 20:53 UTC
Update
Proof-of-Coverage Rates and Lower Rewards Update 5
1) In addition to the Hotspot candidate release, we are looking at a potential fix on a Validator side that requires a @Validator Update. When we have a Validator candidate release, we will let Validator Operators know.
2) Other action we're considering taking: Increase the challenges per block some more
No action is required for Hotspot owners.
Posted May 13, 2022 - 19:38 UTC
Update
Proof-of-Coverage Rates and Lower Rewards Update 4
1) We have a candidate release that tells a Hotspot to connect to a different Validator. Just needs code review and approvals before testing.
2) The team will continue down the path of Light Hotspots and will keep working. We will not rollback at Friday, May 13, 2022 12:00 PM and instead see how the Hotspot software update performs.
No action is required for Hotspot owners.
Posted May 13, 2022 - 19:38 UTC
Update
Proof-of-Coverage Rates and Lower Rewards Update 3
1) Validators heard the siren. Theoretically, the 2700 good Validators should handle the load without them being out of commission. However, Hotspots are still connecting to the bad ones.. this brings us to number 2️⃣ .
2) Hotspots to connect to a different (better) validator is almost done. Once complete, the core team will do a brief test then make it available for Makers to send to their fleet of Hotspots.
3) The core team is increasing the number of challenges per block from 400 to 1000 to make up for lower PoC rates.
No action is required for Hotspot owners.
Posted May 13, 2022 - 19:37 UTC
Update
Proof-of-Coverage Rates and Lower Rewards Update 2
1) Core team has alerted all Validators to update ASAP. We're taking steps to move forward with methods to remove them from participating in Proof-of-Coverage.
2) Logic for Hotspots to reconnect to a different Validator is underway.
3) We have a candidate snapshot. The core team believes this new snapshot at a higher height will get Validators past the stall they're seeing (even if they're upgraded). This snapshot will also help syncing Hotspots catch up faster.
No action is required for Hotspot owners.
Posted May 13, 2022 - 16:03 UTC
Investigating
Overnight, the core team saw more PoC failures, lowered rewards, and stuck Validators (owned by validator operators that did not upgrade).
What we're seeing is that when a Hotspot connects to a Validator that gets stuck, the Hotspot's proof-of-coverage receipts is rejected by the stalled Validator, and the Hotspot never tries to reconnect to another.
Hotspot PoC -> Old Validator > Old Validator does not process the PoC -> Hotspot does not get rewarded.
We are looking to fix this in 3 ways: 1) Identify Validators who have not upgraded and tell them to ASAP. We may consider other methods of removing them from participating in Proof-of-coverage.
2) Add logic to Hotspots (will require a software update, update to come) to reconnect to a different Validator. @hotspot-maker be prepared to issue a software update today.
3) Ensure snapshots are available
No action is required for Hotspot owners.
This is all hands on deck for the core team and our highest priority. We will be updating this status when we know more.
Join our Discord at the #blockchain channel to see the core team work on a resolution. discord.gg/helium