Update - Proof-of-Coverage Update

I wanted to drop by with an update now that it's later in the afternoon on our current progress.

1️⃣ The fix is now in a miner image 05.18 on both Mainnet (on testing Hotspots) and Testnet. This is to fix the missing receipts issue. We're still monitoring the Hotspots but it is too early to tell. We will be back tomorrow to confirm that this is a release candidate and GA for Hotspot Makers.

2️⃣ This particular investigation will not be as fast as the others. There is quite a bit of surgery involved. The update here is.. we're still digging.

Lastly, the AMA video should be out shortly!

May 18, 23:29 UTC
Update - Hello Helium Community: The core team is working on 2 things today.

1️⃣ The core team found an issue overnight and is testing a fix on how a Hotspot communicates to its Validator to make it more reliable. This should increase the number of proof-of-coverage receipts sent back to the Network and should improve rewards.

This is getting testing time on Testnet as well as Mainnet on our Testing Hotspots. If this is a noticeable improvement, we will be issuing a Hotspot release (today or tomorrow).

2️⃣ The team, in collaboration with the community in the technical channels have also identified an interesting occurrence that likely manifests as a Hotspot “flatlining” or “doing nothing for days”:

There is 1 process in the Hotspot that silently dies and never alerts anyone. This process is needed to communicate with the Validator the Hotspot is connected to. If this thing dies, the Hotspot is unable wake itself back up to respond to Proof-of-Coverage or even send Witness receipts; hence flatline.

The core team is going to replace this process with another that already exists in the Validator code.

Unscientifically, this may explain why sometimes Hotspots come back to life (about 25% of the time) after a reboot. This is not something the core team is recommending Hotspot owners do, but if you want to experiment, there’s not much to lose.

Finally, the core team truly empathizes with Hotspot owners on the poor rewards. This is 100% the core team’s entire focus right now. It’s important not to point fingers at each other. We are in this together and in it for the long haul.

If we can get these two issues fixed to bring back reward stability to the network, we can turn off chain syncing as early as next week.

May 18, 19:07 UTC
Update - GM 🌅 Helium Community. We are staying the course. After the updates yesterday, we saw Proof-of-Coverage receipts stay above our threshold of rollback.

Overnight, we saw something happen on Testnet that we suspected, that is, a Hotspot was not able to tell its notifying Validator about its receipt - a likely cause of why some receipts are missing on mainnet.

Today, the core team will do 4 things:
1. chase down the bug we saw on Testnet. If there is a fix, there's going to be a release.
2. understand why sometimes receipts come in out of order and if it affects performance. If it does, fix it.
3. performance improvements to Validator heartbeats and receipts
4. consider increasing the Challenges per block rate from 700. There will be a separate announcement if this happens.

No action required for Hotspots, Makers, or Validators at this time.

May 17, 15:00 UTC
Update - Proof-of-Coverage receipts per block are trending in the right direction as Validators take the upgrade to 1.10.9.
May 17, 03:41 UTC
Update - Lowered PoC Rewards and Receipts Update 5

The core team is seeing very slow improvements to Proof-of-Coverage (and rewards) receipts since the Validator release. This slow change is likely due to Validators going offline to upgrade themselves, get back online and sync to the tip of the blockchain. We’re hopeful that this is the case for the slow improvements (about 30% are not upgraded/online) but we are prepared to issue a Light Hotspot rollback tomorrow morning Tuesday, May 17, 2022 6:00 AM PT

We have identified another bug (see Discord #blockchain ) where the Proof-of-Coverage manager on Validators are crashing out every 50 blocks and purges the challenges. This is being reviewed and will be part of a new @Validator 1.10.9 release. This is not a silver-bullet bug fix, but rather looks to stabilize the receipts we already have from its current pattern so we can have a better view of the Proof-of-Coverage landscape tomorrow morning.

We will roll back the Light Hotspot transition at Tuesday, May 17, 2022 6:00 AM PT if:
1. More than 90% of Validators upgraded (at least 1.10.8) AND
2. Proof-of-Coverage receipts per block is still consistently below 100 receipts.

What will happen if we rollback:
1. Hotspots will challenge again. All Hotspots are still syncing the blockchain and there are consistent snapshots being produced to help those that are out of sync. No action required by Hotspot owners or Makers
2. Validators will continue performing Consensus Group duties and will no longer create Challenges.
3. We will continue our investigation on Testnet and will re-attempt activation at another time.

May 16, 23:14 UTC
Update - Lowered PoC Rewards and Receipts Update 4

About 66% of Validators have upgraded to the latest release.

Once a Validator is updated, it will take 200 blocks (or approximately 3.5 hours in 🕐 time) for them to discard invalid proof-of-coverage challenges and start issuing the updated versions that will result in the graph going 📈 .

TLDR: need more time for rewards to come back and stabilize as Validators upgrade and shed their old challenges. So if your Hotspot is connected to a Validator that just upgraded, it will be 3.5 hours before your Hotspot will be rewarded properly. The graph will not hockey stick.

No action required for Hotspot owners.

May 16, 21:17 UTC
Update - Lowered PoC Rewards and Receipts Update 3

The Validator Upgrade has been available for ~1 hour and 269 Validators have upgraded. We need to see much more Validators update to 1.10.8 before we can see noticeable improvements to Proof-of-Coverage and rewards. The team is continuing to monitor dashboard.helium.com.

No action is required for Hotspot owners.

May 16, 21:17 UTC
Identified - Lowered PoC Rewards and Receipts Update 2

The core team has tagged a release for Validator Operators that includes the bug fix for the known Proof-of-Coverage issue.

After the Validator update goes out, all Validators will be required to take this update ASAP. We will slowly start to see improvements (over the course of 2-3 hours). We are keeping the Challenges per block at 700 for now.

No action required for Hotspot owners.

May 16, 17:44 UTC
Investigating - The core team is continuing to debug the root cause of lowered Proof-of-Coverage receipts (and consequently lowered rewards). We may have a lead and will update community as soon as it's confirmed and we have a fix.

Also note that if you have altered your sys.config file on your Hotspot, you may have inadvertently broke your Hotspot's ability to participate in Proof-of-Coverage. Please check with your Hotspot Maker on what the correct configuration settings should be.

If you have no idea what sys.config means, no action required - the likely cause of low rewards is what the devs are working on.

May 16, 14:27 UTC
Consensus Groups Operational
90 days ago
100.0 % uptime
Today
Proof-of-Coverage Degraded Performance
90 days ago
99.87 % uptime
Today
Router Operational
90 days ago
100.0 % uptime
Today
Onboarding Service Operational
90 days ago
99.64 % uptime
Today
Explorer Operational
90 days ago
99.96 % uptime
Today
Helium App Operational
90 days ago
99.89 % uptime
Today
User Apps Operational
90 days ago
99.94 % uptime
Today
Helium Wallet App ? Operational
90 days ago
100.0 % uptime
Today
Blockchain Operational
90 days ago
100.0 % uptime
Today
API Operational
90 days ago
99.85 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Past Incidents
May 19, 2022

No incidents reported today.

May 18, 2022

Unresolved incident: Lowered Proof-of-Coverage Rate and Rewards.

May 17, 2022
May 16, 2022
May 15, 2022

No incidents reported.

May 14, 2022
Resolved - This incident has been resolved.
May 14, 05:15 UTC
Monitoring - Proof-of-Coverage Rates and Stabilizing Rewards Update 9

At this point, the team is monitoring Proof-of-Coverage receipts and overall network health. There are no outstanding items for the team to address. Unless something goes wrong, there will be no more updates from me.

If you're interested in the metrics, visit https://dashboard.helium.com/. The green line represents Proof-of-Coverage receipts with witnesses (higher is good), the red line represents Proof-of-Coverage receipts with 0 witnesses (we want this as low as possible).

Thanks everyone for their continued patience 🎈

May 14, 00:28 UTC
Identified - Proof-of-Coverage Rates and Lower Rewards Update 8

The Hotspot release is now available. Makers will take this release and send the update to their fleet of Hotspots. It is up to each individual Hotspot manufacturer to send the update to your Hotspot.

This update allows Hotspots to disconnect from a poorly performing Validator and search for a new one if the connected validator is not keeping up with the chain for 15 blocks (approximately 15 minutes).

No action is required for Hotspot owners.

May 13, 22:17 UTC
Update - Proof-of-Coverage Rates and Lower Rewards Update 7

1) Validator release is out. We're seeing PoC receipts trending in the right direction (UpOnly) as Validator operators slowly take the update.

2) Hotspot release is now in testing.

No action is required for Hotspot owners.

May 13, 22:03 UTC
Update - Proof-of-Coverage Rates and Lower Rewards Update 6

1) A Validator release is now available and is mandatory. Validators are upgrading as we speak. It should fix the PoC receipt delivery that Hotspots have been tirelessly trying to send.

2) The Hotspot release is almost through reviews. Testing next. While this fixes connecting to a better Validator, the real fix lies in 1️⃣ with the Validator release.

The team will monitor 1️⃣ and proceed down the path of Light Hotspots and will not rollback at this time.

No action is required for Hotspot owners.

May 13, 20:53 UTC
Update - Proof-of-Coverage Rates and Lower Rewards Update 5

1) In addition to the Hotspot candidate release, we are looking at a potential fix on a Validator side that requires a @Validator Update. When we have a Validator candidate release, we will let Validator Operators know.

2) Other action we're considering taking: Increase the challenges per block some more

No action is required for Hotspot owners.

May 13, 19:38 UTC
Update - Proof-of-Coverage Rates and Lower Rewards Update 4

1) We have a candidate release that tells a Hotspot to connect to a different Validator. Just needs code review and approvals before testing.

2) The team will continue down the path of Light Hotspots and will keep working. We will not rollback at Friday, May 13, 2022 12:00 PM and instead see how the Hotspot software update performs.

No action is required for Hotspot owners.

May 13, 19:38 UTC
Update - Proof-of-Coverage Rates and Lower Rewards Update 3

1) Validators heard the siren. Theoretically, the 2700 good Validators should handle the load without them being out of commission. However, Hotspots are still connecting to the bad ones.. this brings us to number 2️⃣ .

2) Hotspots to connect to a different (better) validator is almost done. Once complete, the core team will do a brief test then make it available for Makers to send to their fleet of Hotspots.

3) The core team is increasing the number of challenges per block from 400 to 1000 to make up for lower PoC rates.

No action is required for Hotspot owners.

May 13, 19:37 UTC
Update - Proof-of-Coverage Rates and Lower Rewards Update 2

1) Core team has alerted all Validators to update ASAP. We're taking steps to move forward with methods to remove them from participating in Proof-of-Coverage.

2) Logic for Hotspots to reconnect to a different Validator is underway.

3) We have a candidate snapshot. The core team believes this new snapshot at a higher height will get Validators past the stall they're seeing (even if they're upgraded). This snapshot will also help syncing Hotspots catch up faster.

No action is required for Hotspot owners.

May 13, 16:03 UTC
Investigating - Overnight, the core team saw more PoC failures, lowered rewards, and stuck Validators (owned by validator operators that did not upgrade).

What we're seeing is that when a Hotspot connects to a Validator that gets stuck, the Hotspot's proof-of-coverage receipts is rejected by the stalled Validator, and the Hotspot never tries to reconnect to another.

Hotspot PoC -> Old Validator > Old Validator does not process the PoC -> Hotspot does not get rewarded.

We are looking to fix this in 3 ways:
1) Identify Validators who have not upgraded and tell them to ASAP. We may consider other methods of removing them from participating in Proof-of-coverage.

2) Add logic to Hotspots (will require a software update, update to come) to reconnect to a different Validator. @hotspot-maker be prepared to issue a software update today.

3) Ensure snapshots are available

No action is required for Hotspot owners.

This is all hands on deck for the core team and our highest priority. We will be updating this status when we know more.

Join our Discord at the #blockchain channel to see the core team work on a resolution. discord.gg/helium

May 13, 15:13 UTC
May 13, 2022
May 12, 2022
Resolved - All required updates to databases are complete.
May 12, 16:31 UTC
Identified - The emergency maintenance last night allowed our API provider to migrate databases, but there are some databases/replicas still inaccessible. Until those are brought back up, this status will remain open.
May 11, 14:02 UTC
Update - The ETL is now catching up and we will be turning the web servers back on in the next 20 minutes. Thanks for hanging in there.
May 11, 07:52 UTC
Investigating - We are currently investigating an issue with the backend database that provides blockchain information to App and Explorer.

During this time:
1. submitting transactions will be unavailable in the Helium Hotspot App and Helium Wallet App.
* transactions include Add Hotspot, Assert Location, Payments, Burns, and Transfer Hotspots

2. information displayed in the Hotspot App, Wallet App, and Explorer may be delayed.

The Blockchain and Network are not affected.
Mining Rewards, device packet transfer, are not affected

May 9, 17:19 UTC
May 11, 2022
Completed - The scheduled maintenance has been completed.
May 11, 10:00 UTC
Verifying - The ETL is back online and the webservers are slowly ramping up. The ETL is behind the blockchain roughly 350 blocks so it will take several hours to catch up. We will continue to monitor and close this once the ETL is caught up.
May 11, 08:29 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 11, 01:00 UTC
Scheduled - API will be unavailable during this time to undergo maintenance to migrate the database in preparation for Light Hotspots.

During this time:
1. submitting transactions will be unavailable in the Helium Hotspot App and Helium Wallet App.
* transactions include Add Hotspot, Assert Location, Payments, Burns, and Transfer Hotspots

2. information displayed in the Hotspot App, Wallet App, and Explorer will be unavailable.

The Blockchain and Network are not affected.
Mining Rewards, device packet transfer, are not affected

May 11, 00:59 UTC
May 10, 2022

No incidents reported.

May 9, 2022
May 8, 2022

No incidents reported.

May 7, 2022
Resolved - This incident has been resolved.
May 7, 21:39 UTC
Monitoring - A fix has been implemented and we are monitoring the results.
May 5, 23:15 UTC
Investigating - We are currently investigating an issue with the backend database that provides blockchain information to App and Explorer.

During this time:
1. submitting transactions will be unavailable in the Helium Hotspot App and Helium Wallet App.
* transactions include Add Hotspot, Assert Location, Payments, Burns, and Transfer Hotspots

2. information displayed in the Hotspot App, Wallet App, and Explorer may be delayed.

The Blockchain and Network are not affected.
Mining Rewards, device packet transfer, are not affected

May 5, 19:14 UTC
May 6, 2022

No incidents reported.

May 5, 2022