Update - Proof-of-Coverage Update
I wanted to drop by with an update now that it's later in the afternoon on our current progress.
1️⃣ The fix is now in a miner image 05.18 on both Mainnet (on testing Hotspots) and Testnet. This is to fix the missing receipts issue. We're still monitoring the Hotspots but it is too early to tell. We will be back tomorrow to confirm that this is a release candidate and GA for Hotspot Makers.
2️⃣ This particular investigation will not be as fast as the others. There is quite a bit of surgery involved. The update here is.. we're still digging.
Lastly, the AMA video should be out shortly!
May 18, 23:29 UTC
I wanted to drop by with an update now that it's later in the afternoon on our current progress.
1️⃣ The fix is now in a miner image 05.18 on both Mainnet (on testing Hotspots) and Testnet. This is to fix the missing receipts issue. We're still monitoring the Hotspots but it is too early to tell. We will be back tomorrow to confirm that this is a release candidate and GA for Hotspot Makers.
2️⃣ This particular investigation will not be as fast as the others. There is quite a bit of surgery involved. The update here is.. we're still digging.
Lastly, the AMA video should be out shortly!
May 18, 23:29 UTC
Update - Hello Helium Community: The core team is working on 2 things today.
1️⃣ The core team found an issue overnight and is testing a fix on how a Hotspot communicates to its Validator to make it more reliable. This should increase the number of proof-of-coverage receipts sent back to the Network and should improve rewards.
This is getting testing time on Testnet as well as Mainnet on our Testing Hotspots. If this is a noticeable improvement, we will be issuing a Hotspot release (today or tomorrow).
2️⃣ The team, in collaboration with the community in the technical channels have also identified an interesting occurrence that likely manifests as a Hotspot “flatlining” or “doing nothing for days”:
There is 1 process in the Hotspot that silently dies and never alerts anyone. This process is needed to communicate with the Validator the Hotspot is connected to. If this thing dies, the Hotspot is unable wake itself back up to respond to Proof-of-Coverage or even send Witness receipts; hence flatline.
The core team is going to replace this process with another that already exists in the Validator code.
Unscientifically, this may explain why sometimes Hotspots come back to life (about 25% of the time) after a reboot. This is not something the core team is recommending Hotspot owners do, but if you want to experiment, there’s not much to lose.
Finally, the core team truly empathizes with Hotspot owners on the poor rewards. This is 100% the core team’s entire focus right now. It’s important not to point fingers at each other. We are in this together and in it for the long haul.
If we can get these two issues fixed to bring back reward stability to the network, we can turn off chain syncing as early as next week.
May 18, 19:07 UTC
1️⃣ The core team found an issue overnight and is testing a fix on how a Hotspot communicates to its Validator to make it more reliable. This should increase the number of proof-of-coverage receipts sent back to the Network and should improve rewards.
This is getting testing time on Testnet as well as Mainnet on our Testing Hotspots. If this is a noticeable improvement, we will be issuing a Hotspot release (today or tomorrow).
2️⃣ The team, in collaboration with the community in the technical channels have also identified an interesting occurrence that likely manifests as a Hotspot “flatlining” or “doing nothing for days”:
There is 1 process in the Hotspot that silently dies and never alerts anyone. This process is needed to communicate with the Validator the Hotspot is connected to. If this thing dies, the Hotspot is unable wake itself back up to respond to Proof-of-Coverage or even send Witness receipts; hence flatline.
The core team is going to replace this process with another that already exists in the Validator code.
Unscientifically, this may explain why sometimes Hotspots come back to life (about 25% of the time) after a reboot. This is not something the core team is recommending Hotspot owners do, but if you want to experiment, there’s not much to lose.
Finally, the core team truly empathizes with Hotspot owners on the poor rewards. This is 100% the core team’s entire focus right now. It’s important not to point fingers at each other. We are in this together and in it for the long haul.
If we can get these two issues fixed to bring back reward stability to the network, we can turn off chain syncing as early as next week.
May 18, 19:07 UTC
Update - GM 🌅 Helium Community. We are staying the course. After the updates yesterday, we saw Proof-of-Coverage receipts stay above our threshold of rollback.
Overnight, we saw something happen on Testnet that we suspected, that is, a Hotspot was not able to tell its notifying Validator about its receipt - a likely cause of why some receipts are missing on mainnet.
Today, the core team will do 4 things:
1. chase down the bug we saw on Testnet. If there is a fix, there's going to be a release.
2. understand why sometimes receipts come in out of order and if it affects performance. If it does, fix it.
3. performance improvements to Validator heartbeats and receipts
4. consider increasing the Challenges per block rate from 700. There will be a separate announcement if this happens.
No action required for Hotspots, Makers, or Validators at this time.
May 17, 15:00 UTC
Overnight, we saw something happen on Testnet that we suspected, that is, a Hotspot was not able to tell its notifying Validator about its receipt - a likely cause of why some receipts are missing on mainnet.
Today, the core team will do 4 things:
1. chase down the bug we saw on Testnet. If there is a fix, there's going to be a release.
2. understand why sometimes receipts come in out of order and if it affects performance. If it does, fix it.
3. performance improvements to Validator heartbeats and receipts
4. consider increasing the Challenges per block rate from 700. There will be a separate announcement if this happens.
No action required for Hotspots, Makers, or Validators at this time.
May 17, 15:00 UTC
Update - Proof-of-Coverage receipts per block are trending in the right direction as Validators take the upgrade to 1.10.9.
May 17, 03:41 UTC
May 17, 03:41 UTC
Update - Lowered PoC Rewards and Receipts Update 5
The core team is seeing very slow improvements to Proof-of-Coverage (and rewards) receipts since the Validator release. This slow change is likely due to Validators going offline to upgrade themselves, get back online and sync to the tip of the blockchain. We’re hopeful that this is the case for the slow improvements (about 30% are not upgraded/online) but we are prepared to issue a Light Hotspot rollback tomorrow morning Tuesday, May 17, 2022 6:00 AM PT
We have identified another bug (see Discord #blockchain ) where the Proof-of-Coverage manager on Validators are crashing out every 50 blocks and purges the challenges. This is being reviewed and will be part of a new @Validator 1.10.9 release. This is not a silver-bullet bug fix, but rather looks to stabilize the receipts we already have from its current pattern so we can have a better view of the Proof-of-Coverage landscape tomorrow morning.
We will roll back the Light Hotspot transition at Tuesday, May 17, 2022 6:00 AM PT if:
1. More than 90% of Validators upgraded (at least 1.10.8) AND
2. Proof-of-Coverage receipts per block is still consistently below 100 receipts.
What will happen if we rollback:
1. Hotspots will challenge again. All Hotspots are still syncing the blockchain and there are consistent snapshots being produced to help those that are out of sync. No action required by Hotspot owners or Makers
2. Validators will continue performing Consensus Group duties and will no longer create Challenges.
3. We will continue our investigation on Testnet and will re-attempt activation at another time.
May 16, 23:14 UTC
The core team is seeing very slow improvements to Proof-of-Coverage (and rewards) receipts since the Validator release. This slow change is likely due to Validators going offline to upgrade themselves, get back online and sync to the tip of the blockchain. We’re hopeful that this is the case for the slow improvements (about 30% are not upgraded/online) but we are prepared to issue a Light Hotspot rollback tomorrow morning Tuesday, May 17, 2022 6:00 AM PT
We have identified another bug (see Discord #blockchain ) where the Proof-of-Coverage manager on Validators are crashing out every 50 blocks and purges the challenges. This is being reviewed and will be part of a new @Validator 1.10.9 release. This is not a silver-bullet bug fix, but rather looks to stabilize the receipts we already have from its current pattern so we can have a better view of the Proof-of-Coverage landscape tomorrow morning.
We will roll back the Light Hotspot transition at Tuesday, May 17, 2022 6:00 AM PT if:
1. More than 90% of Validators upgraded (at least 1.10.8) AND
2. Proof-of-Coverage receipts per block is still consistently below 100 receipts.
What will happen if we rollback:
1. Hotspots will challenge again. All Hotspots are still syncing the blockchain and there are consistent snapshots being produced to help those that are out of sync. No action required by Hotspot owners or Makers
2. Validators will continue performing Consensus Group duties and will no longer create Challenges.
3. We will continue our investigation on Testnet and will re-attempt activation at another time.
May 16, 23:14 UTC
Update - Lowered PoC Rewards and Receipts Update 4
About 66% of Validators have upgraded to the latest release.
Once a Validator is updated, it will take 200 blocks (or approximately 3.5 hours in 🕐 time) for them to discard invalid proof-of-coverage challenges and start issuing the updated versions that will result in the graph going 📈 .
TLDR: need more time for rewards to come back and stabilize as Validators upgrade and shed their old challenges. So if your Hotspot is connected to a Validator that just upgraded, it will be 3.5 hours before your Hotspot will be rewarded properly. The graph will not hockey stick.
No action required for Hotspot owners.
May 16, 21:17 UTC
About 66% of Validators have upgraded to the latest release.
Once a Validator is updated, it will take 200 blocks (or approximately 3.5 hours in 🕐 time) for them to discard invalid proof-of-coverage challenges and start issuing the updated versions that will result in the graph going 📈 .
TLDR: need more time for rewards to come back and stabilize as Validators upgrade and shed their old challenges. So if your Hotspot is connected to a Validator that just upgraded, it will be 3.5 hours before your Hotspot will be rewarded properly. The graph will not hockey stick.
No action required for Hotspot owners.
May 16, 21:17 UTC
Update - Lowered PoC Rewards and Receipts Update 3
The Validator Upgrade has been available for ~1 hour and 269 Validators have upgraded. We need to see much more Validators update to 1.10.8 before we can see noticeable improvements to Proof-of-Coverage and rewards. The team is continuing to monitor dashboard.helium.com.
No action is required for Hotspot owners.
May 16, 21:17 UTC
The Validator Upgrade has been available for ~1 hour and 269 Validators have upgraded. We need to see much more Validators update to 1.10.8 before we can see noticeable improvements to Proof-of-Coverage and rewards. The team is continuing to monitor dashboard.helium.com.
No action is required for Hotspot owners.
May 16, 21:17 UTC
Identified - Lowered PoC Rewards and Receipts Update 2
The core team has tagged a release for Validator Operators that includes the bug fix for the known Proof-of-Coverage issue.
After the Validator update goes out, all Validators will be required to take this update ASAP. We will slowly start to see improvements (over the course of 2-3 hours). We are keeping the Challenges per block at 700 for now.
No action required for Hotspot owners.
May 16, 17:44 UTC
The core team has tagged a release for Validator Operators that includes the bug fix for the known Proof-of-Coverage issue.
After the Validator update goes out, all Validators will be required to take this update ASAP. We will slowly start to see improvements (over the course of 2-3 hours). We are keeping the Challenges per block at 700 for now.
No action required for Hotspot owners.
May 16, 17:44 UTC
Investigating - The core team is continuing to debug the root cause of lowered Proof-of-Coverage receipts (and consequently lowered rewards). We may have a lead and will update community as soon as it's confirmed and we have a fix.
Also note that if you have altered your sys.config file on your Hotspot, you may have inadvertently broke your Hotspot's ability to participate in Proof-of-Coverage. Please check with your Hotspot Maker on what the correct configuration settings should be.
If you have no idea what sys.config means, no action required - the likely cause of low rewards is what the devs are working on.
May 16, 14:27 UTC
Also note that if you have altered your sys.config file on your Hotspot, you may have inadvertently broke your Hotspot's ability to participate in Proof-of-Coverage. Please check with your Hotspot Maker on what the correct configuration settings should be.
If you have no idea what sys.config means, no action required - the likely cause of low rewards is what the devs are working on.
May 16, 14:27 UTC
Consensus Groups
Operational
Proof-of-Coverage
Degraded Performance
Router
Operational
Onboarding Service
Operational
Explorer
Operational
Helium App
Operational
User Apps
Operational
Helium Wallet App
?
Operational
Blockchain
Operational
API
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.