The core team has shifted focus on creating an off-chain PoC mechanism (a HIP will be proposed) that would allow Hotspots to beacon multiple times a day and allow more witnesses than before.
The work that's on-chain today (increased challenges per block, activated chain variables, blockchain sync turned off) should help alleviate lack of PoC activity for the next few months. While there is room for improvement, the team believes the ultimate goal of off-chain PoC will satisfy the needs of Hotspot owners and the network overall.
Posted Jul 18, 2022 - 19:27 UTC
Update
The core team is prepared to activate 6 chain variables related to HIP 54: H3Dex. In addition to enabling HIP 54, the chain variables also include improvements to proof-of-coverage:  . Updates HIP15/17 values so that Hotspot in less-dense areas are not starved of rewards. Keep in mind that PoC selection is still random, so any improvements (in terms of earnings) can take a while (days++) to see.  . A fix that should see more PoC receipts coming in per block. Currently, 250 Challenges go out per block and about 170-180 receipts come back per block. At this rate, the average Hotspot beacons every 3-4 days. Once the fix is in, Hotspots may beacon more frequently than the average 3-4 days, but we will have to wait and see how well this works once activated. The goal is to get as close to 250 receipts back, instead of 170-180 we get today.
Once these chain variables are activated, we expect to see a blockchain performance improvement and more Proof-of-Coverage receipts received, per block, over a period of several days.
Posted Jun 30, 2022 - 17:55 UTC
Update
Hotspot Release Available: Flatline Fix The core team has released a new Hotspot release to fix the flatline issue! Before explaining what the fix entails, I want set expectations that the flatline fix will not be *immediately* visible. A Hotspot Maker will need to deploy the update to their fleet of Hotspots (which can take several days! Keep an eye out on your maker’s discord or social media), then it can take 3-4 days for a Hotspot to beacon or witness and be rewarded.
Flatlined Hotspots can take up to 7 days to come back to life with this bug fix.
A summary of the flatline fix There were two issues have been causing various crashes and out of memory errors on Hotspots that have been preventing them from participating in network and potentially causing "flatline" reports in the community.
Fix 1. The team identified an issue where Hotspots were talking to both Mainnet and Testnet causing a Hotspot to run out of memory and freeze/flatline. Fix 2. A Hotspot can crash due to a latent bug in the bluetooth stack
Makers have been notified of this release and will send the update over-the-air to their Hotspot fleet. Hotspots connected to the internet will automatically download and install this update. Once the update is applied, flatlined Hotspots can take up to 7 days to come back to life. No user-action is required.
If your Hotspot is all alone in a hex without any neighbors that can hear you beacon, the only rewards your Hotspot may earn is Data transfer rewards only. To resolve this, you’ll need more Hotspots in your area to participate in Proof-of-Coverage.
**A note about Hotspots in less dense areas seeing less Proof-of-Coverage Activity** The fix for this is part of a Validator release that is still in testing and requires chain variable activation. There is no update at this time but the activation will not be this week, and will be no sooner than late next week.
Posted Jun 10, 2022 - 23:01 UTC
Update
Helium Community,
The core team believes they have a good handle on the missing receipts and will be issuing several changes this week.
1️⃣ Things we found (with the help of technical community, thanks BigEnigma!) • The Proof-of-Coverage logic has versioning built in. BigEnigma found that the versioning (and thus challenger targeting) was mismatched between ledgers. This is also contributing to receipt timing issues and receipts coming in out of order.
Receipts out of order and the versioning mismatch can cause a number of proof-of-coverage receipts to be dropped, Hotspots not credited for their work, and leads to lowered Hotspot earnings.
This will be fixed in a Validator Release and is in review.
• When a hex is targeted for a Proof-of-Coverage challenge, the logic is backwards. It is currently selecting 14 witnesses first, then filtering for inactive (aka offline) Hotspots, often leading to
• When will it be fix? These changes will be in a Validator Release, so once we have a release candidate and an activation date set, we’ll let the community know.
2️⃣ Light Hotspot Chain Syncing - When will chain sync be turned off? • The core team will be releasing a new Hotspot release this week for Makers that will re-enable mux. There is a small bug that the team is investigating when chain sync is turned off, but once this bug is squashed, we can turn off chain syncing.
• Our target for this no later than next week.
3️⃣ Some Hotspots on the Denylist earning? What’s going on? • With the help of community, we've identified a Validator Operator with custom configuration overrides that is not able to pick up the Denylist list. The operator has been informed and will be fixing this in the next week or so.
• Hotspots that use Validators serviced by this Validator Operator will not be able to exclude Denylist Hotspots participating in proof-of-coverage, leading to some Denylisted Hotspots earning.
4️⃣ Hotspot Flatlining • Core team has recently found a new cause of flatlines. We have an engineer assigned to dig a bit more and if there's a fix, I'll let community know.
Posted Jun 06, 2022 - 17:52 UTC
Update
Proof-of-Coverage Update: Hotspot Update Now Available for Makers
The core team has made a new Hotspot update available for Makers to distribute to their fleet.
This update will have additional fixes for “flatline”, fixes some of the timing with respect to submitting Proof-of-Coverage receipts, and picking a different Validator faster, if the first Validator is behind on blocks (Validators that are behind on blocks cannot properly challenge, affecting a Hotpot’s beacon activity).
To see if your Hotspot’s maker has pushed the update, please check in with their Discord servers or on their social media accounts.
Hotspots receive this update over-the-air and will automatically apply the update if it’s online and connected to the internet. No action required for Hotspot owners.
Hotspot owners should see an overall improvement in Proof-of-Coverage over the next 48-72 hours after the update is installed on your Hotspot.
Posted Jun 01, 2022 - 22:04 UTC
Update
Since the last chain variable activation, the blockchain has stabilized enough for us to look at the data to identify further areas of improvement. The main data point the team is focused on is the number of PoC receipts coming back, per block.
Current Variables: The target Challenges per block is 250. The number coming back has been consistently between 70-71% (around 170 receipts per block). There are 30% of receipts missing. Why?
Missing Receipts 30% of receipts are missing which could be due to Hotspot flatlines, Validators being out of sync, or Validators being out of sync, then resyncing quickly.
To address missing receipts, the core team will be putting together a Hotspot release to help reduce the 30% value and enable more Hotspots to participate in PoC.
1. Hotspot Flatline: there will be another HTTP2 fix in the upcoming Hotspot release. This is being tested on Testnet at the moment. We expect this to bring a few more Hotspots back to life and reduce the 30% number.
2. Validators being out of sync: Hotspots will ditch out of sync Validators sooner (instead of waiting 20 blocks, Hotspots will wait 10 blocks before looking for a new Validator). This provides a better chance for a Hotspot to connect to a good validator for PoC and should help reduce the 30% number.
3. Validators being out of sync for a bit, quickly syncing again, and the Hotspot’s proof-of-coverage challenge becomes invalid. The challenge should not be considered invalid. The fix will also be in this Hotspot release.
We are aiming to push out this Hotspot release to Makers by Thursday 00:00 UTC.
Further Investigations this week The team will be working with community this week to see if there is a correlation within the 30% of receipts missed in terms of maker type, region, or density by running the targeting logic.
This may or may not yield meaningful results, but it’s worth the investigation.
Posted Jun 01, 2022 - 14:37 UTC
Update
The "Flatline" Fix has been well-received by Hotspots. About 36,000 more Hotspots beaconed in the last 24 hours (as noted in explorer.helium.com/hotspots) than before the fix. We will continue to monitor over the weekend.
Posted May 27, 2022 - 23:15 UTC
Update
The core team has tagged a new release for Hotspot Makers 2022.05.24.0. It is up to each Hotspot Manufacturer to send the software update over-the-air to their fleet.
Posted May 25, 2022 - 00:02 UTC
Update
GM 🌅 Helium!
We have optimistic news today thanks to help provided by one of our own community members nmc. With their help, the core team was able to find a root cause of the “flatline” that some Hotspots had been seeing since activating Light Hotspots.
To explain it simply: on a Hotspot, one of the lower-level processes (HTTP2 client) that is used when connecting to a challenger, stalls, and never returns since there is no timeout explicitly set. The fix is to add a default timeout and a monitor so it knows to “wake back up”.
Next steps for the fix 1. The core team will send the fix to Testnet and Mainnet on alpha and beta Hotspots as soon as it is ready. 1-2 hours from now. Let it sit for an hour and confirm that everything works. 2. Tag the build for Hotspot Makers as “Urgent” and “Mandatory”. 3. Hotspot Makers will send the update down to their fleet.
Posted May 24, 2022 - 15:49 UTC
Update
Proof-of-Coverage Update 2 May 20
What to expect over the weekend and what core team is looking at next week:
1️⃣ Network Stability. We had some big chunky blocks today - all stemming from Proof-of-Challenge receipts when we had poc_challenge_rate ramped up to 700.
How did this happen? We had a bunch of bugs that we fixed over the last 9 days + we activated a chain variable this morning that kept more Challenges active. This ultimately increased the number of valid receipts received on the chain.
More receipts = more time spent validating each receipt = bigger block times = unstable network.
To keep things moving, the team reduced challenges per block to 200 and now it’s at 300.
Why 300 today is better than 700 last week About 20% of receipts per block were coming through last week (~140 receipts). With all the updates we made since, we’re seeing numbers much closer to the target of 300. This is better for you as a Hotspot owner at the end of the day.
More receipts = more witnesses = better rewards. And with stable block times, there continues to be consistent rewards.
2️⃣ Validator Release with a bunch of improvements are going to be tested over the weekend. It will be available for Validator Operators shortly.
Included in the Validator release will be improvements to Hotspot inactivity (when do you start selecting a Hotspot for Beaconing again; should be more often with this release), improvements to HIP17 targeting in less-dense Hotspot areas, witness receipt validation improvements (for network stability), and more.
Posted May 21, 2022 - 00:22 UTC
Update
Chain Variable Activation: poc_apply_gc_fix
The team is activating the chain variable poc_apply_gc_fix that will greatly reduce the number of "invalid" or "expired" Proof-of-Coverage challenges. The outcome for Hotspot owners will be higher witness and beacon counts.
Posted May 20, 2022 - 17:58 UTC
Update
Proof-of-Coverage Update May 20
GM 🌅
Team is working on a handful of improvements today, including digging into the http2 Hotspot process crash and Validator-side changes.
This morning, the team will activate a Chain Variable that will greatly reduce the number of "invalid" or "expired" Proof-of-Coverage challenges. We'll let you know when that goes out. The outcome for Hotspot owners will be higher witness and beacon counts.
1️⃣ Manufacturers are still pushing the update out from yesterday. We should continue to see improvements over the next 24 hours.
2️⃣ The team does not have a solution yet for the process crash on Hotspots (good chance it's related to the "flatlines"). Unscientifically, a Hotspot reboot may fix it 25% of the time but the core team does not recommend doing this at this time - do so at your own risk.
3️⃣ The Validator candidate release has a lot of fantastic fixes. Included are: - the HIP17 fix for less dense Hotspot areas* - improvements to witness validation (particularly useful to Validator performance and will stabilize block times)* - better identification of "active" Hotspots for targeting challenges*
*All of these are also controlled by a chain variable, so we won't be able to see its benefits until we activate them next week.
Posted May 20, 2022 - 17:58 UTC
Update
Proof-of-Coverage Update 2
1️⃣ The core team has tagged a new release (2022.05.19.0) for Hotspot Makers to pick up and send down to their respective fleets. Original Helium Hotspots will pick up this update over the next 24 hours (that's how often it checks for a software update).
We expect to see receipts grow over time as more Makers pick up this change.
No action is required by Hotspot owners
2️⃣ In addition to a Hotspot release, the core team has identified a timing fix related to collecting invalid Proof-of-Coverage challenges too early that may recapture some challenges that should still be valid, but was marked invalid.
3️⃣ The Hotspot process crash is still being investigated. No new news.
4️⃣ A bug introduced in HIP17 related to hex targeting is causing less-dense Hotspot regions see their rewards diminish over time. The team is putting together a Validator-side fix that will be controlled by a chain variable. The expected activation will be next week.
5️⃣ As the core team improves on the receipt delivery on the chain, Validators will have more receipts to validate. As a result, block times will increase. To help mitigate this, the team is also working on receipt transaction improvements. This is mostly an FYI for the community as it likely has no bearing on your Hotspot performance.
Posted May 19, 2022 - 17:18 UTC
Update
Proof-of-Coverage Update
GM 🌞
The team has been testing a fix to address the missing receipts issue overnight on mainnet test Hotspots and on Testnet. Results indicate a minor improvement but we think it will still make a difference in PoC receipts and rewards.
An OTA is being prepared and will be made available for makers in a few hours. It is up to each individual maker to send the OTA update to their fleet of Hotspots.
There may be action required for Hotspot owners, but check with your maker to be sure.
Secondly, the core team is still working on fixing the dying process that I mentioned in point 2️⃣ above. This is likely also the reason why the average receipts over a long period of time seems to be trending down slowly.
Posted May 19, 2022 - 17:18 UTC
Update
Proof-of-Coverage Update
I wanted to drop by with an update now that it's later in the afternoon on our current progress.
1️⃣ The fix is now in a miner image 05.18 on both Mainnet (on testing Hotspots) and Testnet. This is to fix the missing receipts issue. We're still monitoring the Hotspots but it is too early to tell. We will be back tomorrow to confirm that this is a release candidate and GA for Hotspot Makers.
2️⃣ This particular investigation will not be as fast as the others. There is quite a bit of surgery involved. The update here is.. we're still digging.
Lastly, the AMA video should be out shortly!
Posted May 18, 2022 - 23:29 UTC
Update
Hello Helium Community: The core team is working on 2 things today.
1️⃣ The core team found an issue overnight and is testing a fix on how a Hotspot communicates to its Validator to make it more reliable. This should increase the number of proof-of-coverage receipts sent back to the Network and should improve rewards.
This is getting testing time on Testnet as well as Mainnet on our Testing Hotspots. If this is a noticeable improvement, we will be issuing a Hotspot release (today or tomorrow).
2️⃣ The team, in collaboration with the community in the technical channels have also identified an interesting occurrence that likely manifests as a Hotspot “flatlining” or “doing nothing for days”:
There is 1 process in the Hotspot that silently dies and never alerts anyone. This process is needed to communicate with the Validator the Hotspot is connected to. If this thing dies, the Hotspot is unable wake itself back up to respond to Proof-of-Coverage or even send Witness receipts; hence flatline.
The core team is going to replace this process with another that already exists in the Validator code.
Unscientifically, this may explain why sometimes Hotspots come back to life (about 25% of the time) after a reboot. This is not something the core team is recommending Hotspot owners do, but if you want to experiment, there’s not much to lose.
Finally, the core team truly empathizes with Hotspot owners on the poor rewards. This is 100% the core team’s entire focus right now. It’s important not to point fingers at each other. We are in this together and in it for the long haul.
If we can get these two issues fixed to bring back reward stability to the network, we can turn off chain syncing as early as next week.
Posted May 18, 2022 - 19:07 UTC
Update
GM 🌅 Helium Community. We are staying the course. After the updates yesterday, we saw Proof-of-Coverage receipts stay above our threshold of rollback.
Overnight, we saw something happen on Testnet that we suspected, that is, a Hotspot was not able to tell its notifying Validator about its receipt - a likely cause of why some receipts are missing on mainnet.
Today, the core team will do 4 things: 1. chase down the bug we saw on Testnet. If there is a fix, there's going to be a release. 2. understand why sometimes receipts come in out of order and if it affects performance. If it does, fix it. 3. performance improvements to Validator heartbeats and receipts 4. consider increasing the Challenges per block rate from 700. There will be a separate announcement if this happens.
No action required for Hotspots, Makers, or Validators at this time.
Posted May 17, 2022 - 15:00 UTC
Update
Proof-of-Coverage receipts per block are trending in the right direction as Validators take the upgrade to 1.10.9.
Posted May 17, 2022 - 03:41 UTC
Update
Lowered PoC Rewards and Receipts Update 5
The core team is seeing very slow improvements to Proof-of-Coverage (and rewards) receipts since the Validator release. This slow change is likely due to Validators going offline to upgrade themselves, get back online and sync to the tip of the blockchain. We’re hopeful that this is the case for the slow improvements (about 30% are not upgraded/online) but we are prepared to issue a Light Hotspot rollback tomorrow morning Tuesday, May 17, 2022 6:00 AM PT
We have identified another bug (see Discord #blockchain ) where the Proof-of-Coverage manager on Validators are crashing out every 50 blocks and purges the challenges. This is being reviewed and will be part of a new @Validator 1.10.9 release. This is not a silver-bullet bug fix, but rather looks to stabilize the receipts we already have from its current pattern so we can have a better view of the Proof-of-Coverage landscape tomorrow morning.
We will roll back the Light Hotspot transition at Tuesday, May 17, 2022 6:00 AM PT if: 1. More than 90% of Validators upgraded (at least 1.10.8) AND 2. Proof-of-Coverage receipts per block is still consistently below 100 receipts.
What will happen if we rollback: 1. Hotspots will challenge again. All Hotspots are still syncing the blockchain and there are consistent snapshots being produced to help those that are out of sync. No action required by Hotspot owners or Makers 2. Validators will continue performing Consensus Group duties and will no longer create Challenges. 3. We will continue our investigation on Testnet and will re-attempt activation at another time.
Posted May 16, 2022 - 23:14 UTC
Update
Lowered PoC Rewards and Receipts Update 4
About 66% of Validators have upgraded to the latest release.
Once a Validator is updated, it will take 200 blocks (or approximately 3.5 hours in 🕐 time) for them to discard invalid proof-of-coverage challenges and start issuing the updated versions that will result in the graph going 📈 .
TLDR: need more time for rewards to come back and stabilize as Validators upgrade and shed their old challenges. So if your Hotspot is connected to a Validator that just upgraded, it will be 3.5 hours before your Hotspot will be rewarded properly. The graph will not hockey stick.
No action required for Hotspot owners.
Posted May 16, 2022 - 21:17 UTC
Update
Lowered PoC Rewards and Receipts Update 3
The Validator Upgrade has been available for ~1 hour and 269 Validators have upgraded. We need to see much more Validators update to 1.10.8 before we can see noticeable improvements to Proof-of-Coverage and rewards. The team is continuing to monitor dashboard.helium.com.
No action is required for Hotspot owners.
Posted May 16, 2022 - 21:17 UTC
Identified
Lowered PoC Rewards and Receipts Update 2
The core team has tagged a release for Validator Operators that includes the bug fix for the known Proof-of-Coverage issue.
After the Validator update goes out, all Validators will be required to take this update ASAP. We will slowly start to see improvements (over the course of 2-3 hours). We are keeping the Challenges per block at 700 for now.
No action required for Hotspot owners.
Posted May 16, 2022 - 17:44 UTC
Investigating
The core team is continuing to debug the root cause of lowered Proof-of-Coverage receipts (and consequently lowered rewards). We may have a lead and will update community as soon as it's confirmed and we have a fix.
Also note that if you have altered your sys.config file on your Hotspot, you may have inadvertently broke your Hotspot's ability to participate in Proof-of-Coverage. Please check with your Hotspot Maker on what the correct configuration settings should be.
If you have no idea what sys.config means, no action required - the likely cause of low rewards is what the devs are working on.