Get email notifications whenever Helium creates, updates or resolves an incident.
Get incident updates and maintenance status messages in Slack.
Subscribe via Slack
By subscribing you acknowledge our Privacy Policy. In addition, you agree to the Atlassian Cloud Terms of Service and acknowledge Atlassian's Privacy Policy.
Visit our support site.
All Systems Operational
Uptime over the past 90 days. View historical uptime.
Consensus Groups
Operational
90 days ago
99.07
% uptime
Today
Proof-of-Coverage
Operational
90 days ago
99.46
% uptime
Today
Router
Operational
90 days ago
99.89
% uptime
Today
Onboarding Service
Operational
90 days ago
99.46
% uptime
Today
Explorer
Operational
90 days ago
99.93
% uptime
Today
Helium App
Operational
90 days ago
100.0
% uptime
Today
User Apps
Operational
90 days ago
99.54
% uptime
Today
Helium Wallet App
?
Operational
90 days ago
100.0
% uptime
Today
Blockchain
Operational
90 days ago
98.94
% uptime
Today
API
Operational
90 days ago
99.69
% uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Related
No incidents or maintenance related to this downtime.
Past Incidents
Nov 27, 2022
No incidents reported today.
Nov 26, 2022
No incidents reported.
Nov 25, 2022
No incidents reported.
Nov 24, 2022
No incidents reported.
Nov 23, 2022
No incidents reported.
Nov 22, 2022
No incidents reported.
Nov 21, 2022
No incidents reported.
Nov 20, 2022
No incidents reported.
Nov 19, 2022
Resolved -
We have closed this incident and transactions are being processed by the Public API normally.
Nov 19, 02:40 UTC
Nov 19, 02:40 UTC
Update -
An ETL release (1.1.165) has been prepared by the core developers. Please upgrade your ETL in order to process block 1621472 and resume operations.
https://github.com/helium/blockchain-etl/releases/tag/1.1.165
We will issue an all clear when the public API has resumed processing blocks and pending transactions can be resumed.
Nov 18, 23:29 UTC
https://github.com/helium/blockchain-etl/releases/tag/1.1.165
We will issue an all clear when the public API has resumed processing blocks and pending transactions can be resumed.
Nov 18, 23:29 UTC
Identified -
The public Helium API is currently not showing block and transaction data after block 1621471 due to a missing ETL patch for a new transaction type. The chain is still performing other activity normally and is not affected.
Funds are safe and activity like Proof of Coverage and Data Transfers continue normally. The API, however, is behind the chain so apps that onboard and move Hotspots, transfer HNT and MOBILE through the API, etc. will be degraded until the API is able to process blocks again. Please do not resubmit transactions until patches are applied on public ETL and API instances. More information on these patches are being shared on Discord and will be published shortly.
Nov 18, 23:11 UTC
Funds are safe and activity like Proof of Coverage and Data Transfers continue normally. The API, however, is behind the chain so apps that onboard and move Hotspots, transfer HNT and MOBILE through the API, etc. will be degraded until the API is able to process blocks again. Please do not resubmit transactions until patches are applied on public ETL and API instances. More information on these patches are being shared on Discord and will be published shortly.
Nov 18, 23:11 UTC
Nov 18, 2022
Nov 17, 2022
Resolved -
The core team has resolved the issue with the help of the validator community.
We appreciate your support and patience as the team worked to get the chain back on track.
Nov 17, 20:27 UTC
We appreciate your support and patience as the team worked to get the chain back on track.
Nov 17, 20:27 UTC
Monitoring -
Update: good news, the validator operator community has answered the call and block production has resumed.
We'll continue to closely monitor status. Thanks again for your patience.
Nov 17, 15:57 UTC
We'll continue to closely monitor status. Thanks again for your patience.
Nov 17, 15:57 UTC
Update -
Update: Unfortunately, the chain remains halted because block production is still prevented by the corrupted cache issue.
The decentralized nature of the network requires the core team to work with the validator operator community to make the necessary changes to resolve this issue.
The core team has repeatedly communicated to operators to fix their cache so normal block production can resume.
🙏🏼 Thank you to those in the validator community that have timely responded to the call and made the updates, however, we’re not quite there yet.
Once again we're sorry about this halt and appreciate your patience while we work to resolve.
Nov 17, 14:39 UTC
The decentralized nature of the network requires the core team to work with the validator operator community to make the necessary changes to resolve this issue.
The core team has repeatedly communicated to operators to fix their cache so normal block production can resume.
🙏🏼 Thank you to those in the validator community that have timely responded to the call and made the updates, however, we’re not quite there yet.
Once again we're sorry about this halt and appreciate your patience while we work to resolve.
Nov 17, 14:39 UTC
Update -
Update: The core team is continuing to work with validator community to resolve the corrupted region cache issue which is preventing block production.
All funds are safe, but pending transactions including rewards will be delayed until the chain can move forward again.
Device packets are still sending on the network despite the chain halt due to the fault-tolerant system the core team has put in place.
Please refrain from submitting duplicate transactions as they will all clear when the chain moves once again.
Nov 17, 13:28 UTC
All funds are safe, but pending transactions including rewards will be delayed until the chain can move forward again.
Device packets are still sending on the network despite the chain halt due to the fault-tolerant system the core team has put in place.
Please refrain from submitting duplicate transactions as they will all clear when the chain moves once again.
Nov 17, 13:28 UTC
Identified -
Validators with a corrupted region cache are preventing block production. The core team is working with operators to fix affected validators to resume block production. Sorry for the inconvenience and thanks for your patience.
At this time, all transactions on chain will be unconfirmed until Validators begin producing blocks again.
Funds are safe but activity like Proof of Coverage, Hotspot onboarding and movement, Token Transfers, etc. are unavailable.
Data Transfer is not affected.
Nov 17, 12:36 UTC
At this time, all transactions on chain will be unconfirmed until Validators begin producing blocks again.
Funds are safe but activity like Proof of Coverage, Hotspot onboarding and movement, Token Transfers, etc. are unavailable.
Data Transfer is not affected.
Nov 17, 12:36 UTC
Resolved -
Explorer and the Helium Apps are now back. Thanks for everyone's patience during this extended maintenance period.
Nov 17, 20:25 UTC
Nov 17, 20:25 UTC
Update -
Explorer and Apps are syncing back up but still about 200 blocks behind.
Nov 17, 19:10 UTC
Nov 17, 19:10 UTC
Update -
GM. The API is still down. During the maintenance, a corrupt database affected how all services could be brought back up. As a result, over 170TB of data had to be rebuilt from scratch.
The main database is up but in order to service all our users and services, we need several more instances.
Nov 17, 15:23 UTC
The main database is up but in order to service all our users and services, we need several more instances.
Nov 17, 15:23 UTC
Identified -
The API is still down at this time. Databases are being brought up to restore services to the Apps, Explorer, and allow pending transactions first. The Public API will return after the other services are back up. We expect that this will be back online in the morning Pacific Time. An update will be provided then if anything changes.
Nov 17, 07:06 UTC
Nov 17, 07:06 UTC
Update -
The API is still down at this time. Databases are being brought up to restore services to the Apps, Explorer, and allow pending transactions first. The Public API will return after the other services are back up.
Please also note that the chain is currently halted, and the Validator operators are working to get the chain moving again.
Nov 17, 04:15 UTC
Please also note that the chain is currently halted, and the Validator operators are working to get the chain moving again.
Nov 17, 04:15 UTC
Update -
Quick update on the API:
Services are slowly being brought back up. This will take time as there is a lot of data to move around. That said, our API provider expects all Helium Apps to be back up in the next 4-5 hours and pending transactions re-enabled in 4-5 hours as well.
The public API will not return for about 8 more hours.
Nov 16, 18:07 UTC
Services are slowly being brought back up. This will take time as there is a lot of data to move around. That said, our API provider expects all Helium Apps to be back up in the next 4-5 hours and pending transactions re-enabled in 4-5 hours as well.
The public API will not return for about 8 more hours.
Nov 16, 18:07 UTC
Investigating -
The Helium API Service is still offline due to unforeseen issues during the maintenance window. The team is working through a resolution and will update time of resolution as soon as possible.
During this time:
▫️Submitting transactions will be unavailable in the Helium Hotspot App and Helium Wallet App. (Transactions include Add Hotspot, Assert Location, Payments, Burns, and Transfer Hotspots)
▫️ Information displayed in the Hotspot App, Wallet App, and Explorer will be unavailable, out-of-date, or delayed.
▫️ Mining, Rewards, and data transfer are not affected, but their transactions will be delayed until the maintenance is over.
While the main Helium API is still down, you might be able to do what you’re trying to do using community data sources such as:
https://hotspotty.net/
https://hntscan.io/
https://www.heliumboard.com/
https://heliumtracker.io/
https://heliumgeek.com/
Give some of these tools developed by the Helium community a spin!
Nov 16, 16:37 UTC
During this time:
▫️Submitting transactions will be unavailable in the Helium Hotspot App and Helium Wallet App. (Transactions include Add Hotspot, Assert Location, Payments, Burns, and Transfer Hotspots)
▫️ Information displayed in the Hotspot App, Wallet App, and Explorer will be unavailable, out-of-date, or delayed.
▫️ Mining, Rewards, and data transfer are not affected, but their transactions will be delayed until the maintenance is over.
While the main Helium API is still down, you might be able to do what you’re trying to do using community data sources such as:
https://hotspotty.net/
https://hntscan.io/
https://www.heliumboard.com/
https://heliumtracker.io/
https://heliumgeek.com/
Give some of these tools developed by the Helium community a spin!
Nov 16, 16:37 UTC
Resolved -
The chain has successfully elected a new group and we have resolved this incident.
Nov 17, 06:59 UTC
Nov 17, 06:59 UTC
Monitoring -
A rescue block has been generated and issued by the core developers and the Helium Foundation in order to restart block production. A new group has been selected.
Community validator operators continue to monitor the chain and block production. We will issue an all clear once we feel confident that the chain is stable.
Nov 17, 06:24 UTC
Community validator operators continue to monitor the chain and block production. We will issue an all clear once we feel confident that the chain is stable.
Nov 17, 06:24 UTC
Update -
The Validator Operators are continuing to work together to get agreement on the next block. The chain is still halted at this time. If you are a Validator Operator, please join us in #validator-ops and restart your node.
Nov 17, 04:15 UTC
Nov 17, 04:15 UTC
Identified -
Quick update on this:
We've requested all Validator Operators to restart their nodes to kickstart the consensus group so we can start producing blocks again.
Nov 17, 00:32 UTC
We've requested all Validator Operators to restart their nodes to kickstart the consensus group so we can start producing blocks again.
Nov 17, 00:32 UTC
Investigating -
The blockchain is currently stalled at block height 1619546 after activating the regional chain variables. The core team is working through the issue with the Validator Operators. We will provide an update as soon as we know more.
Funds are safe.
Nov 16, 23:06 UTC
Funds are safe.
Nov 16, 23:06 UTC
Nov 16, 2022
Completed -
The scheduled maintenance has been completed.
Nov 16, 08:00 UTC
Nov 16, 08:00 UTC
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 16, 05:00 UTC
Nov 16, 05:00 UTC
Scheduled -
Scheduled Maintenance Alert from BlockJoy
We want to make you aware that this Wednesday (11/16) at 00:00 EDT, there will be a scheduled downtime for approximately 3 hours. We will be using this time to add more capacity to our master etl database serving the Helium API Service.
During this maintenance window,
▫️The Helium API Service will be offline.
▫️Submitting transactions will be unavailable in the Helium Hotspot App and Helium Wallet App. (Transactions include Add Hotspot, Assert Location, Payments, Burns, and Transfer Hotspots)
▫️ Information displayed in the Hotspot App, Wallet App, and Explorer will be unavailable.
All operations around the API service including submitting transactions will resume immediately after the maintenance window. We appreciate your patience and understanding.
Nov 15, 17:01 UTC
We want to make you aware that this Wednesday (11/16) at 00:00 EDT, there will be a scheduled downtime for approximately 3 hours. We will be using this time to add more capacity to our master etl database serving the Helium API Service.
During this maintenance window,
▫️The Helium API Service will be offline.
▫️Submitting transactions will be unavailable in the Helium Hotspot App and Helium Wallet App. (Transactions include Add Hotspot, Assert Location, Payments, Burns, and Transfer Hotspots)
▫️ Information displayed in the Hotspot App, Wallet App, and Explorer will be unavailable.
All operations around the API service including submitting transactions will resume immediately after the maintenance window. We appreciate your patience and understanding.
Nov 15, 17:01 UTC
Nov 15, 2022
No incidents reported.
Nov 14, 2022
No incidents reported.
Nov 13, 2022
No incidents reported.