All Systems Operational
Consensus Groups Operational
90 days ago
99.98 % uptime
Today
Proof-of-Coverage Operational
90 days ago
100.0 % uptime
Today
Router Operational
90 days ago
100.0 % uptime
Today
Onboarding Service Operational
90 days ago
100.0 % uptime
Today
Explorer Operational
90 days ago
99.99 % uptime
Today
Helium App Operational
90 days ago
99.99 % uptime
Today
User Apps Operational
90 days ago
99.75 % uptime
Today
Helium Wallet App ? Operational
90 days ago
99.99 % uptime
Today
Blockchain Operational
90 days ago
99.28 % uptime
Today
API Operational
90 days ago
99.99 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Past Incidents
Aug 12, 2022

No incidents reported today.

Aug 11, 2022

No incidents reported.

Aug 10, 2022

No incidents reported.

Aug 9, 2022
Resolved - Final update for the evening. The chain seems to be progressing. As more validators update and reload from a snapshot, we should continue to see them come back online. Core devs are moving to continuous monitoring of the Consensus Group and block production.

Reminder, please see announcements in Discord channels if you operate a chain following node. Hotspot owners do not need to make any adjustments and PoC activity should return to the state it was prior to this halt after several hours.

A postmortem report will be prepared and shared as soon as we can prepare it later this week.

Aug 9, 08:02 UTC
Monitoring - Update: Blocks are moving again but the number of transactions in each block is low (about 10% of expected txns). We still have not seen a reward block, so we're continuing to monitor.

Updates for ETL and Node are currently being tested. Validators have their update already.

Aug 9, 05:09 UTC
Update - Update: We have an emergency release going out to Validators (and another release for ETL, Node, and Router operators shortly).

The emergency release includes a new snapshot that all Validators will sync to to realign ledgers (there was a ledger drift).

Update again soon.

Aug 9, 03:04 UTC
Update - Update: Just setting some expectations here. We expect this to be a long chain halt (hours +) and there will be several emergency releases going out to Validators, Node, and ETL operators. Announcement on when that's available will be made separately. Chain is halted, not dead.

Again, pending transactions will not go through, rewards are affected as the chain is not moving, but it's affecting everyone equally.

Aug 9, 00:37 UTC
Update - Update: We have halted again at 1,469,988 and getting more clarity at the root cause. We believe receipts in blocks cannot be absorbed. More details coming as soon as we have a plan.
Aug 8, 22:45 UTC
Identified - Update: The core team has asked all Validators currently in the Consensus Group to restart their Validators in order to resume block production.
Aug 8, 20:34 UTC
Update - Update: We're still looking, but have a potential solution percolating. In the mean time, I want to also mention that device packets are still sending on the network despite the chain halt due to the fault-tolerant system the core team has put in place.
Aug 8, 20:18 UTC
Update - Some nodes are able to progress to block 1469953 but not all. This issue may auto-resolve but the core team is still investigating.
Aug 8, 19:52 UTC
Investigating - The core team is investigating the chain halt that occurred at block height 1469951 as a result of activating the latest chain variable.

All funds are safe, but pending transactions will be delayed until we can resolve this halt. Rewards will be delayed until the chain can move forward.

Pending Transactions include:
- Add Hotspot
- Assert Location
- Transfer Hotspot
- Payment Transactions
- Burn Transactions

Aug 8, 19:34 UTC
Aug 8, 2022
Aug 7, 2022

No incidents reported.

Aug 6, 2022

No incidents reported.

Aug 5, 2022

No incidents reported.

Aug 4, 2022

No incidents reported.

Aug 3, 2022

No incidents reported.

Aug 2, 2022

No incidents reported.

Aug 1, 2022

No incidents reported.

Jul 31, 2022

No incidents reported.

Jul 30, 2022

No incidents reported.

Jul 29, 2022

No incidents reported.