Delayed API Transactions and Lagging Explorer/App
Incident Report for Helium
Resolved
After an hour of steady ETL operations, we are considering this incident closed. Thanks again.
Posted Apr 21, 2022 - 05:59 UTC
Monitoring
The backlog of transactions has now been fully processed and we have re-enabled transaction submission. Please wait a few hours to double-check for transaction confirmations on Explorer before resubmitting any old transactions. New payments, Hotspot onboarding transactions, and other transactions submitted via the public API will now process normally.

Thank you for your patience over the last several hours. The API service team and the Nova team will continue to monitor the ETL over the next hour or so before considering this issue completely resolved.
Posted Apr 21, 2022 - 04:58 UTC
Identified
We have identified a fix that allows the ETL to more efficiently process blocks and transactions and are seeing progress. Submitting transactions via the public API are still disabled at this time but we will be able to turn them back on in a few hours when the ETL nodes are closer to matching validators and block production.
Posted Apr 20, 2022 - 23:22 UTC
Update
The investigation continues but we are noticing that both the block delay has increased to over 300 blocks and the number of pending transactions has increased to over 8000 in the backlog.

You may see transaction errors in the app or in a CLI / Javascript client if you are submitting new payments or onboarding transactions as we will be disabling transaction submissions for a period of time to try to address the backlog.

We'll be back with an update when we have more details and when we are ready to turn on transaction submission again.
Posted Apr 20, 2022 - 21:24 UTC
Update
The team continues to investigate how to enable the ETL server to catch up on blocks and keep up with the chain. As mentioned earlier, Validators are producing blocks normally and Hotspots are continuing to transfer data and perform PoC normally.

We hope to have another update with some progress soon but we're still seeing the ETL server behind by about 250 blocks
Posted Apr 20, 2022 - 19:49 UTC
Investigating
The Nova team is assisting the API service provider to investigate slow transaction processing on ETL and API nodes. Users may see blocks taking longer to appear in explorer, and transactions submitted via the API (payments in the app, onboarding transactions, etc.) taking longer to clear. We’re hoping this is just temporary “weather” and things are back to normal soon. Validators are producing blocks normally and Hotspots are continuing to transfer data and perform PoC normally.

At this time, the ETL server that powers the API (used by Explorer and App) is still about 200 blocks behind.

No action required at this time by the community but we appreciate your patience.
Posted Apr 20, 2022 - 18:47 UTC
This incident affected: User Apps (Helium Wallet App, API) and Onboarding Service, Explorer, Helium App.