tag:status.helium.com,2005:/historyHelium Status - Incident History2024-03-29T05:23:35ZHeliumtag:status.helium.com,2005:Incident/203141132024-03-21T16:51:11Z2024-03-21T16:51:11ZHelium Wallet API - Scheduled Maintenance<p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>16:51</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>16:11</var> UTC</small><br><strong>Verifying</strong> - The update has been completed and we are currently verifying the status of services.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>14:55</var> UTC</small><br><strong>Update</strong> - The update is progressing without issue but is taking longer than expected. We will provide further updates as necessary.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>21:33</var> UTC</small><br><strong>Scheduled</strong> - The Helium Wallet API will undergo an upgrade sometime during this hour period. <br /><br />The upgrade should be brief but may incur downtime for wallet operations such as viewing hotspots and claiming rewards.</p>tag:status.helium.com,2005:Incident/199404672024-02-08T20:30:56Z2024-02-08T20:30:56ZOnboarding Service - Scheduled Maintenance<p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>20:30</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>20:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>15:59</var> UTC</small><br><strong>Scheduled</strong> - The Onboarding Service will undergo an upgrade to switch Mobile onboarding to the Pyth oracle. We anticipate a short amount of downtime while the upgrade occurs. Please be advised that during the downtime all onboard transactions will fail.</p>tag:status.helium.com,2005:Incident/199207432024-02-06T15:26:59Z2024-02-06T15:26:59ZSolana Halt<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>15:26</var> UTC</small><br><strong>Resolved</strong> - We join the Solana community in thanking validator operators and core developers for their firefighting over the past few hours.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>14:54</var> UTC</small><br><strong>Monitoring</strong> - Block production has resumed on Solana. We will continue to monitor over the next few hours.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>12:12</var> UTC</small><br><strong>Update</strong> - Solana validators have begun a coordinated restart with an updated release version. You can follow status here: https://stakewiz.com/restart</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>11:37</var> UTC</small><br><strong>Update</strong> - Engineers from across the Solana ecosystem are readying a new validator software release, which includes a patch to address an issue which caused the cluster to halt.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>09:53</var> UTC</small><br><strong>Identified</strong> - As noted on status.solana.com, the Solana network is currently experiencing a chain halt which may affect some activity for Helium network users. Notably, deployers will not be able to onboard or move Hotspots and token holders may not be able to perform transfers or participate in other on-chain activity. <br /><br />Rest assured that Solana validators and core developers are working on resuming block production and that Helium oracle activity is not affected.</p>tag:status.helium.com,2005:Incident/196502182024-01-10T17:35:13Z2024-01-10T17:35:14ZHelium Wallet API - Scheduled Maintenance<p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>17:35</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>17:25</var> UTC</small><br><strong>Verifying</strong> - The maintenance has been completed. We are now verifying stability.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>17:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>21:07</var> UTC</small><br><strong>Scheduled</strong> - The Helium Wallet API will undergo an upgrade sometime during this hour period. The upgrade should be brief and we do not expect any downtime.</p>tag:status.helium.com,2005:Incident/192520272023-11-30T19:15:17Z2023-11-30T19:15:18ZIoT Config Service Maintenance<p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>19:15</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>19:09</var> UTC</small><br><strong>Verifying</strong> - The deployment has been completed as planned. We are now verifying to ensure system stability.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>19:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>18:24</var> UTC</small><br><strong>Scheduled</strong> - IoT Config service will experience a brief (<= 2min) of downtime to deploy the capability to authorize HPRs to request gateway hex location</p>tag:status.helium.com,2005:Incident/191189932023-11-14T15:50:34Z2023-11-14T15:58:52ZIncomplete Closure of Epoch 19,674<p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>15:50</var> UTC</small><br><strong>Resolved</strong> - The automated job that was supposed to end the November 13th epoch (19,674) at UTC midnight stalled due to a dropped transaction. The job continued to wait for confirmation on a transaction that was never picked up by Solana. The epoch was manually ended by Core Developers at 15:30 UTC. No further action is needed. The team will be improving alerting to ensure we are notified earlier when epochs fail to close. Additionally, they will be repairing the automated job to include a timeout on transaction confirmations.</p>tag:status.helium.com,2005:Incident/189716862023-10-30T22:00:26Z2023-10-30T22:00:26ZHelium Wallet - Cloudflare Outage/RPC Degraded Performance<p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>22:00</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>20:33</var> UTC</small><br><strong>Monitoring</strong> - A workaround has been provided by the RPC provider and Helium Wallet Application functionality has been restored. We will continue monitoring to ensure stability moving forward.</p><p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>20:12</var> UTC</small><br><strong>Identified</strong> - We have identified a major outage with Cloudflare causing downstream impact to the Solana RPC backing the Helium Wallet App. Wallet App operations are degraded at this time.</p>tag:status.helium.com,2005:Incident/184832582023-10-02T13:57:12Z2023-10-02T13:57:12ZFoundation SDLC Accounts - Scheduled Maintenance<p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>13:57</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>18:29</var> UTC</small><br><strong>Verifying</strong> - The scheduled maintenance has been completed. We are now verifying availability of services</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>13:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>19:58</var> UTC</small><br><strong>Scheduled</strong> - Foundation SDLC accounts will undergo maintenance to update networking configuration. During the maintenance window, Solana devnet access via Foundation infrastructure will be unavailable. Accordingly, Solana devnet interactions via applications such as the Onboarding Service, Helium Wallet App, and Helium Wallet App API will be limited.</p>tag:status.helium.com,2005:Incident/181649162023-08-16T19:16:27Z2023-08-16T19:16:27ZPacket Router Rollout In Progress<p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>19:16</var> UTC</small><br><strong>Resolved</strong> - Based on the monitoring, all packet routers have fully stabilized.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>Update</strong> - The rollout is complete. The core team will monitor for 1 hour before the final status update.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>17:40</var> UTC</small><br><strong>Monitoring</strong> - The core team is rolling out a new packet router release <br />• HPR utility for more debugging visibility on a route level <br />• additional grpc stream management<br /><br />The expected downtime is 15-20 minutes.</p>tag:status.helium.com,2005:Incident/179603482023-07-26T06:12:04Z2023-07-26T06:12:04ZPacket Router Outage<p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>06:12</var> UTC</small><br><strong>Resolved</strong> - The Packet Routers have been stable for the 2.5 hours. Marking this incident resolved and will follow up with post mortem.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>03:39</var> UTC</small><br><strong>Identified</strong> - Starting 8:18PM PT, the U.S. Packet Routers are offline again. The core team hasn't identified the root cause yet as all server metrics were healthy up til the crash, we are going to reboot and remove a load optimization constraint.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>01:46</var> UTC</small><br><strong>Monitoring</strong> - The Packet Routers have been stable for the past hour. The core team will keep monitoring the incident.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>00:35</var> UTC</small><br><strong>Update</strong> - The U.S. Packet Routers have been rebooted. In the meantime, the core team is investigating the outage and will continue the update here.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>00:31</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>00:31</var> UTC</small><br><strong>Update</strong> - Starting 5:19PM PT, the U.S. Packet Routers are offline. The core team is investigating the outage and will continue the update here.</p><p><small>Jul <var data-var='date'>25</var>, <var data-var='time'>23:52</var> UTC</small><br><strong>Monitoring</strong> - The Helium Packet Routers started experiencing intermittent outage from 2:38PM PT due to misconfigured open file limit, the issue was resolved on 3:43 PT. <br /><br />The team is monitoring the incident as there still seem to be some loose ends. Will continue the update here.</p>tag:status.helium.com,2005:Incident/179163982023-07-20T22:50:40Z2023-07-20T22:50:40ZSingapore Packet Router Load Testing In Progress<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>22:50</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>22:01</var> UTC</small><br><strong>Monitoring</strong> - The core team is performing load testing on the Singapore packet routers. We will be closely monitoring the infra metrics and data transfer health.</p>tag:status.helium.com,2005:Incident/178795212023-07-20T19:37:56Z2023-07-20T19:37:56ZHelium Wallet API - Scheduled Maintenance<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>19:37</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>19:26</var> UTC</small><br><strong>Verifying</strong> - The update has been completed and we are currently verifying the status of services.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>17:46</var> UTC</small><br><strong>Update</strong> - The update is progressing without issue but is taking longer than expected. We will provide further updates as necessary.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>17:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>15:51</var> UTC</small><br><strong>Scheduled</strong> - The Helium Wallet API will undergo an upgrade sometime during this hour period. The upgrade should be brief. If there is a problem the service will be rolled back to its previous state.</p>tag:status.helium.com,2005:Incident/177751752023-07-06T18:54:39Z2023-07-06T18:54:40ZBeaconing activity decreased<p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>18:54</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>17:56</var> UTC</small><br><strong>Monitoring</strong> - We believe we've identified and fixed the issue with the Configuration Service and will be monitoring it. Hotspots that have stopped Beaconing due to the outage should resume Beaconing within a couple of hours.</p><p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>16:40</var> UTC</small><br><strong>Update</strong> - After continued monitoring we've determined that the Configuration Service, which tells Hotspots which LoRaWAN region they operate in, has a database issue and is not serving locations properly to some Hotspots. This will prevent these Hotspots from Beaconing, and in some cases, transferring data. We are investigating reloading the Configuration Service's data.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>23:04</var> UTC</small><br><strong>Investigating</strong> - The core developers have noticed a drop off in Beaconing activity and believe that the cause is an earlier problem with the Configuration Service. The developers believe that Hotspots should resume their normal Beaconing some 4 hours after this notice and will continue to monitor the situation for changes.</p>tag:status.helium.com,2005:Incident/177570272023-07-04T05:11:45Z2023-07-04T05:11:47ZHelium Wallet - RPC Degraded Performance<p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>05:11</var> UTC</small><br><strong>Resolved</strong> - This issue is now resolved. Realms and the Helium Wallet are back to normal operations.</p><p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>02:34</var> UTC</small><br><strong>Update</strong> - In furtherance to the Solana RPC degraded performance, we have identified additional impacts affecting Realms and are continuing to coordinate with the third party provider on resolution.</p><p><small>Jul <var data-var='date'> 4</var>, <var data-var='time'>01:07</var> UTC</small><br><strong>Identified</strong> - We have identified an issue with the Solana RPC and are coordinating with the third party provider on resolution. Helium Wallet App performance may be degraded at this time.</p>tag:status.helium.com,2005:Incident/177019472023-06-28T19:00:11Z2023-06-28T19:00:12ZOracle upgrades<p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>19:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>19:01</var> UTC</small><br><strong>Scheduled</strong> - Several IOT and MOBILE service components will be upgraded to improve performance and lay the groundwork for new features. These upgrades will be rolled out individually over time. No interruptions to the network are expected. (Oracles being upgraded: iot_verifier, iot_index, mobile_config, mobile_packet_verifier, and mobile_verifier).</p>tag:status.helium.com,2005:Incident/176890712023-06-27T00:15:37Z2023-06-27T15:04:55ZIssues with Claim Hotspot Rewards in Helium Wallet App<p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>00:15</var> UTC</small><br><strong>Resolved</strong> - Wallet developers have confirmed that this particular indexing issue was resolved overnight.</p><p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>18:14</var> UTC</small><br><strong>Update</strong> - Wallet developers continue to monitor as we've heard some reports of improvements in the Claim Hotspot Rewards feature of the Helium Wallet App. As a reminder, your rewards can be safely claimed once the provider has fully returned.</p><p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>15:04</var> UTC</small><br><strong>Identified</strong> - Wallet developers have identified community reported issues with the Claim Hotspot Rewards feature of the Helium Wallet App. It is temporarily down for a large number of Hotspots due to an upstream RPC provider experiencing issues with indexing. Your rewards can be safely claimed once the provider is back online.</p>tag:status.helium.com,2005:Incident/176899752023-06-26T20:47:53Z2023-06-26T20:47:54ZDegraded Data Transfer in U.S.<p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>20:47</var> UTC</small><br><strong>Resolved</strong> - Based on the monitoring, all packet routers have fully stabilized.</p><p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>19:34</var> UTC</small><br><strong>Update</strong> - The core team has also upgraded the U.S. packet routers, we will continue monitoring the performance.</p><p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>18:06</var> UTC</small><br><strong>Identified</strong> - The core team is going to patch the packet routers in U.S. with disk space optimization, and we will monitor the performance.</p><p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>17:11</var> UTC</small><br><strong>Update</strong> - The issue was caused by the packet routers in U.S. running out of space due to multi-buy service logs. The core team is on it and we will provides updates here.</p><p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>16:37</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>16:37</var> UTC</small><br><strong>Investigating</strong> - Due to a high amount of packet rate, the data transfer in U.S. may be experiencing degraded service. The core team is on it and we will provides updates here.</p>tag:status.helium.com,2005:Incident/175623142023-06-20T18:05:32Z2023-06-20T18:05:33ZOnboarding Service Database Upgrade<p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>18:05</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>18:25</var> UTC</small><br><strong>Scheduled</strong> - The database for the Onboarding Service will undergo an automated upgrade. During the upgrade it may not be possible to onboard new Hotspots to the network. The upgrade should complete within a couple of minutes and service should restore to normal.</p>tag:status.helium.com,2005:Incident/175235442023-06-09T17:34:02Z2023-06-09T17:34:03ZHelium Wallet / RPC Degraded Performance<p><small>Jun <var data-var='date'> 9</var>, <var data-var='time'>17:34</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 9</var>, <var data-var='time'>16:36</var> UTC</small><br><strong>Monitoring</strong> - The fix appears to have addressed the issue but we will continue to monitor to ensure stability is maintained.</p><p><small>Jun <var data-var='date'> 9</var>, <var data-var='time'>15:43</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a potential fix has been deployed.</p><p><small>Jun <var data-var='date'> 9</var>, <var data-var='time'>15:05</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue with the RPC with the third party provider. Explorer and Helium Wallet App performance may be degraded at this time.</p>tag:status.helium.com,2005:Incident/174388062023-06-01T06:33:35Z2023-06-01T06:33:35ZBeaconing and Witnessing in NZ has stopped.<p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>06:33</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>06:12</var> UTC</small><br><strong>Monitoring</strong> - Restarting the IOT PoC Verifier seems to have resolved the issue. The core developers will continue to monitor PoC in New Zealand for further issues.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>05:59</var> UTC</small><br><strong>Update</strong> - The IOT PoC Verifier has been reset and is back up and running. The core developers are now monitoring the verifier's output reports to see if the number of invalidation events is decreasing.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>05:47</var> UTC</small><br><strong>Update</strong> - The core developers are restarting the IOT PoC Verifier service in an attempt to rebuild its region parameter cache. This restart will be noticeable in activity dashboards for some tens of minutes as the service restarts, but it will not ultimately affect rewards as the service will catch up to queued beacon and witness events when it comes back up.</p><p><small>Jun <var data-var='date'> 1</var>, <var data-var='time'>04:49</var> UTC</small><br><strong>Investigating</strong> - The core developers have noticed a rapid drop off of Beaconing and Witnessing events for the New Zealand after today's deployment of the new AS923-1C LoRaWAN region parameters to increase PoC power.</p>tag:status.helium.com,2005:Incident/174348952023-05-31T20:55:56Z2023-05-31T20:55:57ZPacket Router Rollout In Progress<p><small>May <var data-var='date'>31</var>, <var data-var='time'>20:55</var> UTC</small><br><strong>Resolved</strong> - Based on the monitoring, all packet routers have fully stabilized.</p><p><small>May <var data-var='date'>31</var>, <var data-var='time'>19:35</var> UTC</small><br><strong>Update</strong> - The rollout is complete, the core team will closely monitor the packet routers.</p><p><small>May <var data-var='date'>31</var>, <var data-var='time'>19:00</var> UTC</small><br><strong>Monitoring</strong> - In response to the previous degraded data transfer incidents, the core team is doing a rollout for memory optimizations on packet routers, the expected downtime is 15-20 minutes.</p>tag:status.helium.com,2005:Incident/173427472023-05-28T18:38:22Z2023-05-28T18:38:24ZDegraded Data Transfer<p><small>May <var data-var='date'>28</var>, <var data-var='time'>18:38</var> UTC</small><br><strong>Resolved</strong> - All packet routers have stabilized including those in Asia.<br /><br />Thanx.</p><p><small>May <var data-var='date'>24</var>, <var data-var='time'>00:01</var> UTC</small><br><strong>Monitoring</strong> - All packet routers have stabilized except for those in Asia. We will be monitoring the situation and update once they are stabilized as well.<br /><br />Thanx.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>22:03</var> UTC</small><br><strong>Update</strong> - The rollout is complete. We will make another announcement here when the packet routers fully stabilize.<br /><br />Thanx.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>21:16</var> UTC</small><br><strong>Update</strong> - The hot fix has performed well. The core team is beginning a rollout for packet routers, which should fully stabilize in 1 hr.<br /><br />Thanx.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>18:03</var> UTC</small><br><strong>Update</strong> - The core has deployed a hot fix on the packet routers, and our testing devices are now able to join successfully. We will be monitoring the situation along with the community.<br /><br />Thanx.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>17:03</var> UTC</small><br><strong>Update</strong> - Based on preliminary investigations, the issue is related with memory usage on packet routers. The core team is working on a patch and we will provides updates here.<br /><br />Thanx.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>16:26</var> UTC</small><br><strong>Investigating</strong> - Due to a high amount of packet rate, the data transfer in U.S. and Asia may be experiencing degraded service (e.g. join loop), which likely caused the issue above. The core team is investigating and we will provides updates here.<br /><br />Thanx.</p>tag:status.helium.com,2005:Incident/173931582023-05-27T23:19:46Z2023-05-27T23:19:47ZHelium Wallet / RPC Outage<p><small>May <var data-var='date'>27</var>, <var data-var='time'>23:19</var> UTC</small><br><strong>Resolved</strong> - The core developers have verified with the RPC provider that a bad configuration was inadvertently deployed. This configuration has been fixed and the Helium community should no longer see this particular issue.</p><p><small>May <var data-var='date'>27</var>, <var data-var='time'>23:13</var> UTC</small><br><strong>Monitoring</strong> - We are seeing normal activity on Explorer and the Wallet app. Devs will continue to monitor the situation and close the issue after confirming recovery.</p><p><small>May <var data-var='date'>27</var>, <var data-var='time'>22:57</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>May <var data-var='date'>27</var>, <var data-var='time'>22:00</var> UTC</small><br><strong>Identified</strong> - The community and core developers have noticed that apps like the Explorer and Helium Wallet app are experiencing a degraded performance due to downstream RPC provider errors. Core developers are working with the provider to ensure service is restored soon. <br /><br />In the meantime, the apps may show incorrect data and transactions may not submit at this time. This should resolve as soon as the RPC provider is back online.</p>tag:status.helium.com,2005:Incident/173037172023-05-19T16:23:57Z2023-05-19T16:23:58ZDegraded Data Transfer in Asia<p><small>May <var data-var='date'>19</var>, <var data-var='time'>16:23</var> UTC</small><br><strong>Resolved</strong> - The upsizing is complete and the packet routers have fully stabilized.</p><p><small>May <var data-var='date'>19</var>, <var data-var='time'>15:54</var> UTC</small><br><strong>Update</strong> - Memory issues were identified, the core team is upsizing the packet routers in Asia accordingly. The upsizing should complete within 30 minutes and data transfer in Asia may be temporarily unstable during then.<br /><br />Thanx.</p><p><small>May <var data-var='date'>19</var>, <var data-var='time'>15:51</var> UTC</small><br><strong>Identified</strong> - Due to a high amount of packet rate, the data transfer in Asia may be experiencing degraded service. The core team is on it and we will provides updates here.<br /><br />Thanx.</p>tag:status.helium.com,2005:Incident/172313542023-05-12T20:14:31Z2023-05-12T20:14:32ZIOT PoC Oracle Data Delay<p><small>May <var data-var='date'>12</var>, <var data-var='time'>20:14</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>19:54</var> UTC</small><br><strong>Monitoring</strong> - Oracle Data has completed processing its backlog and core developers will continue monitoring these services. Community applications may take some time to recover from the delay.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>19:08</var> UTC</small><br><strong>Identified</strong> - Core developers have identified an issue with an upstream data provider that is causing a delay in oracle data delivery (for the verified iot S3 buckets).<br /><br />Specifically, community applications that track near real-time IOT Proof-of-Coverage data may be out of date which may impact Hotspot owners being able to see recent PoC activity.<br /><br />All of the data is still being recorded and will resume once oracle services are restored.</p>