The Packet Routers have been stable for the 2.5 hours. Marking this incident resolved and will follow up with post mortem.
Posted Jul 26, 2023 - 06:12 UTC
Identified
Starting 8:18PM PT, the U.S. Packet Routers are offline again. The core team hasn't identified the root cause yet as all server metrics were healthy up til the crash, we are going to reboot and remove a load optimization constraint.
Posted Jul 26, 2023 - 03:39 UTC
Monitoring
The Packet Routers have been stable for the past hour. The core team will keep monitoring the incident.
Posted Jul 26, 2023 - 01:46 UTC
Update
The U.S. Packet Routers have been rebooted. In the meantime, the core team is investigating the outage and will continue the update here.
Posted Jul 26, 2023 - 00:35 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Jul 26, 2023 - 00:31 UTC
Update
Starting 5:19PM PT, the U.S. Packet Routers are offline. The core team is investigating the outage and will continue the update here.
Posted Jul 26, 2023 - 00:31 UTC
Monitoring
The Helium Packet Routers started experiencing intermittent outage from 2:38PM PT due to misconfigured open file limit, the issue was resolved on 3:43 PT.
The team is monitoring the incident as there still seem to be some loose ends. Will continue the update here.