Packet Router Rollout In Progress
Incident Report for Helium
Resolved
Based on the monitoring, all packet routers have fully stabilized and rollout is complete.
Posted May 09, 2023 - 18:34 UTC
Update
Based on the monitoring, the core team is rolling out the GRPC connection stability update to all packet routers.
Posted May 09, 2023 - 16:43 UTC
Update
The U.S. packet routers are back up. We are now evaluating the patch performance on GRPC connection stability in the selected few packet routers. After confirmation, we will rollout to the other packet routers.
Posted May 09, 2023 - 16:28 UTC
Monitoring
The core team is doing a staged rollout for packet routers to optimize GRPC process leaks. We’ll start with a rollout on a few packet routers, monitor for a bit before releasing to all packet routers. The rollout is doing fine in other regions, but causing memory issues in U.S. The core team is upgrading the packet router instances in U.S., the expected downtime is 15-20 minutes.
Posted May 09, 2023 - 16:11 UTC
This incident affected: Helium Packet Router.