Packet Router Rollout

Incident Report for Helium

Resolved

Based on the monitoring, all packet routers have fully stabilized.
Posted May 05, 2023 - 05:42 UTC

Update

The core team has been monitoring the rollback for almost 1hr and packet routers have stabilized. We will monitor them for 1 more hour before resolving the incident.
Posted May 05, 2023 - 04:33 UTC

Update

We are continuing to monitor for any further issues.
Posted May 05, 2023 - 04:02 UTC

Update

Based on the monitoring, the core team has begun rollback on the packet routers due to high memory usage.
Posted May 05, 2023 - 04:01 UTC

Monitoring

The core team is rolling out packet routers across all regions to optimize GRPC process leaks. Data transfer may be temporarily unstable during the rollout. Should stabilize within 30 minutes.
Posted May 05, 2023 - 02:03 UTC
This incident affected: Helium Packet Router.