#August 8: Global Corrosion lag caused Machines API app-not-found errors (23:52UTC)

A subset of very large service-definition updates from machines with large numbers of services could not be applied to our global service-discovery database (Corrosion) within the configured timeout on some nodes. These updates were repeatedly retried, slowing down or blocking the application of other updates. Since the Machines API (via Flaps) reads app metadata from global Corrosion, some newly created apps and recent updates were not visible everywhere, resulting in elevated app-not-found errors.

We mitigated the issue by adjusting the timeout and Corrosion’s retry behavior so that failed updates do not retry indefinitely and block other updates. We also rolled out a fix to chunk updates when a machine has a large number of services, preventing such updates from causing similar propagation issues in the future.

#August 4: Certificate issuance delays from Sidekiq overload (14:45UTC)

A surge of certificate-check and renewal work overwhelmed our background job system, causing significant delays issuing and renewing TLS certificates. This was caused by a few reasons, but the main one being an internal DNS resolver dedicated to the certificate renewal path: it is configured with no caching and aggressively recurses to upstream whenever possible in order to minimize delays in observing DNS updates for customers’ domains. That, however, was initially configured at a time where we were operating at a much smaller scale than we are today, and on this specific day, a spike of DNS renewal tasks combined with possibly slow lookups for some domains completely paralyzed the TLS certificate issuance queue. In the process of investigating this issue, we also discovered that the TLS certificate renewal task has been trying to renew certificates for deleted apps (with no success, of course), which did not help with the excessive queuing delays seen on this day.

We have put in several mitigations for this. Firstly, the DNS resolver used for TLS certificate renewal is now allowed to do some minimal caching to act as a buffer against temporary spikes in TLS renewal tasks and domain resolution failures. This does mean that updated DNS configuration could take a little while to propagate to our certificate issuer, but since this is on the order of seconds, it should not be noticeable under normal operation, and it provided enough protection for us to resolve the Sidekiq queuing issue. We also fixed several issues in our codebase that did not filter out deleted apps correctly, so that we are less likely to experience these spikes in the first place.

This is not the first time we’ve had Sidekiq-related incidents this year. We have plans to improve reliability around certificate issuance – our earlier work to move certificates out of Vault due to numerous incidents being an example – and another part of it is to decouple our certificate issuing pipeline from the rest of our API implementation, a Ruby on Rails app. The hope is that in this process we will make it both more scalable and easier to understand and debug.

#August 3: BGP misconfiguration dropped ingress traffic (14:57UTC)

A BGP misconfiguration while provisioning new edge capacity caused most traffic from Europe (and some US) endpoints to be dropped. The misconfiguration has been fixed and we have implemented safeguards against this kind of issue in the future.

#August 1: IAD MPGv2 Patroni etcd lease-expiration storm (16:40UTC)

MPG v2 in the IAD region saw a burst of Patroni health-check failures and timeouts when the region’s shared Patroni etcd cluster became overloaded. Existing databases were unaffected because Patroni entered failsafe mode, but new clusters could not be created. Investigation pointed to an intrinsic etcd problem: adding a new user bumps etcd’s Auth Revision, invalidating existing JWT tokens and forcing clients to reconnect. On reconnect, a thundering herd of clients overloaded etcd and brought it down. We mitigated by scaling up the Patroni etcd machines in IAD and lowering the bcrypt cost, and we’re exploring longer-term fixes.

#July 20: tkdb primary host outage caused token validation failures (07:07UTC)

The host running the primary node of tkdb, our token validation service, went offline due to maintenance and did not manage to reboot successfully due to a separate hardware issue. This caused token validation to time out, which in turn led to widespread failures in API operations that depend on authentication (Machines API, flyctl, etc.) and elevated 500s from platform services. API and dashboard functionality recovered over a few minutes once we restored the host to full service.

The root cause of this incident is, of course, the host failure, but a huge contributing factor is that tkdb is single-primary, and, unlike petsem (which powers app secrets and TLS certs), read replicas of tkdb do not function if the primary node is entirely offline. This is because, as a token validation service, the read replicas must be able to receive notifications about revocation to uphold security guarantees. We are currently looking into ways we can relax this requirement, keeping replicas up during primary outages, while avoiding impact on security properties provided by tokens.

#July 19: Missing app logs because we forgot a `systemctl restart` (14:32UTC)

Some customers reported that their app logs appeared to be missing. Initially we suspected that this was related to log ingestion (we use vector on each physical host), but further investigation revealed that… we forgot to restart the query service after adding a new storage node to the logging cluster, so it only queried the previous set of nodes. Because each app’s log stream is written to one storage node, apps whose logs ended up stored on the new node would return empty or incomplete results. Restarting the log query service restored normal log visibility.

#July 16: Vault throws one last tantrum on its way out the door (11:28UTC)

Recently we have been migrating our certificate storage away from Vault to our Petsem codebase, to reduce the number of distinct cluster-shaped things we need to think about. We reached the 100% mark on the rollout a few days before this incident, so this body of work was all but done.

This is, of course, when Vault decided to break in some manner. Since we were still validating things and hadn’t rolled out our code sans feature flags yet, there remained calls to Vault in both our GraphQL API and the Fly Proxy, particularly as a fallback for unknown certificates. Though these weren’t in the hot path of healthy requests, Vault hanging caused some issues across the services, and caused TLS handshakes to fail for otherwise functional Fly Apps for some number of minutes.

Unlike most writeups here, we don’t have (or, need) a firm root cause on exactly what broke and where. As soon as this kicked off we made the call to rip out the Vault functionality, since that was up next anyway. Without Vault in play, we got to blissfully ignore whatever went wrong with it. And we didn’t have much of a reason to dig into exactly which Vault/Petsem interaction in the proxy misbehaved, as the code that housed it was razed.

#July 14: SJC worker hosts locked up and rebooted (14:53UTC)

While debugging network packet loss in SJC, a tc qdisc (queuing discipline) configuration update was (unnecessarily) pushed out to all of SJC workers, which caused a number of them to lock up. We promptly started rebooting them and nursing them back to health.

Honestly, we are still not sure why this happened in the first place. The tc qdisc configuration was, after further validation, later very carefully deployed to all of our hosts in all regions, and none experienced a similar problem whatsoever. qdisc is not supposed to just randomly lock up entire systems: at most, it should cause network issues that we can easily recover from. So, it seems likely that we must have hit some weird kernel or NIC driver bug, but so far we do not know exactly which one. One thing is for sure though: we’ll have to be much more careful about even the most unassuming changes related to NICs.

#July 9: A couple of deployment issues (15:11UTC)

This day saw 2 separate instances of deployment-related issues centered around IAD happening around the same time.

The first one was related to Depot builders, where flyctl deploy would wait seemingly indefinitely for a Depot builder to become available. Initially, we believed that this was related to a single IAD host under immense I/O pressure, and was only affecting a subset of Depot builders with backing volumes on that host. Failed hosts with Depot volumes can sometimes prevent existing Depot builders from being reused until recreated via an explicit flyctl --recreate-builder flag. However, after we resolved the host issue (which took a considerable amount of time), it became clear that the Depot issue was not limited to the one host, but was impacting builds for any IAD builders. This is when an internal incident was declared and a status page posted.

It turns out that this was another case of capacity-related Depot issue. Because machines used for Depot builders are pretty large, when we’re under capacity constraints, Depot builders may not be able to successfully start with an existing volume, because that host has no spare capacity to run the builder machine. Usually, machines can auto-migrate on start when capacity error happens, but this is explicitly not enabled for machines and volumes as large as builders. Under these circumstances, no new Depot builder should land in IAD, but existing ones will continue to be attempted and fail to start. The fix here, before adding more capacity, was to purge existing Depot builders from IAD in order to have them recreated in other North American regions as needed.

Separately, our Docker registry, which runs as a Fly App managed by us, started throwing a lot of 5xx errors, while the Depot issue above was happening. This accounted for another bulk of deployment issues happening on this day, which unfortunately compounded with Depot-related issues above. Initially we thought the host DNS resolvers were acting up, since the registry app is throwing a lot of DNS-related errors. After ruling that out, we realized that even pinging the host (fdaa::3, where the DNS resolver also lives) from these machines was showing latency up to 100ms – definitely too much for basically loopback traffic! At first, we thought this is due to CPU starvation since the machines were all at 100% utilization. Scaling up just the CPU did bring down the latency, but it was still at several 10s of milliseconds. Some further investigation made us realize that these machines are running a lot of traffic over their TAP interfaces, and we do know that our TAP interfaces, for one reason or another, starts to struggle a bit under higher ultilization (> 1Gbps). Creating more machines for the registry to spread the traffic out immediately solved the issue.

We came out of this incident with some plans for improvement:

1. We should know about widespread deployment issues way sooner than we did on this day. We did not put up a statuspage because we assumed that the issue was related to a downed host, which already had its separate host issue, but it was in fact unrelated. We now have better monitoring for Depot issues related to machine and volume placement, and we will hopefully be paged much, much sooner should something like this happen again.
2. Even when there is capacity pressure, ideally, Depot builders should still be available, just placed in different regions, for new or existing users. We are working with Depot to add support for placing builders using geo region aliases, which allows much more flexible machine placement, and we are planning to integrate this with our API and flyctl so that local capacity issues no longer automatically translate to much wider deployment failures.
3. For registry, besides similar alerting, figuring out the performance bottleneck on our TAP interfaces is on our plan as well. Fixing this will improve experience across all machines, with the caveat that we’ll likely still place constraints to ensure fairness between machines on a host.

#July 3: Power supply failure in ORD (00:07UTC)

For redundancy, servers generally have two power supplies, connected to two independent power feeds: if one power supply fails, or one power feed goes down, the server can keep operating on the other power supply. During normal operation, the load is shared between both supplies.

Our servers are, of course, no exception. However, when one power feed of an ORD datacenter went down, we found our servers’ CPUs heavily throttled, to the point where they were unable to do any useful work. Later discussing with the provider, we found a misconfigured setting: the server would throttle the CPUs when one power supply failed or lost power.

Now, this isn’t a “oh duh, it should obviously not be like that” situation: this is a safety feature designed to not overload one power feed. Single-power feed operation, in some cases, can just be a fallback for the server to shut down safely without the ability to support normal tasks, even though this is not true in our case – we expect it to provide better uptime at full performance. The throttling setting makes sense as a safe default, and we should be explicitly opting out of the safety feature when we are sure single-power feed operation is safe in our case.

As a result of this incident, we are now working with our providers to ensure all of our servers’ settings are configured best for how our power feeds are set up.

#July 2: Certificate issuance outage (21:40UTC)

We issue certificates using Let’s Encrypt, who had a networking hiccup when failing over datacenters for maintenance. This caused some requests to fail, making our issuance jobs bail out and re-queue themselves. Apart from a handful of lucky hostnames who squeaked through, certificate issuance was out of action for a few hours. We renew certificates well before they expire, so no existing traffic was affected. Any apps that were in the process of setting up their hostname would have seen a delay in getting their first certificate. Once the upstream incident was resolved, all queued issuances and renewals were processed.

#July 1: Runaway cleanup job filled Redis (05:44UTC)

Our GraphQL API runs its background jobs on Sidekiq. When an app is deleted, we enqueue a cleanup job to tear down any associated resources, and that job in turn enqueues a job for every Machine the app ever had - even ones that had already been deleted.

One app had a very large Machine history. Its cleanup job loaded that entire history into memory at once to enqueue the per-Machine jobs, which pushed the worker’s memory high enough that the supervisor recycled it - killing the process and restarting the job from scratch on another worker. The loop repeated and quickly piled up tens of millions of per-Machine jobs. That filled the disk on the Redis instance backing Sidekiq; once Redis could no longer write its append-only file, job processing stalled and customers saw API errors for around 40 minutes. Already-running Machines were unaffected and kept serving traffic.

We mitigated by extending the Redis volume and restarting Redis, and shipped a change to make the cleanup job safer - it now iterates in smaller batches without pulling everything into memory and skips already-deleted Machines. Once Redis recovered, the backlog drained and error rates returned to normal.

#June 28: VictoriaMetrics ingestion delays and backlog (08:00UTC)

We had an extended period where our hosted metrics pipeline fell behind, causing Grafana dashboards and alerts (including fly-metrics.net) to show missing or delayed data. The underlying issue was uneven load distribution into our metrics ingestion “aggregator” layer, which led to CPU starvation on a subset of ingestion hosts and large metric queue backlogs that then took significant time to drain. We mitigated by rebalancing ingestion traffic (making the load balancer aware of backend load), tuning queue/throughput settings, and temporarily adding processing capacity to speed up backfill; metrics ingestion resumed and the remaining lag gradually cleared.

#June 25: Corrosion migration caused elevated CPU and routing failures in BOM and NRT (12:06UTC)

An ongoing Corrosion database migration on some hosts in BOM and NRT caused SQLite query planner statistics to become stale. As a result, the SQLite query planner chose full table scans for a particular fly-proxy query instead of using the appropriate indexes. This led to high CPU utilization and increased SQLite lock contention within fly-proxy. The resulting long-running read transactions prevented SQLite WAL truncation, further compounding the issue.

Users experienced HTTP 502 responses and connection resets for traffic routed through the affected regions, and requests to some Machine API endpoints timed out.

The issue was mitigated by stopping the migration and fly-proxy on the affected hosts, truncating the WAL and running ANALYZE on the affected tables to refresh SQLite’s query planner statistics. Once the statistics were refreshed, the query planner resumed using the correct indexes, and fly-proxy was restarted on the affected hosts. The migration script has also been updated to run ANALYZE after migrating each table to prevent this issue from recurring.

#June 23: Corrosion OOMs and wedged systemd (13:45UTC)

Corrosion, our internal state propagation system, was OOM-killed on a small number of hosts. This is usually not a big deal (save for the part where we need to figure out why it exceeded the generous memory limit we gave it), since the systemd unit is configured to restart the service automatically, and most other services on a host fail gracefully when Corrosion is down temporarily. This time though, the systemd units were stuck activating with seemingly no progress made, which triggered us to open an internal incident to investigate, and we also quickly increased Corrosion’s memory limit on all hosts to prevent any further OOM’s.

This turned out to be related to some fly-proxy changes we made recently. 2 facts that directly caused this:

1. fly-proxy‘s systemd unit contains a After=corrosion.service;
2. fly-proxy now waits for much longer on shutdown for unfinished connections.

When Corrosion got killed this time, it was in the middle of a fly-proxy rollout, which means that the fly-proxy unit was still deactivating on a lot of hosts due to the new shutdown logic. systemd enforces a reverse order based on After= constraints on shutdown, which means that before the last fly-proxy process stopped, a new corrosion.service cannot start successfully.

Since the proxy does not necessarily require a running Corrosion process nowadays thanks to our lazy-loading work, especially not on shutdown, the correct fix here is to simply remove that After= dependency, which we promptly did. The other loose end in this incident is why Corrosion OOM’d in the first place. Our conclusion here is that it is also related to the fly-proxy rollout: fly-proxy depends on something called Corrosion “updates”, which are generated in Corrosion and sent through a Corrosion-internal mpsc channel. The channel became blocked while the proxy restarted, which pushed Corrosion’s memory usage over limit, causing the OOM.

Because Corrosion updates are designed not to be fully consistent, and fly-proxy can deal with missed updates gracefully, we shipped changes to start dropping updates under this type of pressure. This should prevent this incident from repeating in the future.

#June 17: Singapore network outage (03:18UTC)

One of our upstream providers in Singapore region experienced an unexpected failure on a core router. This caused intermittent but complete loss of connectivity for a couple hours on June 18th; the issue reappeared on June 22nd, after which the affected device was replaced.

#June 15: Machines API Outage (15:07UTC)

Our token service tkdb became unavailable, which broke our macaroon token verification path. This caused a broad control-plane outage for approximately an hour: dashboard login/SSO and many Machines API operations failed, while existing app traffic continued to route normally.

On this day, tkdb needed to be migrated between two internal hosts. Many Machines API operations rely on tkdb for macaroon minting and verification, and tkdb is itself a Fly App that relies on the Machines API.

Migrating tkdb is something we have a known, written procedure for. The general idea is to:

1. Cordon the tkdb primary
2. Stand up a new Machine, let it catch up as a replica, then promote it to primary
3. Point all replicas at the new primary

During this window, minting new macaroons will be unavailable, but verifications will still work and the Machines API will remain functional.

A while back, a fly-proxy change inadvertently shifted the semantics of Machine cordoning. The new cordoning flow meant that cordoned Machines would not ever be load balanced to, but they were still eligible for requests that specifically name them. That is, requests using fly-replay to an instance, or the fly-force-instance-id header, would now each arrive at a cordoned Machine.

Because of that, when we cordoned the primary tkdb Machine, it would still receive replays from replicas. Additionally, stopping the primary wouldn’t work, as these same replays would auto-start the targeted Machine. The right decision here would have been to restore and uncordon this primary, and pause the migration. The less-right decision, which is what we did, was to destroy the old primary before promoting the new one. In theory this is roughly equivalent, and is a safe way to migrate the broadly similar petsem cluster, which is why it seemed reasonable.

The (overlooked) quirk for tkdb is that in order to guarantee a bounded lag on token revocation, replicas that fall too far behind the primary will throw their toys and proceed to replay all requests to the primary, rather than use a stale revocation list. In practice this meant that we were suddenly left with a cluster of replicas that would not function, which took out the Machines API for customers and for ourselves. Without the Machines API, we were unable to easily bring up the new primary Machine required to resolve this.

From here we had to resort to spinning up the required Machine by hand with flyd on a host, which has its own set of internal hurdles to overcome. The main responder in this incident was based in Europe, which additionally meant that their Machines API operations would prefer flaps, and thus tkdb, in Europe rather than the new primary in North America. These factors weren’t a problem in the initial incident, but they did complicate and slow the response for getting everything back up and running.

The technical root cause here is the cordoning flaw in fly-proxy, and the organizational root cause is that our documentation and runbooks for tkdb were not adequate to prevent this from happening. Both of these things are being fixed: In fly-proxy, cordoning will be fixed to prevent all proxy-routed traffic, and our internal documentation is being reviewed and improved for tkdb, and more broadly where other services have the same gap.

#June 12: Elevated Sprites error rates in SIN (02:27UTC)

Some Sprites API and dashboard requests hitting a SIN edge returned 500s for a subset of orgs with Sprites in the SIN Region. This was caused by a mismatch in code versions running in SIN and SYD regions, as well as the specific way the API handles some organization requests.

In addition to serving general API requests, each API node acts as the ‘manager’ for a subset of organizations in its region. Certain organization request types for some actions will be replayed through the network to that org’s manager node. The ultimate source of truth for this management data lives in object storage. If an API node fails, another node in the region steps in to manage the Sprites the dead node was responsible for. If both nodes in a region fail, API nodes in the next closest region will take up management. So on, so forth.

Earlier in the day one of the two API nodes in SIN failed due to corruption on its local cache volume. The failover process worked as designed, with the other SIN node taking over management of all orgs on the dead node. We brought up a replacement node in SIN and that started re-syncing with the cluster.

While the replacement node was still syncing, the second healthy SIN node restarted. Since it was the only healthy node in the region, this triggered the next closest region (SYD) to take over management of all SIN orgs. Again this succeeded without issue.

However when the two SIN nodes came up, they were unable to replay requests to the nodes in SYD. After some investigation we identified an earlier update had failed to update the SIN nodes to the latest code version. This caused an incompatibility, with the shape of request the SIN nodes were sending not matching what the SYD nodes were expecting. After identifying the issue we pushed out a manual version update to the new SIN nodes and normal operation resumed.

This only impacted requests hitting a SIN region API node, for Sprites that had their management re-homed to the SYD nodes. Requests hitting any other region’s edge for those same Sprites continued working as expected.

#June 11: Everyone lives on NULL island (06:03UTC)

Newly created WireGuard peers and Depot builders were placed in AMS (Amsterdam) regardless of where the user actually was. We had removed some datacenter coordinate metadata from Consul, believing it to be unused – but part of our control plane still sourced region coordinates from it, and without that data it located every region at (0,0). The logic picks the region nearest the user, but with distances all tied at zero it fell through to the alphabetically first region, AMS. Existing peers and builders were unaffected; only newly placed ones landed in the wrong region.

We mitigated this by sourcing region coordinates from our database instead of Consul, then cleaned up the Depot builders that had been pinned to AMS. We’ve also hardened the placement logic to fail loudly if coordinates go missing in the future.

#June 10: Two GRU edges OOM (19:35UTC)

Two edge nodes in the GRU region couldn’t keep up with exporting their metrics, and the backlog caused the hosts to run out of memory. The immediate impact lasted about 17 minutes, at which point we rebooted the affected nodes and returned them to the routing pool. Some traffic was disrupted, though this was not a complete outage in the region. We’ve tidied up our metrics pipeline a little since, paring back some high-cardinality metrics that contributed to the heavy load.

#June 4: Stale 6PN mappings wreaking havoc (09:18UTC)

This is another case where, as we were working towards improving the platform, we ended up with multiple ways of doing one thing, some of which are considered legacy and should eventually be removed, but the removal was never completed. An unexpected interaction between the old and new systems then wreaked havoc.

In this case, the system in question is 6PN, our private network powered by Wireguard that connects all of your Machines. When this system was designed, each Machine’s private 6PN address was bound to the host where it was created. This made routing simple to implement, but also started to cause issues when we migrated Machines between different hosts. The reason is that some apps depended on a static 6PN address per Machine: even our own legacy unmanaged Postgres offering depended on it, despite the fact that these addresses were never meant to be stable.

At some point, we finally decided that this is not sustainable and we should, instead, meet the expectation of a majority of apps: that is, to keep 6PN addresses stable. The first iteration of this work was a simple DNAT, where machines still get new 6PN addresses, but an eBPF program rewrites packets targeting a machine’s old 6PN address(es) to the new one. The price we pay is that, because technically the 6PN address still changes, we need to keep track of every single 6PN address a Machine has ever had. This is all stored in Corrosion, which bloated its storage, not to mention the map we needed to synchronize into the eBPF program.

This was changed roughly a year ago. Instead of keeping track of all old 6PN addresses, we simply made it so that Machines do not get reassigned a new 6PN on a new host if it is migrated. All Machines created after this change retain their initial 6PN after migration. Of course, this alone would break routing, because that depended on a per-host fixed 6PN prefix. Some sort of NAT is still needed, but now we only need to keep track of a Machine’s current host and its initial 6PN.

…which brings us to today. We have two types of “stable 6PN” Machines: some before the change above, and some after. The intention was that when an old-style 6PN Machine gets migrated, it will become a new-style stable 6PN Machine with all the new-style plumbing. We’d delete unneeded Corrosion entries in this case and slowly drain them away as they’re moved around. At some point this year, we realized that the Corrosion subscriptions used for old-style 6PN DNAT were creating a lot of load on Corrosion. As a result, we shipped a change to only apply 6PN DNAT rules once when the service responsible for this is started, since we did not except any new Machines to be created with old-style 6PN anymore. However, there was an oversight: in some cases, the existence of old-style DNAT rules actually overrides new-style stable 6PN’s rewriting logic. So, when a Machine gets migrated to use new-style stable 6PN, it is possible that some peers might still be rewriting its address to a host-specific address that no longer exists.

This exact scenario started happening first for our multi-tenant Consul clusters (used for unmanaged Postgres and LiteFS), and then for some customer Machines as a spike of rebalancing migrations happened for various reasons. A considerable amount of time was spent on triaging the issue because it was an unexpected failure mode. We did not expect that old-style stable 6PN would interact with new stable 6PN in this way, especially not several weeks after the last round of changes were deployed.

We mitigated this problem by adding code to delete old-style 6PN DNAT entries when new-style stable 6PN rules are set up. This, unfortunately, briefly caused another bug where the daemon responsible for this became too slow to catch up with Corrosion (we need Corrosion to decide whether a Machine has been migrated and thus needs rules for stable 6PN), which caused issues with Managed Postgres in LAX for a little while. This was then fixed up by making the cleanup code opportunistic and non-blocking for the main processing path.

We see a few directions as the next steps to preventing this from happening again:

1. Old-style 6PN DNAT mappings should really not exist anymore. We need to migrate all the remaining machines that still use it to new-style stable 6PN addresses.
2. The reason why the Corrosion subscription and its processing code became slow was partially due to the query’s inefficiency; we’re working on addressing that too.
3. We need a way to gracefully recover from such an event; the daemon should not just miss updates.
4. Finally, we should be able to “fill in” missing stable 6PN NAT rules even if the Corrosion subscription happened to miss some updates. The subscription can still be used for updates, but not as the single point of failure.

#May 30: Deploys blocked by billing error (02:18UTC)

For a few hours, deploys for some organizations were failing with a “We require your billing information” error, despite having just added payment methods or credits to their organizations. This was due to a mis-ordered deployment of a new Corrosion schema.

For some context: organization information is managed by our central GraphQL API backed by a local database in iad; when an organization is updated, for instance when the billing information is updated, the GraphQL API pushes the changes to the global Corrosion cluster so it can be read by the Machines API. When new information needs to be stored in Corrosion, we need to deploy two changes: a global change to the Corrosion (sqlite) schema, and a change to the GraphQL API to push the new data to the global cluster.

Earlier in the day, we had prepared a change to push some new organization data to Corrosion. This is usually a safe change, however this time the GraphQL API was deployed prior to the global schema being updated. This caused all organization updates to fail to be propagated to Corrosion, thus causing the Machines API to not know about the updated billing status of organizations. To resolve this incident, we quickly reverted the change to the GraphQL API and backfilled the missing data in Corrosion.

We are looking into ways to alert on repeated sync failures, as well as failing GraphQL API deployments if the Corrosion schema is out of date.

#May 28: West coast edge proxies overloaded (21:08UTC)

This incident requires some background which will become important later:

• fly-proxy: a Rust-based, userspace L4 / L7 load balancer
• Corrosion: our distributed service discovery / state propagation system
• Airtime: fly-proxy‘s inbuilt dynamic defense against sudden traffic spikes; this was put in late last year / earlier this year, before which we had no way to prevent one app from monopolizing bandwidth on a host. Earlier in the year, we spent some time tuning Airtime’s parameters so that it triggers near our bandwidth saturation point for what each of our edge servers can handle.
• Lazy loader: in the long before-times, fly-proxy used to ingest almost all data in Corrosion into its process memory, through Corrosion’s subscription API. That proved to not scale well a long time ago, and we switched to a lazy-loading model where only entries required for active requests are loaded.

The incident started with us noticing flappiness in our US west coast regions, primarily in SJC at the beginning. Our logs and metrics indicated that the lazy loader latency was high, on the order of 500 ms to several seconds. This means that many new requests will need to wait that long or even longer to be served. On the other hand, proxy’s CPU usage was not especially high, and neither was the inbound connection rate. We’ve seen this kind of issue before: it usually is indicative of inefficient sqlite queries, certain apps with excessively large state stored in Corrosion, or general host performance issues. At this point, we happened to have spotted one app with extremely large state in Corrosion, and quickly “concluded” that it must be contributing to the issue, so we put in a temporary mitigation and deployed the proxy in SJC.

It momentarily seemed to improve the situation, but latency quickly shot through the roof again after the new proxy processes warmed up. We began doubting whether it is inefficient sqlite queries, which we ruled out, or whether there was lock contention simply due to our recent growth resulting in increased connection rates. This is also the point where we noticed Airtime reporting increased bandwidth in SJC, but it was below what we have concluded before was the ceiling of what a single edge server could handle. In either case, our edge capacity in SJC was also underprovisioned due to a couple of servers being out of production, so we decided to first shift Anycast traffic to LAX and see if it handles the load better.

Again, initially it seemed to help, but after a while LAX started struggling as well (side note: at certain points we also attempted to shift traffic out of the west coast entirely, which was why edges in other regions may have been momentarily affected). We finally decided to adjust down the bandwidth limit of Airtime, even though we were pretty sure our edges could take the level of traffic seen throughout this incident. It did bring softirq CPU usage and host load average down, but the proxy was still struggling with slow lazy loader queries. We bounced the proxy, which seemed to clear up the lazy loader issues as well. This marks the end of the first acute phase of this incident.

It would have been nice if this was the actual end of the incident. It was not, and it was primarily due to 2 other issues:

1. Airtime, the system we used to limit impact of traffic spikes, works entirely within one single process and does not propagate its knowledge outside. This would not have been a problem (we initiated a hard-kill of all pending-shutdown proxy processes when we bounced them), if not for:
2. Due to a bug with how our proxy deployment script interacts with systemd, we have somehow left multiple instances of the proxy running indefinitely on some of the affected nodes (TLDR: systemctl kill does not actually transition a unit to a stopped state; combined with Restart=always it simply causes the process to restart);

The combination of these two means that any limit we set in Airtime could, at any point, become effectively doubled if some heavy connections landed on a different proxy instance, causing the same issue to repeat after the initial phase was resolved. It is also worth noting that the fact that we needed to bounce proxy processes after tuning Airtime is itself contributing to this issue: that revealed that there are issues with queuing behavior around the lazy loader. Specifically, it seems that it is possible to end up with effectively infinite queues waiting on the sqlite connections when lazy loader itself is slow (due to softirq contending with userspace for CPU under high load, for example), which will not resolve unless the process itself is bounced (and in turn, that revealed the other issues causing recurrence of the incident).

In summary, this incident was caused by a combination of factors:

1. Our edge capacity is underprovisioned in some regions; they have not caught up with our recent growth in user base.
2. Airtime’s tuning no longer matches reality, either due to a shift in traffic patterns or other non-bandwidth scaling issues in the proxy.
3. A bug caused multiple active proxy instances to coexist without code to handle shared state.
4. The lazy loader exhibits runaway queuing behavior at high load.

We’re working hard to address each and every one of these issues. As a starter, we are going to provision significantly more edge capacity in the coming weeks/months. We have addressed the bug that caused multiple proxy instances to coexist, and changed Airtime so that, for now, it applies a much stricter limit when it is not the expected active proxy instance. We have fixed load-shedding behavior in the lazy loader so that there is a more reasonable upper bound on the maximum latency serving requests. Other work is currently under way:

1. We believe that the reason why proxy seems to run into lazy loader-related performance issues much earlier now, compared to before, is due to our single coarse-grained lock on the proxy’s in-memory state is no longer scaling well as we grow. We have observed high queuing delays not in sqlite queries, but simply in trying to insert data into the in-memory service catalog. We’re planning to shard the catalog and move to finer-grained locking, assisted with testing such as Antithesis to ensure migration to this does not cause more outages.
2. We are going to rework Airtime so that it reacts better to overall system load instead of just the proxy. This will hopefully serve as a backstop when we somehow end up with multiple proxy processes running, or when any non-proxy processes on the same host consume any of the bandwidth headroom.
3. We’re looking into better monitoring for when the proxy is not under its expected configuration.

#May 27: App creation timeouts from petsem-certs disk full (03:43UTC)

New app creation requests (including flyctl apps create) were failing with 504 timeouts because they weren’t able to create certificates in petsem-certs, our new certificate store (which we’re in the process of provisioning and migrating to from Vault). While petsem-certs is not yet operational, we are writing certificates to the store, which (disappointingly) caused it to run out of disk space. Existing apps and Machines were unaffected - only app creation timed out. We restored app creation by expanding the storage.

Since we’re dual-writing to both petsem-certs and Vault, and the Fly Proxy is reading from Vault, we didn’t expect the loss of petsem-certs to have any impact and it hadn’t yet been hooked up to our monitoring. We also had excessive retries on requests to the store, which caused the issue to present as a timeout rather than a failure, so we’ve tuned that as well.

#May 20: SYD egress IP networking broken on new workers (04:46UTC)

Some newly provisioned hosts in our Sydney (SYD) region failed to be configured properly for egress IP connectivity. As a result, a number of Machines using egress IPs in the region were unable to access the network. During the incident, we immediately migrated the affected Machines to known-good hosts.

Recently, we moved configuration for some infra components (including the VXLAN interface backing egress IPs) to a new, more scalable system. The rollout appeared to be successful, but an interaction with a legacy deployment method caused the configuration service to not be restarted correctly - so VXLAN worked on existing hosts, but would not be provisioned on new hosts.

Our egress IP monitoring was set up in a world where egress IPs were machine-scoped rather than app-scoped (see this forum post for more context). As such, a couple monitoring Machines were set up in each region, and not every host was being monitored - as that would require one IP address for every host. After this incident, we ported egress IP monitoring to app-scoped IPs with a Machine running on every host.

#May 19: Fly dashboard outage from broken GraphQL API deploy (23:13UTC)

A deploy of Fly’s GraphQL API service (which backs much of flyctl and parts of the dashboard) reduced the number of healthy instances enough that the HAProxy layer in front of it began timing out its /status health checks and returning fast 503s, which showed up as intermittent dashboard/API failures and elevated deploy failure rates. We recovered by removing the broken instances and cloning a known-good Machine to restore capacity until HAProxy backends were stable and green again.

Afterward we found that a local fly deploy of the GraphQL API service could produce a broken image because it skipped a CI-only step that fetches a supporting binary (resulting in an empty placeholder being copied into the image), and we merged a change to prevent that failure mode.

#May 16: FRA managed Postgres control-plane outage (12:23UTC)

Managed Postgres clusters in the FRA region became intermittently unreachable after the regional Kubernetes control plane (FKS) got overloaded/stuck and the Kubernetes API began timing out. Because Patroni uses Kubernetes for coordination in this setup, those API failures prevented clusters from reliably determining primary/replica state, causing widespread connection failures and flapping health. We recovered by reducing resource pressure, defragmenting the affected etcd instances, restoring the control plane’s ability to reconcile, and then repairing clusters one-by-one.

#May 15: Bad TLS cert update broke Consul (16:12UTC)

A configuration automation run accidentally overwrote Consul TLS certificates with invalid ones, which caused Consul lookups to fail across parts of the fleet. We have spent a lot of time in the past couple of years to remove Consul as a key dependency, and as such most aspects of our platform were not directly impacted: all running Machines were unaffected and fly-proxy routing remained functional. The impact was concentrated on the few control-plane operations that still rely on Consul: mainly fly ssh console and OIDC tokens. We restored the correct certificates and restarted Consul agents to pick up the fixed TLS configuration, after which errors and alerts subsided.

#May 12: Usage ingestion blocked by stuck Oban jobs (16:48UTC)

Usage ingestion fell behind and then stopped making progress when several background jobs hung indefinitely, eventually consuming all available worker concurrency for the ingestion queue. This was triggered by a bug in an Elixir decimal dependency where converting certain values (like 0.0) to an integer could loop forever, causing specific volume-usage receipt processing jobs to never finish. We fixed this by updating the dependency to a version containing the upstream bugfix, after which ingestion resumed and the backlog drained; once the queue cleared, the delayed ~18 hours of usage data was backfilled and reflected normally.

#May 11: Kernel upgrade caused machine `stdout` to become wedged by Cloud Hypervisor (14:30UTC)

This is probably one of the more interesting / confusing / complicated bugs we have had since the revival of Infra Log. It began on this day with us receiving reports about an outage of the Upstash Redis extension, reflected on their status page as well. Upstash Redis, when used as a Fly extension, runs on Fly Machines, just like other customers. The only difference is that, for various reasons, we run their Machines using Cloud Hypervisor rather than Firecracker. This has never caused problems before, and initially we were pretty certain this is an issue on Upstash side. As we worked with them to investigate, though, we got something confusing: the report that these Machines are stuck writing to