Fixie Status

[Resolved] Bici group degraded (EU SOCKS)

Mon, 16 Feb 2026 23:48:00 +0000

Resolved - The issue has been resolved. Root cause was an interaction between our load balancer and patch management system that caused hosts to restart and then be replaced. (01:11 UTC — Feb 17)

Investigating - The bici proxy group in EU SOCKS is experiencing periodic spikes in latency and errors. We are investigating. (23:48 UTC — Feb 16)

[Resolved] Elevated latency on Olympic proxy group

Thu, 12 Feb 2026 11:10:00 +0000

Resolved - Latency on the olympic proxy group has returned to normal levels. (15:00 UTC — Feb 12)

Investigating - We are investigating elevated latency on the olympic proxy group in EU HTTP. (11:10 UTC — Feb 12)

[Resolved] Errors reported in bici group (EU SOCKS)

Fri, 30 Jan 2026 01:22:00 +0000

Resolved - The issue has been resolved. Root cause was a routine update where new instances took longer than expected to become available. As a mitigation, we will be updating our deploy strategy to prevent this in the future. (01:28 UTC — Jan 30)

Investigating - Monitoring is reporting errors in the bici proxy group in EU SOCKS. We are investigating the issue. (01:22 UTC — Jan 30)

[Resolved] SOCKS proxy groups experiencing high latency

Mon, 26 Jan 2026 00:50:00 +0000

Resolved - All SOCKS proxy groups have recovered. (00:57 UTC — Jan 26)

Investigating - SOCKS proxy groups bici, century, and speedway are experiencing high latency due to a cascading failure from the previous database connectivity incident. (00:50 UTC — Jan 26)

[Resolved] All proxy groups unavailable

Mon, 26 Jan 2026 00:47:00 +0000

Resolved - All proxy groups have been restored. The root cause was a bad infrastructure as code change that made our databases unreachable from our proxy groups. (00:50 UTC — Jan 26)

Investigating - We are investigating reports that all proxy groups are unavailable. (00:47 UTC — Jan 26)

[Resolved] Degradation in Olympic proxy group

Wed, 07 Jan 2026 16:43:23 +0000

Resolved - Issue been resolved (16:52 UTC — Jan 7)

Investigating - Only olympic proxy group is affected. All other proxy groups in EU function as expected (16:51 UTC — Jan 7)

Investigating - We are investigating issues with both proxy types in EU/US regions (16:43 UTC — Jan 7)

[Resolved] Major outage on Speedway group

Wed, 31 Dec 2025 09:54:46 +0000

Resolved - This incident was resolved at 10:15 AM UTC after approximately 75 minutes of downtime.

The root cause was a dependency update. We had rolled out incrementally for >6 hours without incident, but hit an edge case in error handling that resulted in an unhandled exception. We mitigated the issue with a forward fix and will review our policy for rolling out dependabot upgrades. (10:28 UTC — Dec 31)

Monitoring - We’ve fixed the issue and monitoring the system. The issue was related to an edge condition around latest dependency security update that didn’t surface during the daytime testing and monitoring. (10:19 UTC — Dec 31)

Investigating - We are investigating an outage (09:54 UTC — Dec 31)

[Resolved] Elevated latency in Speedway group

Tue, 30 Dec 2025 05:24:07 +0000

Resolved - The service is fully restored and we will continue monitoring the proxy group (05:28 UTC — Dec 30)

Monitoring - Speedway group had intermittent (short) failures throughout the day. We’ve identified the issue and in the process of testing monitoring and fixes. (05:26 UTC — Dec 30)

Monitoring - Speedway group had intermittent (short) failures throughout the day. We’ve identified the issue and in the process of testing monitoring and fixes. (05:24 UTC — Dec 30)

[Resolved] Elevated latency in Speedway group

Fri, 12 Dec 2025 17:18:10 +0000

Resolved - Outage was resolved by scaling this group vertically (17:25 UTC — Dec 12)

Investigating (17:18 UTC — Dec 12)

Investigating - We are investigating degraded performance in the Speedway socks proxy group (17:18 UTC — Dec 12)

[Resolved] Velodrome group severely degraded because of AWS outage

Mon, 20 Oct 2025 09:31:07 +0000

Resolved (09:35 UTC — Oct 20)

Resolved - AWS experienced a DynamoDB outage in us-east-1 starting at October 20, 2025 at 3:11:00 AM UTC-4.This caused a high number of requests to Velodrome to fail starting at approximately 4:34am UTC-4.Service was restored at 5:25am GMT-4. You can find more information about the underlying AWS outage here: https://health.aws.amazon.com/health/status?ts=20251020 (09:31 UTC — Oct 20)

[Resolved] Elevated latency in the Speedway group

Tue, 01 Apr 2025 14:57:00 +0000

Resolved - The Speedway socks proxy experienced a protracted period of elevated latency on one of its two nodes, causing high latency on ~50% of requests for 90 minutes. This was resolved by scaling vertically. (16:38 UTC — Apr 1)

[Resolved] Elevated latency in Speedway proxy group

Tue, 01 Apr 2025 02:28:40 +0000

Resolved - Fleet update completed and response times have recovered. We will continue to monitor. (02:42 UTC — Apr 1)

Identified - Latency was caused by a significant load spike. We are mitigating the issue by scaling this cluster vertically. Half of the fleet has updated and we are seeing response times recover. (02:39 UTC — Apr 1)

Investigating (02:28 UTC — Apr 1)

[Resolved] Some SOCKS proxies were unavailable in the EU region

Fri, 10 Jan 2025 22:08:00 +0000

Resolved - From 10:08 PM Friday January 10th (UTC) to 1:21 AM Saturday January 11th (UTC), the bici proxy group was unavailable. This proxy group is a SOCKS proxy group in the EU Region.

Fixie uses AWS DocumentDB for some state. AWS made an unannounced change to the cname through which instances in the EU region connect to their managed DocumentDB cluster. This meant that proxy servers that had sufficient available, existing connections in their connection pool were able to serve requests without issue, but proxy servers could not start cleanly. (01:21 UTC — Jan 11)

[Resolved] HTTP Proxies were degraded in the EU region

Mon, 12 Apr 2021 18:42:07 +0000

Resolved (18:44 UTC — Apr 12)

Resolved - HTTP proxies were degraded in the EU region this morning. The issue was resolved at 8:25am CST this morning.

We are improving alerting for EU-specific outages to improve our response in the future. (18:44 UTC — Apr 12)

Monitoring (18:42 UTC — Apr 12)

[Resolved] Signup was down

Sun, 31 May 2020 17:11:33 +0000

Resolved (17:12 UTC — May 31)

Resolved - On Saturday, May 30, the signup service was down for most of the day. This is now resolved. (17:11 UTC — May 31)

[Resolved] Network issues in EU Region

Sun, 15 Mar 2020 15:33:55 +0000

Resolved (15:34 UTC — Mar 15)

Resolved - Monitoring indicates that proxies in our EU region were unreachable from application hosts in US West. The proxies are healthy and are resolving without issue from US East.

We believe this is a temporary network partition on the AWS side, and is now resolved. (15:33 UTC — Mar 15)

[Resolved] All Fixie proxies degraded

Sun, 29 Sep 2019 11:30:00 +0000

Resolved - From approximately 11:30 UTC to 14:30 UTC, Fixie was degraded across all proxy groups. Response times spiked and a high number of requests timed out. The root cause was a load spike which, downstream, caused a spike in load to our database hosts, eventually causing them to run out of CPU credits. This is now resolved. (14:30 UTC — Sep 29)