Overview
Convex is a reactive database designed for web developers, allowing them to write their entire backend in TypeScript. Since Convex replaces developers’ existing stacks, consistent performance is a top priority. After running into unexpected latency and memory regressions in production, Convex turned to Polar Signals Cloud for continuous profiling and insights.
Challenge
Before Polar Signals, profiling memory usage in production was unreliable, highly manual, and only done by a few engineers on the team. Symbolization was finicky and often didn’t work in the middle of an outage. These challenges led to unpredictable memory spikes, causing the system to oversubscribe resources without clear diagnostics. Additionally, debugging CPU utilization required a similar amount of manual effort, and the team mostly profiled and optimized in their development environment..
Solution
Polar Signals Cloud provided essential situational awareness for both memory and On-CPU time across the system. A critical issue affecting p99 latency for a customer was diagnosed immediately by identifying an unexpected On-CPU time spike. Once pinpointed, the issue was quickly resolved, leading to improved system performance.
Key features that stood out during the selection process included:
- Automated Symbolization: Eliminated the need for manual debugging, making profiling more efficient.
- Seamless Debug Symbol Management: Unlike manual collection, Polar Signals Cloud ensured stripped debug symbols still worked for profiling.
"A particularly nasty bug, where we had really high p99 latency for one of our customers, was diagnosed immediately by seeing a large On-CPU time span for an unexpected part of our system. After finding that issue, we were able to fix it quickly." - Sujay Jayakar, Co-Founder of Convex
With these capabilities, Convex streamlined their debugging process and improved their ability to diagnose and fix performance issues.
Why Polar Signals
Polar Signals Cloud provided a significant advantage by automating the profiling and symbolization process. The ability to seamlessly work with stripped debug symbols was a major improvement over previous manual efforts.
Results
By leveraging Polar Signals Cloud, Convex experienced key improvements, including:
- Faster Issue Resolution: Quickly diagnosing and resolving a major On-CPU bottleneck that impacted p99 latency.
- Enhanced Memory Profiling: Gained visibility into memory spikes and oversubscription issues, leading to better resource management.