API Architecture for Massive Scale
A little background
I usually like to skip the backstory and dive right into the technical details, but this one deserves some context.
Picture this: It’s 2020. The world is locked down in the middle of a global pandemic. I’m working as a contractor for NBC Universal, building data pipelines to migrate their massive media catalog to the cloud for international syndication. Life is… uncertain.
With COVID-19 death tolls climbing daily, the reality hit hard: I needed a full-time position with healthcare benefits for my family. That’s when CNN came calling with an offer to join as Principal Architect in their Digital Politics Group.
Talk about timing. The 2020 Presidential Election was shaping up to be one of the wildest in American history, and suddenly I found myself right in the middle of it all. Working alongside brilliant engineers and seasoned political analysts, I got a crash course in just how messy—and fascinating—U.S. politics really is.
Here’s the thing about CNN Politics data: when election season hits, everyone consumes it, regardless of their political leanings. And in 2020, we weren’t just dealing with record traffic. Between foreign actors probing our systems and the relentless “fake news” rhetoric, our APIs had bullseyes painted on them.
The mission was crystal clear: deliver election results to a global audience. With viewership at all-time highs (thanks, lockdowns) and political tensions through the roof, we needed a system that would absolutely, positively not fail.
Oh, and one more thing—I had less than two months to design, build, and deploy the entire system before Election Day.
No pressure.
Design considerations
Before architecting any solution, I needed to understand what we were really dealing with:
- High write throughput: Vote data could update every second during live counting, then sit dormant for days or months during the officiating process (and all those recounts…)
- Extreme read throughput: We’re talking billions of requests served
- Multiple data views: Different audiences needed different transformations and aggregations of the same underlying data
- 100% uptime: Failure was simply not an option
Traditional approaches (and why they’d crash and burn)
Let’s talk about what wouldn’t work.
Your typical API setup looks something like this: requests come in, hit your application layer, which queries a database, performs joins and transformations, and returns a response. Whether you’re using SQL with normalized tables or NoSQL with document types, you’re essentially doing real-time data composition on every request.
This design has some glaring problems at scale:
Data layer bottlenecks:
- Can you provision enough read/write nodes for billions of requests?
- What happens when writes start deadlocking?
- How do you handle replication lag?
- Why write data that hasn’t even changed?
Expensive access patterns:
- You’re recomputing the same transformations over and over, even when the underlying data is identical
- Your URL structure becomes a tangled mess trying to map every possible data view
Disaster recovery nightmares:
- How many hot/hot stacks do you need across multiple regions to guarantee uptime?
Sure, this pattern works great for most applications. But when you’re staring down billions of requests for data that rarely changes? You’re toast.
“Just add caching!” you might say. And you’d be partially right.
A cache layer can definitely help reduce unnecessary computation. But here’s the catch: caching is a balancing act. Cache too aggressively, and you serve stale data. Cache too conservatively, and you’re not solving the problem.
Remember, we were operating under the fake news microscope. Data integrity wasn’t just important—it was everything. One instance of serving incorrect election data could undermine trust in the entire system.
What about microservices? Each resource gets its own API that can scale independently, right?
The microservice pattern does help with data hotspots. Some datasets might be relatively static with few reads, while others get hammered constantly. Independent scaling prevents one hot endpoint from dragging down the rest.
But here’s the thing: microservices don’t solve our fundamental scale and disaster recovery challenges. They just distribute the problem across more services. And GraphQL? It helps with URL sprawl, but the only fallback for GraphQL going down… is more GraphQL.
We needed something fundamentally different.
Reactive architecture to the rescue
Throughout my career—from integrating robotic systems with real-time synchronization to cloud-scale media distribution—I’ve noticed a pattern: event-driven architecture almost always wins.
For this project, I took event-driven thinking a step further by implementing Flow Based Programming. This approach allows data to change and flow independently of how fast (or slow) it’s being consumed.
That’s the key insight: decouple data volatility from consumption patterns.
Let me break down the three loosely-coupled components that make this work.
Intelligent writes + CDC
First, we have our data sources producing events. These might come from API polling, message brokers, or batch processes—doesn’t matter.
Here’s the clever part: before writing any data, we compute a hash. If the hash matches what we wrote last time, we skip the write entirely. No unnecessary I/O, no wasteful downstream processing.
This keeps our Change Data Capture (CDC) output stream clean—it only contains actual changes. You get closer to “exactly once” processing semantics, though that can introduce its own challenges around data freshness that I’ll save for another article.
Transform reactively
Now that we have a clean stream of real changes, we can transform the data only when it actually changes rather than on every request.
Think about that for a second. In the traditional design, you compute transformations and aggregations during the request/response cycle, potentially thousands of times per second, even when the underlying data hasn’t changed. In our reactive design, we pre-compute every permutation once when the data changes, then serve pre-computed results at lightning speed.
This completely decouples write patterns from read requirements.
The secret sauce is the content-based router pattern. Each transformation component subscribes only to the specific events it cares about. Want a dataset showing only Senate races where a woman is leading? Your component can filter for exactly that and ignore everything else.
Even better: transformations can produce new events that trigger other transformations, creating powerful data processing pipelines. And because each transformation is independent, you can add new data views without touching existing ones.
Distribution
Finally, we need to serve all these pre-computed datasets to the world.
In a large organization, different business units often need the same data but must remain isolated in separate cloud accounts for security, compliance, or billing reasons. By letting each API stack subscribe to specific datasets via content-based routing, every team can scale independently while sharing the same underlying data stream.
Since we’ve pre-computed every permutation, our API routing becomes beautifully simple:
/{resource-category}/{resource-type}/{resource-key}
Real examples:
/weather/humidity/US.json/weather/humidity/US-GA.json/covid19/reported-cases/NYC.json/results/county-races/2022-AG-CO.json
Notice something? These are just file paths. That means we can write the same pre-computed data to both our high-speed compute layer (backed by Redis) and to object storage like AWS S3.
This dual-write strategy was my sleep-at-night insurance policy. Even if our entire API infrastructure went down, we could fall back to serving static files from S3. Sure, S3’s eventual consistency might mean the data is a few seconds stale compared to our compute layer, but we’d still be serving accurate election data to the world.
That redundancy was non-negotiable.
Oh, one more thing
Here’s a bonus feature that falls out almost for free: reactive notifications to external systems when data changes.
This was crucial for our CDN cache invalidation strategy. The moment fresh data hit our system, we could automatically invalidate cached entries at the edge. No stale data, no manual intervention.
You can imagine countless other use cases: triggering Slack notifications when specific conditions are met, firing webhooks for downstream systems, kicking off automated testing when certain thresholds are exceeded—the possibilities are endless.
The outcome
Election week 2020 was one of the most intense periods of my career. Days blurred together as we monitored dashboards, optimized queries, and held our breath every time another state started reporting results.
The system held.
We served billions of requests with accurate, real-time election data to a global audience. CNN recorded two of the highest traffic days in its 40+ year history, and our architecture didn’t even break a sweat.
I’m proud of the design and implementation, but this success was never about one person. It was the result of an incredible team at CNN Digital Politics who trusted this unconventional approach, worked around the clock to make it happen, and had my back every step of the way.
To that team: thank you. ❤️
Until next time…
If you’re building systems that need to scale to extreme levels while maintaining perfect reliability, consider reactive architectures. Decouple your data changes from your consumption patterns. Pre-compute what you can. Build redundancy at every layer.
And remember: sometimes the best solution isn’t the one everyone else is using—it’s the one that fits your specific constraints.
Stay curious, keep building, and I’ll catch you in the next one.