Election night, behind the dashboard

We forecasted that reading across the Parse.ly network would reach about 1.76x that of a normal Tuesday midterm election night, based on our 2018 and 2022 data. It hit 2.54x. We may have underestimated the attention this midterm would get, but we called the attention peak’s timing within 15 minutes (11:15pm ET!).

One thing to note as you’re reading: there were really two peaks on election night. The most traffic hit our servers around 9:30 pm. The biggest jump over a normal Tuesday came at 11:15 pm, when a “typical Tuesday night” has already gone quiet.

*Reading across the network on midterm election night, as a multiple of a normal Tuesday. Actual vs. what we expected from 2018 and 2022.*

Our work to support this scale of traffic didn’t just start on Tuesday.

Our customers are the names you read every day, so we plan for traffic to keep climbing. And when it rains, it pours: our last highest-traffic day was the Sunday right before the election. In the days leading up to Tuesday, we scaled the servers up and added alerts for any time we approached a limit, so when election night arrived, we knew we had it covered.

At 7:22 pm Tuesday, hours before the peak, our on-call team saw traffic had already climbed to some of the highest levels we’ve seen. Because of that preemptive work, our servers were at only about 15% of total capacity. Even at our busiest point, just after 9:30 pm, processing load stayed under 37%.

*Sunday’s peak was mostly app traffic. Tuesday’s was mostly web, which is more taxing on our servers.*

The rest of the night, roughly:

8:35 pm. A lookup service slowed down and metadata for new articles briefly started falling behind. Pageviews were fine and the slowdown stayed under 30 seconds, so the team prepped a fix and kept watching.

9:30 pm. Busiest point for the backend, about 73,000 pixels a second. Realtime processing was 37% busy.

9:45 pm. A metadata lag started growing, so the team deployed the fix it had already prepared. The service caught up within 10 minutes.

11:15 pm. Biggest jump over a normal Tuesday, 2.54x.

12:02 am. Even the West Coast was winding down, and on-call called it.

*Pixels per second coming into Parse.ly on election night, against the same hours on a normal Tuesday (Oct 20).*

Through all of that, most pageviews appeared in Parse.ly about two seconds after they occurred. Eight weeks ago, that was closer to ten seconds. We’ve made significant improvements to our efficiency since then. Even during the slowest 30-second stretch, 90% of pageviews were settled within 39 seconds (and most were much faster than that).

Seconds from a pageview happening to it landing in Parse.ly. The spikes came when a surge of website updates slowed some of our database writes. The pipeline slows itself down by design to protect them and self-heal afterward.

A lot of what went right came from the weeks before. The team load-tested up to 150k pixels per second at the end of October, added standby servers, and made our pipelines more efficient.

That said, there’s still work to be done. A few things are on the list before the next big night, including an update to our load balancers, which should unlock at least 3x the capacity we have today.

We’ve already started on the next round. After profiling election-day traffic, the team cut CPU use per unit of traffic by 19%, made one of our busiest pipeline steps 2.7x faster, and freed 80–140 GiB of memory, with zero failures along the way.

We tracked over 40 million “reads” (one view of an article about a race or measure) over election week. We’re proud of the improvements we’ve made and the team’s dedication to meeting our customers’ expectations for timely data and insights during their most important events.