Taking 70 million messages a day off SQS

This post tells the story of a change we made to the Decision API, the service behind AB Tasty’s Feature Experimentation & Rollout product. We moved the processing of its activation events out of an SQS queue and a Lambda function and into the API itself, keeping the old path as a fallback. I’ll go through why we did it, how it works, how we rolled it out, and what it changed, on the AWS bill and elsewhere.

The Decision API in a nutshell #

When an application needs to know which variation of a feature a visitor should see, its SDK asks the Decision API. The API looks at the campaigns configured for that environment, at the targeting rules and at what the visitor has already been assigned, and answers with a decision.

When the visitor is actually exposed to a variation, the SDK calls the /activate endpoint, often with several activations grouped in a single request. Each activation has to be recorded: an event is sent to the data collection pipeline so that the experiment’s reporting can count that visitor. For some features (keeping a visitor in the same variation across sessions, for instance), the assignment is also written to a dedicated DynamoDB table.

None of this needs to happen before the SDK gets its answer, so it is done asynchronously. That is where SQS came in.

How events used to travel #

The Decision API runs on EC2 Spot instances, in an Auto Scaling group in each of our 12 AWS regions: from two instances in the quietest regions to about ten in Paris. Each request that produced events sent one message to a regional SQS queue. A Lambda function consumed that queue in batches of 50 messages (or whatever had arrived after 30 seconds), removed the duplicates, sent all the events of the batch to data collection in a single HTTP request, and wrote the assignments to DynamoDB.

It is a textbook setup, and it worked well for years. The queue absorbs bursts, the Lambda function scales on its own, and the API can answer the SDK right away instead of waiting for the events to reach the reporting pipeline. It only became a problem because of the volume, and the way SQS is billed.

Paying per message #

SQS charges per request: every message costs one request to send it, plus its share of the requests that receive and delete it. At $0.40 per million requests, that is nothing, until you send around 70 million messages a day. That was our daily average across the 12 queues, with the Lambda function invoked about 1.4 million times a day to consume them. SQS alone cost us around $1,800 a month, more than $20,000 a year.

That cost had an annoying side effect. The /activate endpoint is rate limited, with limits that depend on the customer’s plan. Since every call turned into a paid message, raising the limit of a large customer meant a significant cost increase on the SQS side alone. Some of them were hitting their limit and getting HTTP 429 errors from their SDKs, and that is what put the subject at the top of the list. So we wanted to stop paying SQS for something that did not need to leave the process, and make raising those limits cheap again.

Moving the work into the API #

We first looked at a solution on the SDK side, letting the SDKs skip the Decision API when an activation did not need it. Acting on the server side turned out to be simpler and quicker to ship, and it would benefit every customer at once, whatever their SDK version.

Keeping SQS and packing the events of many requests into each message would also have cut the cost: SQS bills per 64 KB of payload, so one message carrying many events costs far fewer requests than one message each. But to do that, the API has to hold events in memory before sending them, and that is where the durability goes. And once the events are in memory anyway, the API might as well process them itself.

The Decision API is a long-running Go process, on instances with plenty of spare capacity, and the work the Lambda function is not heavy. So that’s what we did: SQS and the Lambda function stay exactly as they were, as an overflow for when the local side cannot keep up.

The design has two stages, both of them bounded in-memory queues:

• An aggregator collects the events: one per request, whatever the number of activations it carries. It holds up to 1,000 of them and builds batches of 50 (or whatever has arrived after 30 seconds). Those are the same numbers as the Lambda trigger, so data collection receives batches of the same size as before.
• A worker pool processes the batches: a couple of workers per CPU, with a short waiting line of batches in front of them. A worker does what the Lambda function used to do, with the same code: remove the duplicates, send the batch to data collection in one request, write the assignments to DynamoDB.

When one of these queues is full, nothing blocks and nothing is dropped: the event that does not fit in the aggregator, or the batch that does not fit in the worker pool, is sent to SQS, exactly as before. The request never waits on a slow downstream service, and under pressure, the API simply goes back to the old architecture. Since the queue and the Lambda function were already there, this safety net came for free.

Around that core, there was the usual work to make it production ready: a clean shutdown that flushes what is in memory when an instance stops, and Prometheus metrics at every step with a Grafana dashboard on top, since once SQS goes quiet they are the only way to see what is going on.

Load testing #

Our DevOps engineer set up load tests with k6 on our staging environment. After a first round at a moderate rate went fine, we pushed to 1,500 requests per second, well above what a single production instance usually sees, to watch the fallback in action.

The first run was not great: almost a third of the events could not even enter the aggregator and went to SQS. The cause was a little embarrassing. The aggregator does one thing at a time: it takes new events in, and hands full batches to the worker pool. When the pool was full, it sent the batch to SQS instead, one message at a time, and took no new events while doing so. Its own buffer filled up. Once the batch was sent in groups of ten messages in parallel, the share of events turned away at the aggregator was cut in half on the same test. The rest still went through SQS, as designed.

Rolling it out, one wave at a time #

The original plan was cautious: deploy to our quietest region, wait about ten days, then go everywhere progressively. In the end, the DevOps engineer and I went through the 12 regions in four waves over three days:

1. Three of our quietest regions, plus Canada, added on purpose because the others did not have enough traffic to prove anything.
2. Three mid-sized regions the next morning.
3. Three larger ones the same afternoon.
4. Paris and Northern California, our two largest regions, the day after.

Between the waves, I kept an eye on the Grafana dashboard and made several test requests to /activate in different regions, then checked that the corresponding events landed in our ClickHouse data warehouse, to test the whole chain end to end. After the last wave, the SQS charts were flat in every region, and they have stayed that way.

The results #

The saving on SQS is around $1,700 a month, or $20,000 a year, which is about what we had estimated before starting. That is not going to change the company’s life, but it is real money. And it grows with the traffic: the Decision API now handles about 117 million events a day, against 70 million SQS messages per day before (one event replaces one message, so the two numbers can be compared). That is about 70% more, absorbed at no extra cost on the queueing side. With the old pipeline, this traffic would cost around $3,000 a month in SQS today, so the saving is now closer to $35,000 a year. On the instances, I could not see any change in CPU usage around the rollout, and the fleet did not grow because of it.

As for the rate limit, which is where all this started, it did not go away at once. Limits are set per customer, and with the cost argument gone, changing them became a product decision only. The customers who needed more room could get their limits increased.

Is the fallback still worth it? #

Here is every message that went to SQS since the rollout.

About 21,000 messages in total, on roughly one day out of six. The largest spike, about 12,000 messages, came from Paris within a single hour. Most days, the fallback does nothing at all.

I think it is still worth it. It costs almost nothing (the Lambda function keeps polling the empty queues, for a dollar or two a month), and in exchange a burst that the local queues cannot absorb turns into a few thousand SQS messages instead of lost events or errors. The chart also shows that the chosen queue sizes are more than enough in production, where the traffic is spread over many instances and regions.

What we gave up #

With SQS, an event was written to a durable queue before anything else happened to it. Now, events sit in memory for a short while. If an instance is killed without a clean shutdown (and Spot instances can disappear), whatever is in memory is lost.

On paper, it is a step down. Then, a few months after the rollout, we went through an episode of DynamoDB throttling, and the question came up: did the local queue retry the failed writes? It does not: the error is logged and the worker moves on. But neither did the Lambda function, since it was the same code. The SQS message was deleted after a logged failure, exactly like a local batch is now marked as done.

So SQS protected us if the Lambda function crashed, but not when a downstream service said no, which happens a lot more often. We gave up less durability than it seemed: a queue keeps the messages, but what happens when processing fails is up to the consumer.

There is still room for improvement here: retrying failed writes with a backoff, or simply sending the failed batches to SQS, since the fallback path is already there.

Wrapping up #

The patterns in this design are well known: bounded queues, batching, and a fallback to the previous system. Put together, they took 70 million messages a day off SQS, which saves around $35,000 a year at today’s traffic, and cost is no longer what decides the /activate limits. Keeping the old path in place also made each step of the rollout easy to undo, and it still catches the rare bursts today.