Most systems are sized against an average. Requests per second, spread across a day, with a comfortable multiplier on top. That reasoning is fine for products where demand is roughly continuous, and it is completely wrong for a whole class of systems where it is not.
Contest platforms. Ticket sales. Exam results. Tax portals. Course registration. Anything with a deadline attached.
These share a traffic shape: near-zero for weeks, then most of the total volume inside a window of an hour or two, then near-zero again. The average is a fiction. Nobody ever experiences it.
What the shape does to your assumptions
I worked on a photography contest platform where submissions closed at a fixed time. Photographers, being human, submitted at the last possible moment. And each submission was not a JSON body - it was a full-resolution image, three orders of magnitude larger than any other request the API handled.
So the peak had two dimensions compounding each other: many more requests, and each request enormously more expensive. A capacity plan built on request counts would have missed the second one entirely.
Three things break in that window, in this order:
Connections saturate before CPU does. Each upload holds a connection open for seconds or minutes on a slow domestic uplink. Concurrency is bounded by how many of those you can hold simultaneously, not by how fast you can process them. Your CPU graph will look calm during an outage.
Memory goes before disk. Anything that buffers a file in memory - a naive body parser, a middleware that reads a stream to inspect it - multiplies by concurrency. A hundred concurrent 30MB uploads is three gigabytes, and the OOM killer does not care why.
Timeouts cascade. Slow uploads occupy the proxy's worker pool. New requests queue at the proxy. Health checks queue with them, fail, and the orchestrator starts killing healthy instances for being unresponsive - during the exact window when you need every one of them.
That last one is the real teacher. The outage is rarely the load. It is your infrastructure's reaction to the load.
Move the bytes off the critical path
The single highest-leverage change: user file bytes must never travel through your application server.
Instead of accepting an upload and forwarding it to object storage, the API issues a pre-signed URL and gets out of the way. The client uploads directly to S3. The application handles a few hundred bytes of metadata per submission instead of thirty megabytes.
// The API's entire involvement in a 30MB upload.
app.post("/entries/upload-url", requireAuth, async (req, res) => {
const key = `contests/${req.body.contestId}/${crypto.randomUUID()}`;
const url = await getSignedUrl(
s3,
new PutObjectCommand({
Bucket: BUCKET,
Key: key,
ContentType: req.body.contentType,
}),
{ expiresIn: 900 },
);
res.json({ url, key });
});
The consequences go further than reduced bandwidth:
- Concurrency is now bounded by S3's limits, not your process's.
- A slow client cannot occupy an application worker.
- Upload failures do not consume application capacity to retry.
- Your instances scale on metadata volume, which is small and predictable.
The client then confirms with the returned key, and a background job validates that the object actually exists and matches expectations. Confirming without verifying means trusting a client to tell you what it uploaded, which is not a position you want to be in when there is prize money attached.
Everything third-party belongs in a queue
The second failure mode is quieter. Submission confirmation emails go out in bursts, one per entry, into the same two-hour spike.
Sending inline ties your request latency to a mail provider's worst day. Worse, providers rate-limit - so at exactly the moment your traffic peaks, your email calls start failing, and if that failure propagates into the request the user sees a submission error for a submission that succeeded.
Queueing decouples all of it. On that platform, agenda handled scheduling on
top of Mongo, which was already there:
agenda.define("send-entry-confirmation", { concurrency: 5 }, async (job) => {
const { entryId } = job.attrs.data;
const entry = await entries.findById(entryId);
if (!entry || entry.confirmationSentAt) return; // idempotent
await mailer.send(buildConfirmation(entry));
await entries.updateOne({ _id: entryId }, { $set: { confirmationSentAt: new Date() } });
});
// In the request path - this is all the user waits for.
await agenda.now("send-entry-confirmation", { entryId: entry._id });
concurrency: 5 is doing real work there. It is a deliberate ceiling that keeps
you inside the provider's rate limit no matter how sharp the spike is. The queue
absorbs the burst and drains at a rate the downstream can survive. Users get
their email a minute late instead of not at all.
Make the read path cheap
Traffic before a deadline is not only writes. Entrants browse galleries, check their own submissions, refresh the leaderboard. That read volume rides the same spike.
Most of it is identical requests for identical data. On the client, cache-aware fetching - RTK Query in this case - collapses repeated views into one request. On the server, gallery listings are cacheable for a short window; a thirty-second stale window is imperceptible to a user and removes most of the query load.
The cheapest request remains the one you never make.
Test the shape, not the average
If your load test ramps smoothly to a target RPS and holds, it is testing a system you do not have. The spike is a step function, and step functions expose things ramps do not: cold connection pools, autoscaling lag, cache stampedes when everything expires simultaneously.
Three things worth deliberately checking:
- Step, don't ramp. Go from idle to peak in seconds. Autoscaling that reacts in three minutes is decorative against a two-hour event.
- Use realistic payloads. A load test with 2KB bodies proves nothing about a system whose problem is 30MB bodies.
- Fail a dependency during peak. Turn off the mail provider mid-test. If a submission fails because a confirmation email could not send, the coupling is still there and you have not actually queued anything.
The general rule
Deadline systems reward a specific instinct: the request path should do as little as it possibly can, and everything else should be something that happens afterwards.
Bytes go to object storage. Third-party calls go to a queue. Reads come from cache. What remains in the request is a small, fast, predictable amount of work - and predictable is the entire point, because on the day, you do not get to intervene. You find out whether the design was right by watching it.