Your Capture Awards is a contest platform for photographers. Entries, judging, galleries, results. On paper it is CRUD with images attached.
The thing that makes it an engineering problem is a single fact: a contest has a deadline, and photographers submit against it. Weeks of near-zero traffic, then most of the total volume inside a two-hour window, at full-resolution image sizes, on a schedule that cannot move.
I built the backend, the storage pipeline and the deployment.
Why the average is useless here
Size this system on requests per second averaged over a week and you will provision something that falls over on the only day it matters.
The peak has two dimensions compounding. Request volume multiplies, and each request is three orders of magnitude larger than a normal API call. A capacity plan built on request counts alone misses the second dimension entirely and gives you a number that looks reassuring right up until submissions open.
What actually breaks, in order:
Connections saturate before CPU. A 30MB upload on a domestic uplink holds a connection open for minutes. Concurrency is bounded by how many of those you can hold at once, not by processing speed. Your CPU graph stays calm through the whole outage, which is a genuinely disorienting thing to watch.
Memory goes before disk. Anything buffering a file in memory multiplies by concurrency. A hundred concurrent 30MB uploads is three gigabytes.
Timeouts cascade. Slow uploads occupy the proxy's worker pool. Health checks queue behind them, fail, and the platform starts restarting healthy instances during the exact window you need all of them.
That last one is the important one. The outage is rarely the load itself. It is the infrastructure's reaction to the load.
The decision that made the rest tractable
User file bytes never travel through the application server.
The API issues a pre-signed S3 URL and steps out of the path. The client uploads directly to object storage. The application handles a few hundred bytes of metadata per entry instead of thirty megabytes.
app.post("/entries/upload-url", requireAuth, async (req, res) => {
const key = `contests/${req.body.contestId}/${crypto.randomUUID()}`;
const url = await getSignedUrl(
s3,
new PutObjectCommand({
Bucket: BUCKET,
Key: key,
ContentType: req.body.contentType,
}),
{ expiresIn: 900 },
);
res.json({ url, key });
});
The knock-on effects are larger than the bandwidth saving:
- Upload concurrency is bounded by S3, which is built for it, rather than by Node processes, which are not.
- A slow client cannot occupy an application worker.
- Failed uploads consume no application capacity when retried.
- Instances now scale on metadata volume - small, cheap, predictable.
The client confirms with the returned key afterwards, and a background job verifies the object exists and matches the declared content type and size limits. Trusting a client's claim about what it uploaded is not a position to be in when there is prize money attached.
Email that cannot block a request
Confirmations, judging notices and results all go out in bursts, into the same spike.
Sending inline ties request latency to a mail provider's worst day. Worse, providers rate-limit - so precisely when traffic peaks, mail calls start failing, and if that failure propagates into the request handler, a photographer sees a submission error for a submission that succeeded. That is the failure that generates support tickets and mistrust.
agenda handled scheduling on top of the Mongo instance already running:
agenda.define("send-entry-confirmation", { concurrency: 5 }, async (job) => {
const { entryId } = job.attrs.data;
const entry = await entries.findById(entryId);
if (!entry || entry.confirmationSentAt) return; // idempotent by design
await mailer.send(buildConfirmation(entry));
await entries.updateOne(
{ _id: entryId },
{ $set: { confirmationSentAt: new Date() } },
);
});
Two details carry the weight. concurrency: 5 is a deliberate ceiling that keeps
throughput inside the provider's rate limit regardless of how sharp the spike is
- the queue absorbs the burst and drains at a rate the downstream survives. And
the
confirmationSentAtguard makes the job idempotent, so a retry after a partial failure cannot double-send.
Choosing Agenda over a dedicated broker was a scope decision. It uses the database already in the stack, needs no new infrastructure to operate, and is entirely adequate at this volume. At higher throughput I would want something purpose-built, with real dead-letter handling. Adding a broker to this system would have been infrastructure nobody was staffed to run.
Reads ride the same spike
Traffic before a deadline is not only writes. Entrants browse galleries, check their own entries, refresh standings - and most of it is identical requests for identical data.
RTK Query on the Next.js client collapses repeated views into a single request with cache-aware fetching. Gallery listings tolerate a short server-side stale window; thirty seconds is imperceptible to a user and removes most of the query load at peak.
The cheapest request is still the one never made.
Real-time, used sparingly
Socket.IO pushes judging and contest updates rather than having thousands of clients poll during the window when the system is least able to absorb polling.
Same two rules as everywhere else: persist before emitting, so the database stays the source of truth; and reconcile on reconnect, so a dropped connection is a latency problem rather than a hole in someone's view of the contest.
Deployment
GitHub Actions builds and deploys to DigitalOcean, behind Nginx as reverse proxy. Domain via Namecheap.
Making deploys a pipeline rather than an SSH session mattered most in the days before a contest closed, when small fixes go out under time pressure. A repeatable deploy is a deploy that a nervous person can run correctly.
Nginx also carries real load here - TLS termination and static serving are work the Node processes should never be doing during a spike.
What I would watch, knowing what I know now
Standard dashboards are close to useless for this shape. Averaged over an hour, the spike disappears.
The signals worth having:
- Pre-signed URLs issued versus objects confirmed. A widening gap is the earliest indicator that client-side uploads are failing - invisible from server metrics because the failing requests never reach the server.
- Queue depth and oldest job age. Depth alone is ambiguous during a burst. Age tells you whether it is draining.
- p99 upload-URL latency, not p50. The tail is where a spike shows up first.
What I would do differently
Verify uploads harder, sooner. The background verification job was good. Enforcing size and type constraints in the pre-signed policy itself would have been better - rejecting at the storage layer beats detecting afterwards.
Rehearse the spike. We reasoned about the traffic shape correctly and designed for it, but never staged a step-function load test. Deliberately going idle-to-peak in seconds would have told us what autoscaling actually did, rather than what we assumed.
Explicit dead-letter handling. Agenda's retries were adequate. A jobs graveyard that a human reviews after each contest would have turned "probably fine" into "confirmed fine".
The transferable part
Deadline-shaped systems all reward the same instinct: the request path should do as little as it possibly can.
Bytes to object storage. Third-party calls to a queue. Reads from cache. What remains is small, fast and predictable - which is the entire goal, because on the day, nobody gets to intervene. You watch, and you find out whether the design was right.