There are two kinds of engineering tasks.
The first one is the one your project manager imagines.
"Just optimize the report."
The second one is the one reality gives you.
You tighten one screw.
A cabinet falls off.
You move the cabinet.
There's mold.
You remove the drywall.
Now you're in the attic wondering why there's a canoe.
This is that story.
It Started With a CPU Spike
Not a crash.
Not an exception.
Not a stack trace screaming for attention.
Just...
Railway looking increasingly concerned.
High CPU.
Memory climbing.
Reports taking forever.
Operations had developed what I like to call a folk remedy.
"Restart the server."
And to make things worse...
It worked.
Nothing is more dangerous than a workaround that works just often enough to stop people asking questions.
The First Thing I Did Wasn't Optimize Anything
People love jumping straight to optimization.
I like knowing what I'm optimizing first.
So the first PR wasn't glamorous.
I added timeouts.
Maximum date range validation.
Better logging.
Timing around expensive operations.
Consistent log prefixes.
Usernames in report logs.
Basically...
Smoke detectors.
Not because they stop fires.
Because I'd like to know which room is burning.
For the first time, I could actually follow a report from the API all the way through Bull, Prisma, Elasticsearch, Excel generation, and back again.
Turns out...
That's useful.
Who knew?
Then the House Started Talking
Once the logs existed...
The house became very chatty.
I found analytics endpoints creating 300,000 Date objects for absolutely no reason.
The dates were already ISO strings.
We were converting strings into Date objects...
Just to extract the month...
Then immediately throwing the objects away.
Computers are remarkably obedient.
Ask them to do something unnecessary three hundred thousand times...
They'll do it without judging you.
I found report endpoints repeatedly asking Elasticsearch the same questions.
Added Redis caching.
Seventeen-second requests suddenly became...
Three milliseconds.
Bull jobs weren't cleaning up failed executions.
Redis slowly collected dead jobs like an attic collecting Christmas decorations nobody remembers buying.
That got cleaned too.
None of these fixes solved the problem.
They just made the house a little less haunted.
Then Someone Reported an Export Problem
This one sounded familiar.
"The export gets stuck."
No exceptions.
No obvious errors.
Just...
CPU climbing.
Memory climbing.
Restart the server.
Everything works again.
Someone pointed out that the customer was exporting around 1.2 million rows.
Reasonable conclusion.
That's a lot of rows.
Maybe that's just...
Heavy.
Except...
Something felt wrong.
Big exports should be slow.
They shouldn't look like Node.js was attempting to brute-force the meaning of life.
So...
Back to the logs.
The Most Expensive Pair of Braces I've Ever Seen
Eventually I found this.
{}That's all.
Just...
An empty object.
The export request had no filters.
No warehouse.
No status.
No dates.
Nothing.
Software is wonderfully literal.
Prisma looked at that payload and politely interpreted it as,
"Ah, you'd like everything."
Not everything this month.
Not everything matching your report.
Everything.
Every order.
Every relation.
Every object.
Every questionable business decision anyone had ever inserted into the database.
That explained the CPU.
That explained the memory.
That explained why restarting the server "fixed" it.
It wasn't fixing anything.
It was interrupting a process that was enthusiastically attempting to hydrate the known universe.
So I Went Looking for the Frontend
The logs told me what happened.
Now I wanted to know who sent {}.
That trail led straight to the frontend.
Eventually I found it.
A useEffect.
Of course it was a useEffect.
The report parameters were being constructed inside it.
Most of the time...
React finished updating state before the request fired.
Sometimes...
It didn't.
The export request left before the parameters existed.
Which meant the backend received...
{}
Not because anyone intended it.
Just because two pieces of asynchronous code politely disagreed about what "now" meant.
The fix was almost disappointingly simple.
Build the parameters inside the submit handler.
The payload now reflects exactly what the user submitted.
No race.
No stale state.
No hoping React had already finished whatever philosophical journey it was on.
Then I Found Another Way Into the Basement
Feeling pretty good about myself...
I checked the retry endpoint.
Because production has taught me one thing.
If there's one way into the basement...
There's probably another.
Turns out...
Legacy reports stored NULL parameters.
The retry endpoint quietly reconstructed them like this.
parameters ?? {}Well...
There's our old friend again.
{}Different bug.
Same apocalypse.
This time the fix wasn't moving code.
It was validation.
If the parameters are invalid...
Reject the request.
Return a 400.
Don't silently promote "missing data" into "please fetch the entire database."
The Funny Thing Is...
Even If It Worked...
Imagine it actually succeeded.
Imagine Prisma happily hydrated 1.2 million records.
Great.
Now what?
Now those objects get handed to Excel.
Node has to build an enormous workbook.
ZIP it.
Stream it.
Write it.
At that point the export isn't a report anymore.
It's a hostage negotiation between V8's garbage collector and ZIP compression.
The process wasn't slow because Excel was inefficient.
It was slow because we'd accidentally asked it to summarize civilization.
Since I Was Already Under the House...
By this point the actual production bug was fixed.
I could've stopped.
Closed the ticket.
Pretended none of this ever happened.
Instead...
I looked around.
If you've already ripped the drywall open...
You might as well replace the wiring.
So I started cleaning.
Progress updates stopped hammering Supabase fifty thousand times per export.
Date formatters became shared instead of recreated hundreds of thousands of times.
Excel row commits became batched.
Bull locks stopped expiring during long-running jobs.
Prisma stopped fetching data nobody ever looked at.
Failed jobs stopped accumulating in Redis forever.
Analytics stopped rebuilding the same answers over and over.
The report dialog stopped relying on a race condition disguised as a useEffect.
The retry endpoint stopped accepting "¯\(ツ)/¯" as valid report parameters.
None of these changes made for exciting release notes.
Nobody opens an app hoping to experience dramatically fewer Intl.DateTimeFormat allocations.
Nobody celebrates a generated SQL query that's now 18% leaner.
Nobody throws a party because Redis has fewer abandoned jobs.
And that's exactly the point.
The Best Updates Are the Ones Nobody Notices
Nobody emailed saying,
"Wow... exports only updated progress twenty-five times instead of fifty thousand."
Nobody noticed Bull quietly renewing job locks in the background.
Nobody applauded because the frontend stopped sending {} into production.
Nobody thanked the validation layer for refusing to weaponize NULL.
The users just...
Kept working.
Reports stopped mysteriously hanging.
Servers stopped sounding like they were rendering Pixar movies.
Operations stopped performing ritual server restarts like they were trying to appease ancient infrastructure gods.
That's what success usually looks like.
Not flashy dashboards.
Not conference talks.
Not a LinkedIn post about "10x engineering."
Just...
Silence.
No alerts.
No panicked messages.
No one asking if the server needs another reboot.
The best software updates aren't the ones users rave about.
They're the ones users never realize happened.
Because somewhere along the way, you quietly replaced the rotten beams, fixed the leaking pipes, chased the raccoons out of the attic, and rewired the house...
...before anyone else noticed the ceiling was about to fall in.