Two coaches, one interview, and a race nobody could see

This was built at Uravo, as part of the engineering team. The job was to remove a coordination problem: someone submits a form asking for an interview, and a coach has to pick it up. That was happening by hand, over messages, with the usual result that requests sat around waiting for somebody to notice them.

The pipeline itself is not complicated.

HubSpot form  ->  SQS  ->  Lambda  ->  Slack message with a button
                                            |
                            coach taps  ->  webhook  ->  AWS  ->  booked

A form submission enters a queue. A Lambda picks it up and posts an actionable message into Slack. A coach taps the button, the interaction comes back into AWS, and the booking completes.

The queue is there because HubSpot's side and our side should not be coupled to each other's availability. If the Lambda is failing, submissions wait in SQS instead of disappearing.

That part worked on the first try. The part that did not was subtler.

The bug that was invisible

Two coaches see the same message. Both want it. Both tap within a second or two of each other.

Both got a success response.

Only one booking existed. The other coach believed they had claimed an interview that was not theirs, and would find out later, or not at all. There was nothing in the logs that read as an error, because from each request's point of view nothing had gone wrong. Each one checked whether the request was claimed, found that it was not, and claimed it. Both were telling the truth at the moment they looked.

This is the classic shape of a check-then-act race, and it is nasty in a human-facing system specifically because it produces no error anywhere. It produces two people who are confident about incompatible things, and a disagreement that surfaces days later in a calendar.

The fix

The claim moved into DynamoDB as a conditional write: create this claim only if no claim already exists for this submission.

The condition is evaluated by the database at write time, not by application code beforehand. The first request succeeds. The second fails the condition and comes back as a rejection, which the handler turns into a message telling that coach the interview has already been taken and who has it.

Nothing about the traffic changed. What changed is that the check and the act became one operation instead of two, so there is no window between them for another request to slip through.

Why not a lock

Locking was the first instinct and it was the wrong one. A lock introduces questions the problem does not need answered: how long is it held, what happens if the holder dies, who cleans up. All of that exists to protect a window that does not have to exist in the first place.

A conditional write does not create the window. That is a better fix than guarding it well.

The part that mattered to the people using it

The engineering result was that a submission can be claimed exactly once. The result the coaches noticed was different: they could see who took what.

Because claims are now recorded rather than raced for, the order is known and the outcome is visible. Most of the confusion the race had been causing was not really about double bookings. It was about nobody being able to reconstruct what had happened after the fact.

That is a pattern I have hit more than once since. A concurrency bug rarely announces itself as a concurrency bug. It shows up as people disagreeing about what the system did, and the fix is usually to make the system record a decision rather than infer one.

What I took from it

Any time application code reads a value, makes a decision, and then writes based on that decision, there is a gap. Most of the time nothing gets into it. The question is what happens the one time something does, and whether you would ever find out.

If the answer is that two users would each be told yes and nobody would see an error, that gap needs closing at the storage layer, where the check and the write can be the same operation.