Exercise 02 of 4/Foundations/1 hour

The bank that freezes itself

The same bank, now with one lock per account so unrelated transfers stop queueing behind each other. It is faster, correct under every test you wrote last time, and it will stop dead the first time two customers pay each other at once.

Language

At the end of the last exercise the bank was correct and slightly embarrassing: one global mutex, so a transfer between two accounts in Mumbai blocks a transfer between two accounts in Lisbon. With a million accounts that is not a bank, it is a queue.

The obvious fix is to give each account its own lock. That instinct is right, it is how real systems are built, and it will deadlock within seconds.

Give each account its own lock

Do this

Rewrite transfer so it locks only the two accounts involved, rather than the whole bank. Then run exercise 01's one-directional test against it — 1,000 threads all moving money from alice to bob — and confirm it passes.

public final class Account {
    final ReentrantLock lock = new ReentrantLock();
    long balance;
}
 
public final class Bank {
    private final Map<String, Account> accounts;
 
    public void transfer(String from, String to, long amount) {
        Account src = accounts.get(from), dst = accounts.get(to);
 
        src.lock.lock();
        try {
            dst.lock.lock();
            try {
                if (src.balance < amount) throw new IllegalStateException("insufficient funds");
                src.balance -= amount;
                dst.balance += amount;
            } finally { dst.lock.unlock(); }
        } finally { dst.lock.unlock(); }
    }
}

Read those two finally blocks before you go looking for the deadlock. There is a bug on this page that fires on the first transfer, long before any concurrency is involved.

type Account struct {
	mu      sync.Mutex
	balance int64
}
 
type Bank struct {
	accounts map[string]*Account
}
 
func (b *Bank) Transfer(from, to string, amount int64) error {
	src, dst := b.accounts[from], b.accounts[to]
 
	src.mu.Lock()
	defer src.mu.Unlock()
	dst.mu.Lock()
	defer dst.mu.Unlock()
 
	if src.balance < amount {
		return errors.New("insufficient funds")
	}
	src.balance -= amount
	dst.balance += amount
	return nil
}
class Account:
    def __init__(self, balance: int) -> None:
        self.lock = threading.Lock()
        self.balance = balance
 
class Bank:
    def __init__(self, accounts: dict[str, Account]) -> None:
        self._accounts = accounts
 
    def transfer(self, from_id: str, to_id: str, amount: int) -> None:
        src, dst = self._accounts[from_id], self._accounts[to_id]
 
        with src.lock:
            with dst.lock:
                if src.balance < amount:
                    raise ValueError("insufficient funds")
                src.balance -= amount
                dst.balance += amount

That test passes cleanly, every time. Have you fixed anything?

AnswerCommit to one first

You have fixed the contention, and you have introduced two bugs the test cannot see.

A one-directional test only ever locks the accounts in one order, so the deadlock this design makes possible never gets a chance to happen. And the second bug does not need concurrency at all — it fires on a single transfer, on one thread.

Run one transfer, on one thread

Do this

Before any of the concurrency work: call transfer exactly once, on the main thread, and watch what happens.

Run one transfer. Not a thousand from fifty threads — one, on the main thread. Here is what actually happens, from a real run:

THREW IllegalMonitorStateException
alice=900 bob=1100        <- the money moved
lower-id account locked = true
second transfer, other thread: still alive after 3s = true

Three distinct harms from one wrong identifier, and they get worse as you go down:

src is never released. The outer finally unlocks dst a second time instead of unlocking src, so src stays held by a thread that has already returned. Every future transfer touching that account blocks forever. One call permanently poisons an account, and nothing in the logs points at the call that did it.

dst is unlocked twice. ReentrantLock.unlock() from a thread that does not hold the lock throws IllegalMonitorStateException. Which is, at least, loud.

The exception is thrown after the money has already moved. This is the one to sit with. The debit and credit completed inside the inner try; the failure happens on the way out. So the caller sees an exception from a transfer that succeeded — and a caller that catches it and retries has now moved the money twice.

That last property is the worst shape a bug can take in a ledger: it looks like a failure and was a success. Any retry logic you have makes it worse rather than better.

One transfer, step by step

alice1000bob1000src.lockfreedst.lockfree
transfer() on one thread
src.lock.lock();
dst.lock.lock();
src.balance -= 100; dst.balance += 100;
} finally { dst.lock.unlock(); } // inner
} finally { dst.lock.unlock(); } // outerIllegalMonitorStateException

 

No second thread. No deadlock. One typo.

The fix is one identifier. The habit that prevents it is structural: every finally releases the lock its own try acquired, and nothing else. Nesting the blocks is what makes that checkable by eye, which is why the solution below nests rather than flattens.

Now make it freeze

Do this

Write the test that deadlocks it: goroutines transferring alice to bob and bob to alice at the same time. Make the freeze reliable, not occasional.

An intermittent deadlock is nearly useless to learn from, because you cannot tell a fix from luck. Widen the window on purpose, between taking the first lock and the second:

src.lock.lock();
Thread.sleep(1);        // remove once the bug is understood
dst.lock.lock();
src.mu.Lock()
time.Sleep(time.Millisecond) // remove once the bug is understood
dst.mu.Lock()
src.lock.acquire()
time.sleep(0.001)       # remove once the bug is understood
dst.lock.acquire()

Now the failure is close to deterministic. Run it, and the test hangs forever.

If every goroutine in the process is blocked, Go will tell you outright:

fatal error: all goroutines are asleep - deadlock!

Nothing is printed — the JVM does not volunteer this. But unlike the other two it will tell you if you ask: jstack <pid> or jcmd <pid> Thread.print ends with "Found one Java-level deadlock" and names both threads and both monitors.

Nothing is printed, and there is no equivalent of jstack. The process sits there looking healthy and idle. You need faulthandler.dump_traceback_later(10, exit=False) armed in advance, or py-spy dump --pid <pid> attached from outside.

Go's detector only fires when nothing can proceed. A real server has a health-check goroutine and an idle HTTP listener, so the runtime stays happy while your transfer handler is permanently wedged. In production this bug does not announce itself; it appears as a handler whose latency graph goes vertical. Learn to reach for SIGQUIT, or curl /debug/pprof/goroutine?debug=2, and read the stacks.

Work out why

Two threads, two locks, four lock operations. Write out the interleaving that freezes.

AnswerCommit to one first

A holds alice and wants bob. B holds bob and wants alice. Neither will release what it holds, so neither can proceed.

The bug is not that two locks are held. It is that different threads acquire the same two locks in different orders.

Hint 1Draw the two goroutines side by side

Goroutine A is transferring alice → bob. Goroutine B is transferring bob → alice. Write out the four lock operations in the order they can interleave, and you will see it.

Hint 2The condition has a name and four parts

This is a textbook hold-and-wait cycle: A holds alice and wants bob; B holds bob and wants alice. Neither will release what it holds. Breaking any of the four Coffman conditions is enough to fix it — which is why there is more than one valid answer below.

Fix it so a cycle cannot form

Do this

Make deadlock impossible for any number of concurrent transfers between any accounts — while keeping two unrelated transfers running in parallel. Then add a test for transfer(alice, alice, 100).

Solution — Java: the JVM will name the cycle for youTry it first
public final class Account {
    final String id;                 // immutable, unique — the ordering key
    private final ReentrantLock lock = new ReentrantLock();
    private long balance;
 
    Account(String id, long balance) { this.id = id; this.balance = balance; }
 
    ReentrantLock lock() { return lock; }
    long balance()       { return balance; }
    void add(long d)     { balance += d; }   // callers hold the lock
}
 
public final class Bank {
    private final Map<String, Account> accounts;
 
    public void transfer(String from, String to, long amount) {
        Account src = accounts.get(from), dst = accounts.get(to);
        if (src == null || dst == null) throw new NoSuchElementException();
        if (src == dst) return;
 
        Account first = src, second = dst;
        if (first.id.compareTo(second.id) > 0) { first = dst; second = src; }
 
        first.lock().lock();
        try {
            second.lock().lock();
            try {
                if (src.balance() < amount) throw new IllegalStateException("insufficient funds");
                src.add(-amount);
                dst.add(amount);
            } finally { second.lock().unlock(); }
        } finally { first.lock().unlock(); }
    }
}

The ordering rule is identical. Three things about Java are not.

The JVM finds this bug for you, and Go does not. Take a thread dump of the deadlocked program — jstack <pid>, or jcmd <pid> Thread.print — and the output ends with:

Found one Java-level deadlock:
=============================
"pool-1-thread-3":
  waiting to lock monitor 0x00007f... (object 0x...,  a Account),
  which is held by "pool-1-thread-7"
"pool-1-thread-7":
  waiting to lock monitor 0x00007f... (object 0x...,  a Account),
  which is held by "pool-1-thread-3"

That is the cycle, named, with both threads and both objects. ThreadMXBean.findDeadlockedThreads() gives you the same thing programmatically, so you can assert on it in a test or alert on it in production. This works for synchronized monitors and for ReentrantLock. It is one of the genuinely nicer corners of the platform, and it is the reason a Java developer often has an easier time with this specific bug than a Go developer.

synchronized is reentrant, so the self-transfer bug quietly disappears. In Go, Transfer("alice", "alice", 0) locks alice twice and hangs forever. In Java, a thread already holding a monitor can re-enter it, so the same code completes — with a subtle wrongness instead of a hang: the balance is read, decremented, and incremented under one lock acquisition, and the arithmetic happens to cancel out. You do not get punished, which means you do not learn, which means the guard is missing when you later port the logic somewhere non-reentrant. Keep the src == dst check even though Java does not force you to.

try/finally is not optional, and neither is nesting it correctly. Go's defer runs on panic and pairs each release with its acquisition automatically. Java's lock() has no such affordance. Note the nesting above — the inner lock() is inside the outer try, so a failure to acquire the second lock still releases the first, and each finally mentions exactly the lock its own try took. The starter code at the top of this page unlocks second in both blocks, which is how the step-zero bug happens: a permanently held lock, plus an exception raised after the money moved.

Solution — Go: order the locks, do not just take themTry it first

The bug is not that two locks are held. It is that different goroutines acquire the same two locks in different orders. Fix the order and the cycle cannot form.

Give every account a stable, unique identity and always lock the lower one first:

type Account struct {
	id      string // immutable, unique
	mu      sync.Mutex
	balance int64
}
 
func (b *Bank) Transfer(from, to string, amount int64) error {
	src, dst := b.accounts[from], b.accounts[to]
	if src == nil || dst == nil {
		return errors.New("no such account")
	}
	// Self-transfer: one lock, or we deadlock against ourselves.
	if src == dst {
		return nil
	}
 
	first, second := src, dst
	if first.id > second.id {
		first, second = second, first
	}
 
	first.mu.Lock()
	defer first.mu.Unlock()
	second.mu.Lock()
	defer second.mu.Unlock()
 
	if src.balance < amount {
		return errors.New("insufficient funds")
	}
	src.balance -= amount
	dst.balance += amount
	return nil
}

Why this works, stated precisely: if every goroutine acquires locks in ascending id order, then a goroutine holding lock X only ever waits for a lock greater than X. A cycle would require some goroutine to wait for a lock smaller than one it holds, which no goroutine ever does. The cycle is not unlikely — it is unconstructible.

The self-transfer case is not a detail. sync.Mutex is not reentrant. Transfer("alice", "alice", 0) on the original code locks alice, then blocks forever waiting for alice — a single goroutine deadlocking against itself. Every lock-ordering scheme needs this guard, and it is the case test suites forget.

Order by identity, never by address. uintptr(unsafe.Pointer(src)) is a tempting one-liner and it is sound for ordering today, but it encodes an assumption about a non-moving garbage collector that is not part of Go's contract. Use a field you control.

The two other fixes, and why they are worse

Try-lock with backoff. Take the first lock, TryLock the second, and if it fails, release the first, back off, retry. This breaks hold-and-wait rather than circular-wait, and it does work. But it converts a correctness property into a performance property: under contention you now have livelock risk and unbounded retries, and the tail latency is nobody's friend. Reach for it only when a stable ordering genuinely is not available.

Go back to one global lock. Correct, trivial, and the thing you were trying to escape. Worth measuring anyway — for a few thousand accounts with short critical sections, the global mutex often wins, because uncontended mutexes are cheap and you have removed all the ordering logic. "Two locks" is a decision that should be earned with a benchmark, not assumed.

Solution — Python: the GIL does nothing for you hereTry it first
import threading
 
class Account:
    def __init__(self, id: str, balance: int) -> None:
        self.id = id                      # immutable, unique
        self.lock = threading.Lock()
        self.balance = balance
 
class Bank:
    def __init__(self, accounts: dict[str, Account]) -> None:
        self._accounts = accounts
 
    def transfer(self, from_id: str, to_id: str, amount: int) -> None:
        src, dst = self._accounts[from_id], self._accounts[to_id]
        if src is dst:
            return
 
        first, second = (src, dst) if src.id < dst.id else (dst, src)
 
        with first.lock:
            with second.lock:
                if src.balance < amount:
                    raise ValueError("insufficient funds")
                src.balance -= amount
                dst.balance += amount

Python's version is the shortest and the worst-behaved when it breaks.

This is the exercise that proves the GIL is not a concurrency strategy. The previous exercise's lost update at least involved the GIL — bytecode granularity was what hid it. Deadlock does not care about the GIL at all. A thread blocked on lock.acquire() releases the GIL while it waits, which is what lets the other thread run and take the second lock. The global interpreter lock neither causes nor prevents this; it is simply irrelevant, and a reader who has been told "Python threading is safe" has no model for why.

Nothing will tell you it happened. There is no jstack, and no equivalent of Go's all goroutines are asleep message — the main thread sits in join() and the process looks alive and idle. Your options are faulthandler.dump_traceback_later(10, exit=False) armed in advance, sending SIGABRT with faulthandler.enable(), or attaching py-spy dump --pid <pid> from outside. Learn py-spy now rather than during an incident; it needs no cooperation from the process.

with is the right tool and the nesting is load-bearing. The nested with blocks release in reverse order on any exception, which is what Go's stacked defers and Java's nested try/finally also achieve. Resist flattening to with first.lock, second.lock: if you ever need a timeout — Lock.acquire(timeout=...) does not compose into the with form, and exercise 07's shutdown work will want it.

threading.Lock is not reentrant, so Python punishes the self-transfer exactly like Go. RLock exists; reach for it deliberately, not to make a hang go away.

Check the edges

Why does transfer(alice, alice, 100) hang in Go and Python but complete in Java?

AnswerCommit to one first

sync.Mutex and threading.Lock are not reentrant: a thread that already holds the lock and tries to take it again waits for itself, forever.

Java's synchronized and ReentrantLock are reentrant, so the same code completes — with a quiet wrongness rather than a hang, since the balance is read, decremented and incremented under one acquisition.

Java not punishing you is the problem. You never learn the guard is needed, and it is missing when you port the logic somewhere non-reentrant.

Your server hangs in production. Which runtime tells you it deadlocked?

AnswerCommit to one first

Only Java, and only if you ask.

  • Javajstack <pid> prints "Found one Java-level deadlock" and names both threads and both monitors. ThreadMXBean.findDeadlockedThreads() exposes the same to code, so you can alert on it.
  • Gofatal error: all goroutines are asleep fires only when every goroutine is blocked. A real server has a health-check goroutine, so it never trips. You read stacks from SIGQUIT or /debug/pprof/goroutine?debug=2.
  • Python — nothing at all. The process looks healthy and idle. You need faulthandler armed in advance, or py-spy dump attached from outside.

Deadlock in three languages: who tells you?

Same bug, same fix. What changes is whether the runtime is on your side.

Java

A real detector, and reentrancy that hides one bug. jstack prints "Found one Java-level deadlock" with both threads and both monitors, and ThreadMXBean.findDeadlockedThreads() exposes the same to code — you can alert on it. Against that, synchronized and ReentrantLock are reentrant, so the self-transfer case completes instead of hanging and you never find out it was wrong. And lock() without try/finally turns any exception into a permanently held lock.

Go

Half a detector, and no reentrancy. fatal error: all goroutines are asleep - deadlock! fires only when every goroutine is blocked, so a real server with a health-check goroutine never trips it — the bug shows up as one handler's latency going vertical. Diagnosis is SIGQUIT or /debug/pprof/goroutine?debug=2 and reading stacks yourself. sync.Mutex is not reentrant, so the self-transfer hangs immediately, which at least teaches you the lesson early.

Python

No detector at all, and the GIL is beside the point. The process hangs looking healthy; there is no built-in dump. You need faulthandler armed ahead of time or py-spy attached from outside. Worth sitting with: a blocked thread releases the GIL, which is exactly what allows the cycle to form. Every intuition of the form "the GIL protects me" is silent on this entire class of bug.

The transferable lesson. Lock ordering is a property of your code, not of your language — none of the three offers a way to declare it, and all three let you violate it in a function three commits away. What differs is only the length of the debugging session afterwards. If you take one habit from this exercise, make it this: decide the total order once, write it down next to the lock, and treat any Lock() not taken in that order as a bug in review, whichever language you are in.

Before you move on

You should be able to explain each of these without looking.

  • The exact interleaving of four lock operations that produces the freeze.
  • Why a stable total order over locks makes a cycle unconstructible, not just rare.
  • Why Go's all goroutines are asleep detector will not fire for this bug in a real server, and what you would look at instead.
  • What Transfer("alice", "alice", 100) does on the unguarded version, and why.
  • Why the same self-transfer completes in Java and hangs in Go and Python.
  • Which runtime names the deadlock cycle for you, and what you run in the other two.
  • Why the GIL is irrelevant to this bug, unlike the previous one.

Go further

  • Extend to a three-account atomic transfer (A → B → C settled together). Confirm the ordering rule needs no change — this is the payoff of ordering by identity rather than special-casing pairs.
  • Add a Total() that is consistent across all accounts. You now need every lock at once, in order, and you should think hard about whether you want that method to exist at all in a bank with a million accounts.
  • Remove the time.Sleep and run the broken version under -race with -count=1000. It will mostly pass. Sit with what that means about tests as evidence for concurrency bugs.
  • Introduce a deliberate lock-order violation in a second method (Withdraw that locks destination first), and see how a bug three commits away breaks a function that was correct.