Debug Software Systematically

Debugging is not a contest to guess the bug fastest. It is the disciplined process of turning an unclear symptom into evidence, then using that evidence to eliminate possible causes.

Imagine a customer says: “Sometimes, after paying, I see two orders.” A rushed fix might disable the button after the first click. That may improve the symptom while leaving retries, network timeouts, or concurrent requests able to create duplicates. The goal is to understand the failure mode, not make the screen look calmer.

The debugging loop

Observe → Reproduce → Localize → Hypothesize → Test
   ↑                                         ↓
   └──────────── Verify the fix ←────────────┘

Write the steps down. Debugging becomes much less mysterious when the process is visible.

Step 1: Describe the symptom precisely

Replace “checkout is broken” with a statement you could prove or disprove:

When two equivalent checkout requests arrive within a short interval, the service can persist two order records for the same intended purchase.

A useful bug report includes:

• Expected behaviour.
• Actual behaviour.
• Steps to reproduce, if known.
• Frequency and relevant conditions.
• Environment, version, and time window.
• Relevant logs, request IDs, screenshots, or test output—with secrets and personal data removed.

Keep the original observation separate from your explanation. “The API returned 500” is an observation. “The database is broken” is a hypothesis.

Step 2: Reproduce before editing

A reproduction is a small experiment that reliably triggers the failure. Start with the shortest path you can find. Use a local environment or safe test data whenever possible; do not repeatedly trigger a suspected production failure just to gather evidence.

If you cannot reproduce it, shrink the uncertainty:

• Does it happen with one user or many?
• Does it depend on timing, data shape, permissions, or environment?
• Is it deterministic or intermittent?
• Did a recent deploy, configuration change, or dependency update coincide with it?

For intermittent problems, capture enough context to compare successful and failed attempts. Request IDs, timestamps, structured events, and correlation across service boundaries may reveal a pattern that a plain error message cannot.

Step 3: Find the boundary where reality diverges

Think of a request as a sequence of transformations. At each boundary, compare the expected input and output.

For checkout, inspect:

1. The browser’s submitted payload.
2. The API’s parsed and validated request.
3. The service’s business-rule decisions.
4. The database operation and its constraints.
5. The response the client receives.

This is often more effective than staring at a long stack trace from top to bottom. The stack trace tells you where an error surfaced; it does not always identify the original cause.

Use a debugger, logs, breakpoints, queries against safe test data, or a focused test. Prefer temporary, targeted instrumentation over adding a dozen noisy log statements. Avoid logging tokens, passwords, session cookies, or unnecessary personal data.

Step 4: State one testable hypothesis

A good hypothesis predicts an observation.

Weak:

Maybe checkout has some race condition.

Better:

The handler creates an order before it checks whether the request has already been processed. If two equivalent requests run concurrently, both may pass the check and create a record.

Now choose an experiment that distinguishes this explanation from alternatives. Inspect transaction boundaries. Write a test that submits equivalent requests in the relevant sequence. Check whether the database enforces the required uniqueness invariant.

Change one variable at a time when possible. If you alter the handler, retry policy, database schema, and UI simultaneously, it becomes difficult to know what actually fixed the behaviour.

Step 5: Fix the cause at the right boundary

A common principle in reliable systems is to enforce important invariants at a boundary that can actually guarantee them.

For duplicate orders, disabling a button may improve user experience, but it cannot guarantee correctness under retries or concurrent requests. Depending on the system, a robust design might combine an idempotency key, an atomic database constraint or transaction, and client-side feedback. The correct design depends on how orders, payments, and retries are represented in that application.

Do not copy a solution from a different system without understanding the invariant you need to preserve.

Step 6: Write the regression test

A regression test should fail for the old behaviour and pass for the corrected behaviour. It should assert the business outcome, not an implementation detail that can change harmlessly.

For example, a test might express:

Given one intended purchase and the same idempotency key,
when the request is submitted twice,
then only one order is created,
and both responses represent the same outcome.

This is a specification, not a drop-in test for every stack. Adapt it to the product’s actual contract and distinguish duplicate requests from two separate purchases made intentionally by the customer.

Run the test against the old code if practical to prove it detects the bug. Then run the focused suite, relevant broader tests, and the application’s normal checks.

Step 7: Explain the fix and remaining risk

Your pull request should state the symptom, root cause, fix, tests, and risks or limitations. If the fix depends on a database migration or rollout order, include that. If you could not reproduce the production failure and used a simulated case, say so.

A debugging session is not complete merely because the error disappeared once. You should know what evidence supports your explanation and what remains uncertain.

Debugging with AI

AI assistants can suggest likely causes, explain stack traces, draft tests, and compare possible fixes. They can also latch onto the first plausible explanation.

Ask for competing hypotheses, not just one answer:

Given this symptom and these sanitized logs, list three plausible causes. For each, tell me what observation would support or rule it out. Do not propose a code change until we identify a discriminating experiment.

Then run the experiment. The test output and runtime evidence are the authority—not the explanation that sounds most sophisticated.

A compact debugging checklist

• I can state expected versus actual behaviour.
• I have a safe, minimal reproduction—or documented why I cannot reproduce it.
• I have located the boundary where actual behaviour diverges.
• My hypothesis predicts a testable observation.
• I changed as little as possible to test the hypothesis.
• A regression test captures the intended behaviour.
• I ran the relevant checks and reviewed the diff.
• The fix and remaining risks are documented.

The habit that compounds

The fastest debugger is often not the person with the most clever guesses. It is the person who wastes the fewest experiments.

Observe carefully. Make uncertainty explicit. Run the smallest experiment that teaches you something.