← Back to the notebook

A customer once reported that their website had gone down a few hours after a code deployment. The delay mattered: the problem was not simply a deployment command failing. Something in the application’s behavior after the release needed a closer look.

My first concern was the customer’s immediate impact. I asked them to revert to the last stable code version so they could bring the site back while I investigated. A rollback was a recovery step; it did not, by itself, explain the failure.

Start with a timeline, not a conclusion

I reviewed the deployment logs, application behavior, and available metrics. I wanted to understand the relationship between the release and the later outage, rather than assume that timing alone proved the cause.

The investigation pointed to a function in the new code being invoked repeatedly. PHP processes were increasing and continuing to run instead of finishing normally. That behavior was consistent with work accumulating until the application could no longer serve requests normally.

The useful distinction was between the visible symptom—the site being unavailable—and the application behavior contributing to it. Simply adding capacity would not explain why that work was failing to complete.

Make the findings useful to the developers

Once I had identified the problematic behavior, I explained it to the customer’s development team by email and on a Zoom call. I walked them through what I had observed and recommended a workaround.

Written communication gave the team something they could refer back to. The call gave us space to discuss the behavior and clarify questions. Both were part of the technical work: an investigation is only useful if the people responsible for the next change can understand it.

I avoid describing the exact code path or workaround here because this is an anonymized account, not a reproduction of the customer’s application. I also would not label the mechanism as recursion, a memory leak, or a particular PHP configuration problem without the evidence needed to establish that distinction.

Close the loop with an RCA

The customer requested a root-cause analysis, which I prepared for them. An RCA gives an incident a lasting explanation beyond the immediate support conversation.

For this kind of report, I aim to separate the sequence of events, observations, explanation, mitigation, and recommendations. It should be clear which statements are supported by evidence and which improvements still need validation.

What I carry into the next case

This experience reinforced three habits: prioritize recovery, investigate beyond the first symptom, and communicate the findings in a way that supports action. A successful deployment does not necessarily mean that the application will remain healthy under subsequent work.

For a similar investigation, a useful starting checklist is:

• Establish when the deployment completed and when the symptoms began.
• Discuss a safe recovery option with the customer.
• Compare application behavior and resource trends around that interval.
• Look for work that repeats, stalls, or fails to complete.
• Document the evidence and explain the next step to the development team.

The question I keep returning to is simple: what changed in how the application behaves, not just in what was deployed?

Customer and application details have been omitted. This is a personal reflection on a support case, not an official incident report.