Diagnose a failed CI/CD pipeline and its artifacts

Before you begin

A shell and access to the target environment; use authorized, non-production practice systems for changes.

Confirm that you are authorized to inspect this system and that you have the right environment selected. On production, the checks below are read-only unless a command explicitly says otherwise. Changes can interrupt traffic, alter access, spend cloud budget, or destroy evidence; make them only under the owner’s change process. For practice, prefer a disposable VM, local container, or isolated cluster with synthetic data.

Record the start time, user impact, recent deployments or configuration changes, and the exact object involved. Check that you have an alternate access path before touching networking or authentication. Do not run a command copied from a ticket until you understand each argument and destination.

Trace the delivery object

Use the checks in order and keep the scope fixed: same host or cluster, same account, same time window, and same resource. Save the exact command and timestamp in your incident notes. Do not paste raw environment dumps, full process arguments, configuration files, or unfiltered logs into a public ticket; those often contain credentials or customer data.

1. Confirm the symptom. Reproduce or observe it once and write down the affected endpoint, unit, path, job, or object. If the symptom is not current, preserve timestamps and compare against the last known good state.
2. Establish the baseline. Start with the first command below. Note whether it returns a result, an error, or no matching record. No output can mean the resource is absent, the query is too narrow, permissions hide it, or the event has aged out.
3. Check a second signal. Use the next command to compare the relevant state with logs, metadata, or a neighboring layer. If both signals agree, follow the smallest corresponding fix. If they disagree, stop and resolve the scope or time-window mismatch before changing anything.
4. Check dependencies. Confirm the upstream name, identity, resource, or configuration that this component consumes. A downstream error often reports the point of failure rather than the original cause.
5. Choose a branch. If the expected resource is missing, check its owner and provisioning path. If it exists but is unhealthy, inspect recent changes and dependency status. If checks succeed but users still fail, repeat from the user’s network or runtime path.

These commands are starting points, not proof by themselves. Read the local manual page when flags or output differ. The relevant official reference is linked below. Keep the evidence needed to compare before and after, then remove any temporary files created during diagnosis.

Step 1: capture the baseline

# Read-only diagnostic step 1
rg -n "cache|artifact|upload-artifact|download-artifact" .github .gitlab-ci.yml
git show --stat --oneline COMMIT_SHA

Read the exact target and status returned by this baseline check. Compare it with the expected value described above; if the result is missing or ambiguous, verify permissions and scope before changing state.

Step 2: narrow the affected scope

# Read-only diagnostic step 2
node --version
npm test

Read the exact target and status returned by this follow-up check. Compare it with the expected value described above; if the result is missing or ambiguous, verify permissions and scope before changing state.

Inspect and understand

A cache speeds repeat work and should be regenerable; an artifact is a named output for later jobs or release. Check the commit, runtime, lockfile, registry access, and artifact identity without printing secrets.

Step 1: capture the baseline

# Read-only diagnostic step 1
rg -n "cache|artifact|upload-artifact|download-artifact" .github .gitlab-ci.yml
git show --stat --oneline COMMIT_SHA

Read the exact target and status returned by this baseline check. Compare it with the expected value described above; if the result is missing or ambiguous, verify permissions and scope before changing state.

Step 2: narrow the affected scope

# Read-only diagnostic step 2
node --version
npm test

Read the exact target and status returned by this follow-up check. Compare it with the expected value described above; if the result is missing or ambiguous, verify permissions and scope before changing state.

Read each result against the host’s expected state. A zero exit status means the tool completed; it does not necessarily mean the service is healthy. Pay attention to timestamps, namespaces, identities, mount points, and address families. If your account cannot see process ownership, journal entries, or cluster events, request the minimum read access needed instead of switching to an unrestricted account by habit.

When the first check is empty, do not immediately broaden a command across the whole filesystem or organization. Confirm the spelling, namespace, time range, and current context first. For commands that can print configuration or process arguments, review locally and redact secrets before sharing.

Act on what you find

Choose the smallest correction that directly addresses the confirmed cause. Prefer the owning service manager, deployment definition, package manager, or source-controlled configuration over editing generated state on a live host. Change one variable at a time, preserve the prior value, and write down who approved the action and when it was applied.

Compare the deployed artifact with the one that passed checks; preserve original logs and note what changed between retries.

# Change only after confirming scope, impact, approval, and rollback
npm ci

State the expected impact before running any change. A restart can drop active work; permissions can expose data or block a service; infrastructure changes can replace resources; and cleanup can remove evidence or customer data. If the change could affect users and the cause is uncertain, stop and ask the system owner to review the evidence.

Rollback plan

Before applying a change, capture the previous configuration or value through the supported tool. Define the condition that triggers rollback, the person who can authorize it, and the exact supported reversal. Do not assume that repeating a command reverses it: package upgrades, state operations, deletions, and database changes may not be reversible. If a rollback requires restoring a backup or redeploying a prior artifact, confirm that procedure and its recovery point before the change.

Verification and follow-up

Repeat the original check and use a second signal that observes the user-facing path. Confirm that the error rate or symptom has stopped, that dependent services remain healthy, and that logs do not show a new failure. Compare the same target and time range used for the baseline; otherwise the before-and-after comparison is misleading.

Keep the change note concise: symptom, evidence, exact change, verification result, rollback trigger, and remaining uncertainty. If the issue returns, preserve the new timestamp and logs before trying another fix. A successful command exit does not prove that the full request path works.

# Verify the same target after the change
Compare the deployed artifact with the one that passed checks; preserve original logs and note what changed between retries.

Common mistakes and version notes

Do not confuse a local check with an end-to-end check. A process can be running while its socket is bound to the wrong address; a deployment can be successful while an older artifact is still serving traffic; and a DNS response can be valid for one resolver but wrong for the application’s configured search path. Follow the path used by the affected user or workload.

Output and available flags vary with distribution, tool release, privilege, and configuration. The guide intentionally avoids assuming a specific release where the repository has no verified host matrix. Debian and Ubuntu commonly use APT, while Fedora and many current RHEL-family systems use DNF; systemd-specific commands require a systemd host. Container and cluster commands depend on the selected context. Check the installed manual and provider documentation before applying platform-specific steps.

One common false lead is treating missing output as a clean bill of health. Access controls, rotation, retention, caching, namespace selection, and stale local state can all hide the relevant evidence. Verify your scope and permissions, then compare a second source before closing the incident.