Why Your CI Pipeline Is Slow, and the Benchmarks to Prove It

Our CI pipeline took eighteen minutes per run for most of last year, and everyone on the team just accepted it as a fact of life, the way you accept that your commute takes forty minutes regardless of how you feel about it. I got curious one Friday afternoon and decided to actually profile it instead of complaining about it, and it turned out that eighteen minutes had almost nothing to do with our actual test suite, which ran in under three minutes. The rest was overhead we had never bothered to measure.

Treat CI like you would treat a slow endpoint

The mistake most teams make is thinking about CI duration as one number instead of a pipeline of stages you can profile independently. Every CI platform, GitHub Actions, GitLab CI, CircleCI, gives you per-step timing if you look for it. I pulled the last thirty runs of our main pipeline and broke down the average time per step. Here is roughly what it looked like before any changes:

Checkout and setup: 2 min 10 sec

Dependency install: 6 min 40 sec

Build: 3 min 20 sec

Test suite: 2 min 50 sec

Lint and type check: 1 min 50 sec

Docker image build and push: 4 min 15 sec

Deploy step (staging): 1 min 5 sec

Add it up and you get roughly twenty-two minutes worst case, though parallelism across some steps brought the wall-clock time down to eighteen. The single biggest line item, by a wide margin, was dependency installation at six minutes forty seconds, more than double our actual test suite. That was the first thing worth fixing, and also the most boring, which is usually a sign you are looking at the right thing.

Fixing dependency installs

We were running a plain npm ci on every job in the workflow, and critically, we had four separate jobs, lint, test, build, and a type-check job, each doing their own fresh install because they were split across parallel jobs for isolation. That meant we were paying the six-minute install cost four times over, once per job, even though the lockfile had not changed.

The fix was caching the node_modules directory keyed on the lockfile hash, combined with restructuring so that install happens once in a setup job and the other jobs restore from that cache rather than reinstalling from scratch.

uses: actions/cache@v4

with:

path: node_modules

key: node-modules-${{ hashFiles('package-lock.json') }}

This alone took the four separate six-minute installs down to one six-minute install on cache miss and about forty seconds on cache hit, which for a lockfile that changes maybe twice a week meant almost every run was hitting the fast path. That single change cut roughly fifteen minutes of aggregate job time off the pipeline, even though wall-clock time dropped less dramatically because of parallelism, the total compute cost dropped enormously, which matters if you are paying by the minute.

The Docker build was rebuilding everything from layer zero

The second offender was the Docker build and push step at four minutes fifteen seconds. Our Dockerfile copied the entire repository before running the install step, which meant any change to any file, including a README typo fix, invalidated the layer cache back to the very first COPY instruction and forced a full reinstall inside the container, completely independent of whatever caching we had set up at the CI job level.

Reordering the Dockerfile so that package.json and the lockfile are copied and installed before the rest of the source code is copied in is one of the oldest tricks in the book, but it is astonishing how often it gets skipped in real projects that grew organically.

FROM node:20-slim AS deps

WORKDIR /app

COPY package.json package-lock.json ./

RUN npm ci --omit=dev

FROM node:20-slim AS build

WORKDIR /app

COPY --from=deps /app/node_modules ./node_modules

COPY . .

RUN npm run build

Combined with enabling BuildKit's registry cache so that layer cache persists between CI runs, not just within a single run, this brought the Docker step down to about ninety seconds on most builds, since the deps layer stayed cached across commits that only touched application code.

The step nobody questioned: sequential jobs that did not need to be sequential

The last big win was almost embarrassing once we noticed it. Our workflow had lint, type-check, test, and build configured to run sequentially in a single job, presumably because that is how the pipeline was originally written by someone following a tutorial, and nobody had revisited the structure since. None of these four steps actually depend on each other's output, they are all independent checks against the same codebase. Splitting them into four parallel jobs that all check out the same commit and restore the same dependency cache meant the wall-clock time for that whole block dropped from the sum of all four steps to roughly the duration of the slowest one, which was the test suite at just under three minutes.

The results

After all three changes, dependency caching, Docker layer reordering, and job parallelization, our pipeline's median wall-clock duration dropped from eighteen minutes to six minutes and forty seconds. More importantly for cost, aggregate compute minutes per run dropped by around 55 percent, because caching eliminated so much redundant work across the parallel jobs rather than just moving it around.

What I would check first if I were you

If you have not looked at your CI pipeline's per-step breakdown in the last few months, that is the very first thing to do, before changing anything. Do not guess which step is slow, measure it, the same way you would not guess which database query is slow in a production incident. In my experience the biggest wins are almost always in dependency installation and container build caching, not in the actual test or build logic, because those are the steps most likely to have been copy-pasted from a template years ago and never revisited as the project grew. The test suite is usually not your bottleneck. The scaffolding around it usually is, and scaffolding is exactly the kind of thing that nobody questions once it is working, right up until someone finally profiles it.

One more thing worth mentioning: we set up a lightweight monthly check, just a single dashboard panel tracking median and p95 pipeline duration over time, specifically so we would notice the next time this crept back up rather than discovering it again by accident a year later. Pipelines have a way of slowly regressing as a codebase grows, more dependencies, more test files, more Docker layers, and without an explicit metric being watched, that regression is invisible until someone happens to feel impatient enough to complain. Since adding that panel, we caught one regression early, a new integration test suite that someone had accidentally configured to run sequentially rather than in parallel with the rest of the suite, and fixed it within a day of it landing rather than letting it sit for months the way our original dependency install problem did.

The last thing worth naming plainly: this whole exercise cost less than a single wasted sprint would have, maybe a day and a half of one engineer's time across the profiling, the caching changes, and the Dockerfile rework. Compare that to the aggregate cost across a whole team waiting an extra twelve minutes per run, multiple times a day, for months, and the return on that day and a half is not close. I bring this up because slow CI has a strange way of feeling like background weather, something everyone grumbles about but nobody prioritizes fixing, precisely because no single instance of the slowness feels expensive enough to justify stopping and investigating. It is only once you add it up, the way we eventually did, that the actual cost becomes impossible to ignore, and I would encourage any team sitting on a similarly slow pipeline to do that addition explicitly before deciding it is not worth a sprint's attention.