Back to blog

Your p95 API latency is on a dashboard. Your database query times are traced. Your memory usage pages you at 3 a.m..

Your CI pipeline? You probably know it's "slow lately" and that one test "keeps failing on Tuesdays."

CI is the most-run piece of infrastructure at most engineering teams — and almost universally the least observed.

The cost you're not measuring

Most teams think of CI cost as the line item on a bill. But the real cost is developer time.

A 12-minute build means 12 minutes of context-switching minimum — usually 25+ once the engineer re-focuses. A flaky test suite means someone triages failures every morning. A mystery slowdown means a senior engineer spending an afternoon on git bisect through workflow files.

At a 20-person engineering org, if CI adds 30 minutes of friction per engineer per day, that's 150 engineer-hours per month — before you touch the compute bill.

Why CI stays dark

CI tools were built to answer a binary question: did this pass or fail? That binary answer is useful, but it's the floor, not the ceiling.

What you actually want to know:

• Which step is slow, and is it getting slower over time?
• Is this failure correlated with a specific runner, a time of day, a test ordering?
• When my build time doubled last Tuesday, what changed?

None of these questions are answerable with pass/fail. They require time-series data, distributed traces, and the ability to correlate across runs.

What "observed CI" looks like in practice

When CI emits telemetry the same way your application does, the questions above become routine.

A Prometheus query shows you which workflow step's p95 duration spiked after a dependency upgrade. A trace through a failed run shows you the test that always fails after a specific setup step — because they share a port. A dashboard shows you that your nightly builds are 40% slower than your PR builds because they run on a different runner class.

This isn't exotic. It's the same tooling your on-call rotation already uses — applied to the system that runs your code before your users do.

The shift

Treating CI as observable infrastructure isn't a big-team luxury. It's what separates teams that continuously speed up their pipelines from teams that periodically panic about them.

Your CI pipeline runs on every commit, every day. It deserves a dashboard.

If you want to see what's actually happening in your builds, connect your repos and take a look. The first thing most people notice is a job they never knew was the bottleneck.