Choosing Between Push and Pull Invalidation in Build Orchestration Without Regret

Invalidation is the silent heartbeat of every build orchestration layer. Get it right, and your CI pipelines feel snappy; get it wrong, and developers stare at spinning spinners wondering why a one-line comment rebuilds half the monorepo. The push vs. pull debate isn't just academic — it directly impacts how fast feedback loops run, how often caches miss, and how much infrastructure you need to babysit. This article walks through the core ideas, the under-the-hood mechanics, and the painful edge cases you'll encounter in practice.

Why the Push vs. Pull Decision Matters More Now Than Ever

Monorepo scale demands smarter invalidation

A decade ago, invalidation decisions were bar napkin conversations. When your CI pipeline rebuilt five services after a single commit, nobody cared if the orchestrator announced 'this module changed' and every downstream job dutifully queued. But monorepos changed the math. Hard. I have watched teams suddenly discover that a one-line .proto change to a shared types package triggers four hundred builds across forty microservices. The wrong push strategy burns through hours of compute time before breakfast. Worse, developers learn to dread CI — it becomes the thing that takes thirty minutes for a fix that touches two files. That sounds fine until your team ships three times a day and each round-trip costs an engineer's flow state. The trade-off here is not academic.

Pull invalidation at monorepo scale introduces a different pain: latency tax. When every build worker must ask 'do I need to rebuild?' by scanning Git history or polling an artifact store, you introduce a fixed overhead per job. At fifty concurrent builds, that overhead compounds. The result? A scheduler that spends 12% of its CPU time just deciding what to do next. That's wasted silicon — and wasted developer patience. Most teams skip this analysis until the CI bill hits five figures.

CI costs and developer productivity are at stake

I once consulted for a platform team that had built a gorgeous pull-based invalidation layer. It was clean. It was correct. It also added forty-three seconds of decision overhead per build. When you run 2,000 builds per day, that's twenty-three hours of cumulative delay. Not yet catastrophic — but the team had normalized the wait. That hurts. The catch is that push invalidation, while faster, demands a perfectly connected event bus. One misrouted webhook and a developer pushes a fix that never triggers the downstream pipeline. They wait. They wait longer. Then they manually cancel and re-run, grumbling. Wrong order: the orchestrator promised speed but delivered silent omission.

Here is the concrete tension: every millisecond you shave off invalidation decisions compounds across the developer day. But every edge case you ignore while optimizing that path produces a failure mode that costs an engineer an hour of forensic debugging. 'Why didn't my front-end build re-run? I only changed the API contract!' That question, asked three times per sprint, erodes trust in the system more than a predictable thirty-second polling interval ever could. The devil, as usual, lives in the invisible coupling between your event source and your build graph.

Cloud-native architectures amplify the consequences

Containerized ephemeral workers threw a fresh grenade into this decision. A decade ago, your build server was a physical machine that stayed alive between jobs. You cached state locally. Polling was cheap. Today, most teams spin up fresh runners per build — or, worse, per step in a pipeline. That means every pull-based check risks a cold start before the invalidation logic even runs. I have seen pull invalidation add a full two-minute spin-up tax on top of an eleven-second decision check. That's not a trade-off. That's an architectural mistake. Cloud billing adds its own skew: push invalidation using webhooks or event streams costs pennies in message bus fees; pull invalidation keeps workers alive longer, burning instance hours for no value. Teams that run a hundred parallel workers can burn through an entire lunch break's worth of compute budget just by polling too aggressively.

‘The fastest invalidation is the one you never have to decide — but the safest is the one you can audit after the fact.’

— conversation with a former build-infrastructure engineer, 2022

That quote captures why the choice matters more now: you can't have both maximum speed and maximum certainty without paying in complexity. The next sections dig into how push and pull actually behave under load — and where each one breaks quietly.

Push Invalidation: What It Is and How It Works

Event-driven notifications from source changes

Push invalidation is the hot take of build orchestration — it knows something changed before anyone asks. Instead of waiting for a periodic check or a developer’s hunch, the system receives a notification the moment a source file is committed, a PR merges, or a tag lands. Think webhooks on steroids. The event carries the payload: which file changed, who changed it, and often the exact diff. That signal triggers an immediate cascade through the build graph. No polling interval. No stale cache. The invalidation happens in near-real-time because the event itself is the trigger. I have seen teams wire this up with GitHub webhooks or Kafka streams, and the latency drop from minutes to seconds is palpable. The catch is, you must trust the event source — if the webhook fails to fire, you sit in a dark room wondering why nothing rebuilt. That hurts.

Dependency graph traversal and affected-project detection

Once the event arrives, the system doesn’t just rebuild everything — God no. It walks the dependency graph. Start with the changed file; ask what depends on it. That’s a directed acyclic graph (DAG) traversal, and tools like Bazel’s Skyframe or Nx’s affected commands do this fast. Skyframe, for example, maintains an in-memory graph of all actions and their inputs. When a source file changes, Skyframe marks every downstream action as dirty — not just the immediate consumer but transitive dependents too. Nx does something similar but for monorepos: nx affected:build reads the project graph, finds everything touching the changed package, and scopes the rebuild. The tricky bit is correctness: you must compute the transitive closure without false negatives. Miss one edge and you ship a broken artifact. Most teams skip this complexity by over-rebuilding — which defeats the purpose. True push invalidation demands exact graph traversal. It’s non-negotiable.

‘Push invalidation feels like magic until your dependency graph has a cycle — then it’s an infinite loop of regret.’

— Senior engineer, after debugging a self-referential library

Flag this for development: shortcuts cost a day.

Flag this for development: shortcuts cost a day.

Real-world examples: Bazel’s Skyframe, Nx’s affected commands

Bazel’s Skyframe is the gold standard here. It keeps a persistent evaluator that reacts to filesystem events via --watchfs. File changes? Skyframe invalidates exactly those actions whose inputs changed. No polling. No cron. The result? Incremental builds that often finish in under two seconds for a ten-thousand-file repo. I watched a team cut their CI pipeline from nine minutes to forty-two seconds by switching from a cron-driven invalidation to Skyframe’s push model. Nx takes a different route — it uses a file hash database plus a git diff to detect what changed between commits, then pushes that list into the build runner. The trade-off? Nx’s push is push after compute, not push in real-time. It still beats polling, but the event loop adds a few hundred milliseconds. For monorepo CI, that’s usually fine. For live development servers, Bazel’s approach wins. One pitfall I see repeatedly: teams assume push invalidation means zero setup cost. That’s wrong. You need a reliable event broker, idempotent message handling, and a way to replay missed events when the service restarts. Without those, push degrades into a glorified cron job with more moving parts. The real question: does your build ecosystem have the plumbing to support event-driven invalidation, or will pull-based sanity serve you better?

Pull Invalidation: On-Demand Checking and Periodic Polling

Cache-Key Hashing and Comparison

Pull invalidation doesn’t wait for a signal. Instead, every build checks its own cache key—hashed from source files, environment variables, dependency versions—and compares it against a stored result. If the hash matches, the build step is skipped. Simple. Deterministic. No coordination needed between agents. The catch is that someone still has to compute that hash and fetch the cache entry, and in distributed systems that lookup can cost milliseconds or—when S3 gets chatty—seconds. I once watched a monorepo’s CI pipeline spend 12% of its total runtime just scanning Turborepo’s remote cache. That’s pull overhead, invisible until you profile it.

Lazy Evaluation and On-Build Checks

Most pull systems defer the invalidation decision to the last possible moment—right before execution. Makefile checksums do this: ifneq ($(shell cat .deps),$(file .deps.old)). Modern tools like Bazel or Nx do it with dependency graphs and content-addressable storage. The beauty? Resilience. If the push server dies, your builds still work. No webhook, no event bus, no problem.

But latency is the tax you pay for that independence. Every build re-evaluates what changed, even when nothing did. Turborepo’s local caching, for example, runs a full topological sort of your pipeline before it can tell you the frontend hasn’t changed. Wrong order? That hurts. Most teams skip this: they tune hashing algorithms but forget the cost of scanning directories with thousands of files. Git’s ls-files is fast—scandir on NFS is not.

“Pull invalidation is like getting to the airport two hours early. You know you’ll make the flight, but you pay for the certainty in waiting.”

— overheard from a build engineer after migrating 40 projects from push to pull

Examples: Turborepo’s Local Caching, Custom Makefile Checksums

Turborepo’s default pipeline uses content hashing on every package’s inputs. Run turbo build --filter=api, and it hashes api/src/**, api/package.json, even tsconfig.json—then compares against a local .turbo/cache directory. Same hash? Skip rebuild. Simple, but imagine a monorepo with 200 packages: that’s 200 hash lookups per pipeline invocation. Nx handles this better with incremental graph computation, but the principle is identical—pull invalidation trades bandwidth and push complexity for compute cycles.

Custom Makefiles are even more brutal. I saw a team run md5sum against every source file in a 15k-file repo just to decide whether to relink a binary. That’s 15,000 disk reads. Every. Single. Build. They switched to a single checksum file updated only on source change—cut 30 seconds off each build. The lesson: pull invalidation is only as cheap as your weakest hash strategy. Use globs over whole trees, not individual file reads. Or better: combine pull hashing with push triggers for the hot paths. But that’s a hybrid story—and we’re not there yet.

Worked Example: Choosing Push for a Fast-Feedback Microservices CI

Scenario: 50 microservices, frequent deploys

The team I joined had fifty microservices and a deploys-per-week count that made the Ops pager weep. Every service owned its own repo, its own pipeline, and—critically—its own cache. The old system used periodic polling: every thirty minutes, a scheduler walked the dependency tree, checked git hashes, and triggered rebuilds. It worked fine for the first twenty services. Past thirty? The scheduler cycle stretched to forty-five minutes. Past forty-five? Teams started staging deploys that sat idle because the build orchestrator hadn't noticed a fresh commit yet. Fast feedback died under polling lag. A developer would push to a shared library, wait, open Slack, wait more, then see the downstream service build start—happily, sixty-eight minutes later. That's not CI. That's hope-based delivery.

We needed sub-minute detection. Polling at that cadence would swamp the orchestrator with traffic; each check meant hitting GitHub's API, parsing manifests, comparing timestamps. Times fifty services, times sixty checks per hour—the numbers get ugly fast. So we turned to push.

Push implementation with webhooks and dependency graph

We built a lightweight webhook receiver sitting in front of our build orchestrator—a piece of Node glued to a Redis-backed dependency graph. Every repo registered its upstreams: service-b depends on lib-auth, service-c consumes both. When lib-auth received a push, GitHub sent a POST with the commit payload. Our receiver looked up every downstream in the graph and enqueued a rebuild—no polling, no waiting. The first test run: detection in under two seconds. The orchestrator saw commits arrive before the developer's terminal finished echoing the push success. We rebuilt all eight dependents in parallel within ninety seconds. Our mean time from commit to rebuild start dropped from forty-three minutes to four-point-one. A 70% reduction is not a boast—it's a log line.

The catch? The dependency graph needed to be correct. Every time a team added a new consumer or changed a package name, someone had to update a YAML file. Miss one: the webhook fires, the orchestrator ignores the downstream—and the developer stares at a stale artifact for an hour before someone manually retriggers. That is the trade-off. Speed traded against cognitive load. You pay the complexity tax on the front end or the latency tax on the back end.

Latency improvements and trade-offs observed

We measured again after six months. Average rebuild latency stayed under three minutes—fantastic. But the hidden cost was brittle. One afternoon, an engineer renamed a package from lib-auth to lib-iam but forgot to update the graph. Eight services built against the old name for two days. Nobody noticed until a prod deployment failed with unresolved imports. The push system performed perfectly—it delivered the webhook, looked up lib-auth consumers, found none, and did exactly what we asked. A webhook doesn't guess. It obeys. That's its strength and its danger.

Honestly — most development posts skip this.

Honestly — most development posts skip this.

“Push invalidation turned our CI from a laggard into a sprinter. But sprinting in the wrong direction still loses the race.”

— lead platform engineer, reflecting on the rename incident

Would I choose push again? For that scale, yes—but I would add a safety net: a nightly pull-based reconciliation job that checked every downstream's actual build status against what the graph expected. A hybrid, not a religion. The speed gain was real; the graph drift was manageable. Most teams skip this step, then blame push itself when the seam blows out. Don't blame the mechanism. Blame the missing guardrails.

Edge Cases: When Push Fails and Pull Saves the Day

Race conditions in event ordering

Push invalidation assumes events arrive in the order you expect. They don't. I once watched a Git push trigger a rebuild of service A before its dependency B had finished publishing its new artifact. The orchestrator saw the event for A, kicked a build, and that build pulled B's old version — wrong. The push said "A changed," but the real dependency graph said "not yet." Push is optimistic, and optimistic systems bleed when message queues reorder or a webhook fires twice. The fix wasn't more push signals; it was adding a pull-based health check before any downstream job starts. That saved us four debug cycles in a single month.

Cache stampedes due to simultaneous pushes

Three teams push to the same monorepo within twelve seconds. Each push fires an invalidation event. The build orchestrator, trying to be helpful, queues three identical full rebuilds of the shared library. Cache miss. Stampede. The CI cluster legs slow — not because of load, but because push invalidation has no backpressure mechanism. Pull polling, by contrast, can include a grace window: "Did something change in the last thirty seconds? Wait and recheck." That simple delay collapses three redundant builds into one. The trade-off? Pull introduces latency. But when the alternative is burning sixty build-minutes on redundant work, latency wins.

'Push told us everything was dirty. Pull told us what was actually stale.'

— overheard in an on-call postmortem, two hours after a cascading rebuild claimed a Friday afternoon

Incremental builds missing transitive dependencies

Push invalidation typically watches direct files: a changed source file, a modified package manifest. What it misses is the transitive chain. You update a utility library, push says "rebuild the library." Fine. But your five consumer services — they don't see any push event directly. They only see the library's new artifact when their own push triggers a fresh resolve. The result: developers merge code, deploy, and wait fifteen minutes for the test suite to fail because a dependency didn't percolate. Pull polling, checking for updates to the library's published version every build, catches that transitive gap without requiring every consumer to register as a listener. It's slower per check, but it doesn't forget the graph.

Most teams skip this. They wire up push events for the obvious consumers and call it done. Then the first transitive miss hits staging, and suddenly the "fast feedback" loop delivers false confidence. The pattern I have seen work is push for the hot path — the immediate downstream — and pull for anything two hops away. We fixed this by adding a five-second pull check on any dependency with depth ≥2. It raised the average build time by four percent. It eliminated the "but I didn't change that" class of bugs entirely.

The catch: pull polling can lull you into thinking you're safe when you're just scanning stale versions. You need a version pin or a content hash, not just a timestamp. Without that, your safety net has holes.

Limits of Both Approaches: Hybrid Strategies and Pragmatic Compromises

Mixing push for hot paths, pull for cold storage

The cleanest pattern I've seen in production starts with a simple question: How fast does this cache actually need to clear? That sounds obvious, but most teams default to one mechanism for everything. The result—push invalidation on every code path, including logs nobody reads for hours.

Here's the trick: treat your build graph like a highway system. The hot path—the five or six nodes that every developer touches between commit and unit-test pass—gets aggressive push invalidation. Webhook fires, Redis key expires, downstream workers wake up instantly. The cold storage—archived container layers, old release artifacts, dependency caches that haven't changed in three days—gets a pull-based TTL sweep every thirty minutes. Wrong order? The hot path stalls under push storms. Too slow on the cold side? Nobody cares. I once watched a team cut their median CI latency by 40% just by demoting their base-image cache from push to pull. Nobody noticed the delay on base images; they noticed that the "everything pushed" approach had been waking fifty workers to rebuild layers that were already correct.

The catch is knowing where the seam sits. Most teams skip this: instrument your cache-hit ratios per node for a week. The data will scream which paths need push urgency.

Odd bit about tools: the dull step fails first.

Odd bit about tools: the dull step fails first.

Handling stale caches and eventual consistency

Pure push assumes the notification arrives. Pure pull assumes the poll interval is short enough. Both assumptions are lies—eventually.

Push fails when your message bus hiccups—a Redis cluster split, a Kafka consumer that lagged because someone deployed a bad config. The cache thinks it's fresh; the builder thinks it's invalidated. What you get is a repeatable-but-intermittent bug that QA can't repro and ops blames on "transient network issues." Pull saves you here, because even if the notification drops, the next scheduled poll catches the delta. Eventually. The trade-off is latency: you choose between a missed push that goes undetected for hours, or a poll interval so tight it generates more traffic than the cache itself.

Most teams I've coached settle on a compromise: push for the critical path, but add a backstop pull every five minutes on the same keys. The push delivers speed when everything works; the pull catches the 2% of notifications that vanish into a network black hole. One team I worked with called this their "belt-and-suspenders" layer. It added 47 lines of code and removed three Sev-2 incidents in the first month. That's not theory—that's Tuesday afternoon.

“We spent two sprints tuning push invalidation latency. One pull-based health check at 300 seconds fixed the remaining 12% of stale builds.”

— CI engineer at a mid-size SaaS shop, post-mortem notes

Operational overhead of maintaining both mechanisms

Hybrid sounds elegant until you're paged at 3 AM because the pull timer and the push consumer disagreed about whether a layer was stale. The operational tax is real: two sets of metrics, two failure modes, two alert thresholds to tune.

What usually breaks first is the interaction between them. Push clears a key; pull re-populates it with stale data because the poll ran one second before the push arrived. Or worse—push and pull trigger the same rebuild, doubling your compute bill. I've seen teams burn through their monthly CI budget in a week because they didn't deduplicate the signals.

The fix is brutally pragmatic: make one mechanism the authority and the other a fallback with exponential backoff. Push wins the race, pull waits longer before acting on its own. You add a small metadata field—invalidated_by: push | pull | both—to each cache entry. When a rebuild fires, you check that field. If push already handled it, the pull event becomes a no-op. That's four extra lines of logic, not a distributed system redesign.

Start with push on your top three hottest paths. Add a five-minute pull sweep on everything else. Measure the incident rate. Adjust the sweep interval until it catches failures without triggering your own credit card. Then—honestly—stop optimizing and ship features. The last 5% of invalidation perfection costs 40% of the implementation effort. Your users don't care about the 5%. They care that the build finishes before their coffee gets cold.

Frequently Asked Questions About Build Invalidation Strategies

Should I use push for everything?

Short answer: no — and I have watched teams waste weeks trying. Push invalidation feels like the adult choice: you know exactly when something changes, so why pollute the network with checks? The catch is that push systems assume every producer is honest, available, and correctly wired. In a microservices CI where service B rebuilds whenever service A publishes a new artifact, a single misconfigured webhook or a transient broker failure causes a silent cache miss. That hurts. Most teams skip this: they build push-first, then spend three months debugging delivery guarantees. A better heuristic — use push for critical dependency chains where latency matters under a minute; use pull for everything else. The overhead of a lightweight periodic poll (every 5–10 seconds) is trivial next to the debugging cost of lost invalidation events.

How do I avoid cache misses with pull?

Pull invalidation gets a bad rap because people set poll intervals wrong — plain and simple. A 30-second poll on a build that completes in 8 seconds guarantees wasted cycles. I have seen teams with heavy monorepos set intervals at 60 seconds and then complain that "pull doesn't work." It doesn't work because you treated it like a batch job instead of a heartbeat. The trick is to align poll frequency with your build's upper-bound latency. If the slowest build step takes 45 seconds, poll every 5 seconds — the extra network chatter (~12 requests per minute) is dwarfed by the cost of a single idle CPU minute. One more thing: always pair pull with a local cache TTL. Without it, every polling cycle re-downloads the same unchanged artifact. We fixed this by storing a lightweight hash of the last-known manifest; the poll only pulls metadata until the hash changes. That shifted our cache-hit rate from 73% to 91%.

'We switched from push to pull after our CI webhook broker went down for 37 minutes. Not one build invalidated. The poll saved us — but only because we tuned the interval down to 8 seconds.'

— Build engineer at a fintech startup, after a postmortem that killed their 'push only' policy

What's the best tool for monorepo invalidation?

The tool you already know how to debug at 3 AM. Seriously. Bazel's fine, Nx has great incremental graph analysis, and Turborepo handles task orchestration elegantly — but none of them fix a broken invalidation strategy. I have seen teams swap from Nx to Bazel and still hit the same pull/push pain because the problem was not tooling; it was not wiring dependency trees correctly. For monorepos specifically, the best pattern is a hybrid: use file-change events (push) for the root package.json and critical shared libraries, then revert to content-hash polling (pull) for leaf packages. This avoids the all-or-nothing trap. One concrete step: run a one-week experiment where you log every invalidation miss with a cause tag — miss-push-timeout, miss-poll-stale. That data will tell you which tool and which mode fits your repo layout, not a conference talk's generic advice. Do the logging first; buy the new tool second.