When Incremental Builds Stop Being Incremental — How to Spot the Mistake in Your Task Graph

So you've done the incremental build dance. Annotations, caches, task outputs—all configured. And for a while, builds fly. Then one Monday morning, the CI run takes 40 minutes again. Or your local ./gradlew assembleDebug decides to recompile everything after a simple string change. The task graph says UP-TO-DATE, but it's lying. Here's how to catch it.

Where Incremental Builds Actually Break — A Field Guide

Monday morning CI regressions

The build was fine Friday afternoon. Monday arrives, the CI server takes forty minutes—double what it did last week. Nobody changed the Gradle files. Or did they? I have seen this exact scene play out across three different teams, and the culprit is rarely a single rogue plugin. What usually breaks first is a composite build dependency that quietly re-resolves on every branch. You merge a feature branch that touched nothing directly related to caching—just a comment in a settings file, maybe an unused import—and suddenly the entire task graph thinks it must recompile core modules. Wrong order. The team wastes two hours hunting for a gradle.properties change that never happened.

Local builds after a branch switch

You switch from main to a colleague's feature branch. You run ./gradlew test and it sits there, executing tasks you swear didn't change. The real trick is hidden in task inputs: a generated source directory that Git treats as untracked but Gradle treats as stale. The build system sees file timestamps shifting—even though the content is identical. That hurts. The fix isn't a clean build; it's understanding that version-controlled and generated directories collide. We fixed this once by adding exclude 'build/' in the task's input normalization—three lines that cut local build time from six minutes to ninety seconds. The catch is that most developers don't know input normalization exists as a dial, not an on-off switch.

Release pipelines that mysteriously slow down

Release builds should be fast because nothing changes between the last CI green and the signed artifact. Except they aren't. The seam blows out when your build runs inside a Docker container that resets .gradle caches on every invocation—or worse, uses a shared cache volume that multiple agents trample. I once watched a release pipeline degrade over six weeks: 12 minutes, then 17, then 29. The root cause? A custom Copy task that expanded a ZIP file into the build directory without declaring its outputs. Each release appended files to an existing directory, and Gradle could not determine what was fresh. The team's response? They added clean to the release script. That's like fixing a leaky pipe by turning off the water to the whole house.

'We lost two sprints to build debugging. The saddest part? The fix was removing one @CacheableTask annotation that was lying about its inputs.'

— Staff engineer at a 300-person SaaS company, after a migration to Kotlin DSL

Where the field guide leaves you

Spot these breaks early. Monday morning regression? Check composite build resolution. Branch switch slowdown? Audit input normalization. Release pipeline creep? Run --scan and look for tasks with zero cache hits despite unchanged sources. One rhetorical question worth asking: if your build is slower today than last quarter, and you haven't touched the configuration, what environmental factor shifted under you? That's where the real detective work begins—and where most teams stop too soon.

What Developers Get Wrong About Incremental Builds

UP-TO-DATE vs. FROM-CACHE vs. EXECUTED — the Three States You Actually Have

I have watched teams stare at a build log for fifteen minutes, see 'UP-TO-DATE' on half their tasks, and declare victory. The build was still slow. That's the first mistake: treating UP-TO-DATE like a trophy. It's not. UP-TO-DATE just means Gradle checked your inputs, found nothing new, and skipped the task. That's the ideal—zero work, zero time. But here is what developers miss: FROM-CACHE looks identical in the log and behaves very differently. When a task is marked FROM-CACHE, Gradle downloaded a pre-built result from a remote cache. No computation happened on your machine. No inputs were verified against local disk state. The seam between these two states is where builds break silently. The task graph says everything is fine. Your CPU says otherwise if the cache server was slow, or if the build cache key changed because someone added a timestamp to a file name. EXECUTED is the red flag—it means the task ran from scratch, and if you see EXECUTED on a task you expected to be stable, something in your input declaration is lying.

Task inputs vs. outputs — the confusion that costs you a rebuild

Developers treat inputs and outputs like a checklist: "I declared my Java source files, I declared my output JAR, done." The catch is what you forget to declare. A common sight: a code generation task that reads a schema file but also reads a local configuration directory. The developer annotated the schema file as @InputFile but omitted the config directory. On the second run, Gradle sees the schema file is unchanged, marks the task UP-TO-DATE, and your generated code quietly diverges from what the config directory expects. That's not a false positive—it's a correct Gradle decision based on incomplete information. The real problem? When you finally notice the divergence, the instinct is to run 'clean' and move on. Most teams skip this: they don't audit their task's @InputFiles annotations after adding a new source of truth. I once spent three days tracking down an incremental build that "worked" on every engineer's machine except mine. The culprit: a shared build cache that stored the output from a @Input property that was actually a relative path, not an absolute one.

'We fixed it by cleaning every morning. Then we fixed it by cleaning every commit. Then we gave up and moved to a monorepo.'

— Anonymous engineer, after two months of incremental build drift

Why 'clean build' isn't the answer — it's the symptom

Clean builds are a leak. They mask the exact moment your task graph becomes stale. When a team says "we just run clean on CI," they have accepted that their incremental model is broken—and they stop looking for the root cause. Worse, clean builds train developers to distrust the build system. You stop thinking about input boundaries. You stop questioning why a task re-runs. Instead, you reach for the nuclear option and lose thirty seconds to two minutes per invocation. That sounds fine until you multiply by twenty engineers, each running clean ten times a day. The cost is not time alone; it's the feedback loop. When a build takes five seconds, you stay in the flow. When it takes two minutes, you switch tabs, check email, and the context is gone. The editorial signal here is clear: if your team has a 'clean first' ritual, your task graph has a hidden dependency. Find it. Annotate it. Your muscle memory is masking the real issue. Not yet ready to audit every annotation? Start with one: pick the task that runs most often in your local workflow and verify every @Input and @OutputFile against what actually changes between runs. Wrong order? You will know in the first afternoon—that EXECUTED label will stare back at you.

Odd bit about development: the dull step fails first.

Patterns That Actually Keep Builds Fast

Relocatable Task Outputs with Relative Paths

Most teams break incrementality the moment they hardcode an absolute path into a task. I have watched engineers spend an entire afternoon wondering why :app:compileKotlin reruns on every commit — then we find /Users/ci/jenkins/workspace/ baked into a transform. The fix is boring, fast, and painfully obvious: use relative output directories and let Gradle resolve the project directory. A task that writes to ${buildDir}/intermediates/classes stays invalidated only when its inputs change. That sounds trivial. It's. Yet I see projects where three different custom tasks reference System.getProperty('user.dir') — instant cache miss, every time, across machines.

What about upToDateWhen? Most teams skip this: the predicate should never reach into the filesystem for metadata you don't track as an input. "But we only check the timestamp of output.jar" — that already works, because Gradle does that for you. Adding upToDateWhen { file('stamp.txt').exists() } creates a hidden dependency that the task graph doesn't see. Quick reality check — if your task re-runs when nothing changed, trace every upToDateWhen block first. That's where incrementality goes to die.

Stable Task Names and Configuration Avoidance

Task names that change per developer or per branch gut your build cache. I fixed a build where a code-generator used the branch name in its task identity: generateSources_main_bugfix_LOGIN_42. Every merge created a brand new task. Cache hit ratio: zero. The pattern that works is locking task names to project coordinates — group, module, variant — never to environment variables or timestamps. You can pass dynamic values as inputs; just don't let them leak into the task's declared name or @OutputDirectory path.

Configuration avoidance is the other half. register over create. whenConfigured instead of eager blocks. That gets preached endlessly. The catch is that configuration avoidance only helps if your custom tasks use Provider APIs for inputs. A TaskContainer.register that then calls .configure { inputFile.set(someLazyProperty) } stays lazy. A register that calls inputFile = file('x') in a closure is not lazy — it evaluates during configuration. Most teams don't realize this until their build spends forty seconds configuring tasks they never run.

'We registered everything, but configuration still took twelve seconds.' — real statement from a team that used create in a loop that iterated over a resolved file collection.

— that team fixed it by switching to register and passing fileTree as a Provider. The loop ran at configuration time anyway because they resolved the file collection eagerly. Painful. Common.

Using Build Cache with Deterministic Inputs

The remote build cache is not magic — it's a deterministic function of your inputs. Change the JDK micro-version that runs the task? Different hash, cache miss. Embed the current timestamp in a generated BuildConfig field? Every build is new. The pattern that preserves cache hits is isolating non-determinism to a single, explicit task that the rest of the graph doesn't depend on. A :app:generateBuildTimestamp that emits a constant placeholder — then the runtime replaces that placeholder — keeps every compile cacheable.

Here is the trade-off: deterministic inputs sometimes mean you can't use the same cache across OS families. That's fine. Split your cache namespace by operating system, and live with the hit. What hurts is having a cache key that varies because a plugin enumerates files in random order from fileTree — Gradle's default ordering is filesystem-dependent, so two machines get different SortedSet inputs. Explicit .sort() on your input collection. Did that last week on a project that was getting 18 % cache hit rate. After sorting: 83 %. Wrong order in the set was the entire problem.

One rhetorical question before you close this section: is your buildCache block configured for push = false on developer machines? If yes, nobody writes to the cache except CI — but developers also never pull from it if remote = HttpBuildCache { url = ...; isPush = false } is set correctly. That's correct behavior. The mistake is pushing from developer laptops with stale classpaths. Keep CI as the sole pusher. Your teammates will thank you when their local tasks stop recompiling because someone's Android SDK patch version differed by three digits.

Anti-Patterns That Send Teams Back to Clean Builds

Non-deterministic task inputs — timestamps, random IDs, and build-system poison

The fastest way to kill an incremental build is to feed it something that changes every single time. I’ve watched teams embed a UUID.randomUUID() into a generated resource file — just to tag a version — and then wonder why every compile ran from scratch. That randomness propagates: task B sees task A’s output changed, so B re-runs, then C, then D. Whole graph collapses. Another common offender? Writing the current system time into a header or a manifest field. It feels harmless — a timestamp in a build-info.properties — but Gradle’s up-to-date check compares inputs byte-for-byte. A new value every minute means a fresh execution every minute. The fix is brutal but simple: strip all non-deterministic values from declared inputs, or move them into a separate task that downstream consumers treat as volatile. Either way, you stop lying to the task graph.

Leaking absolute paths into task outputs — the portability trap

Absolute paths are a silent contract breaker. When a task writes something like /home/ci-user/build/output.jar into a generated file, that string becomes an input to every downstream task that reads that file. Move the checkout to /tmp/build on a different agent? Every cached result invalidates. Worse, two developers on macOS and Linux see different path prefixes — so one person’s incremental build is another person’s clean build, and nobody agrees on what “cached” means. We fixed this once by replacing a custom annotation processor that emitted File.path output with a relative-path variant. The build went from 100% cache misses down to 40%. The trade-off: you lose some developer convenience when debugging because stack traces no longer include workstation-specific roots. That’s a cost. But paying it means your CI cache actually works across machines.

Field note: android plans crack at handoff.

Over-declaring inputs via glob patterns — the productivity illusion

The natural instinct is to cast a wide net: inputs.files(fileTree('src') { include '**/*' }). Safe, right? Wrong. Every file change — even a whitespace tweak in a deeply nested README that has nothing to do with compilation — now marks the task dirty. I’ve seen builds where a single CSS comment change triggered re-minification of every JavaScript bundle, just because the glob grabbed everything under src/assets/. Tighten the pattern. Be explicit. List only the actual source set files. Quick reality check — if your task graph has a node that declares **/*.properties but only reads build.properties, you’re paying for every new .properties file someone adds to the repo. That turns a five-second incremental change into a forty-second rerun. Not yet a clean build, but close enough that frustrated devs start running clean out of habit. And once that habit sets in, incrementality dies.

“We reverted to clean builds because we couldn’t trust the cache. Turned out we didn’t understand what our own globs were capturing.”

— Staff engineer at a mid-size SaaS team, after three weeks of debugging

Aggregate outputs that mask granular changes

Sometimes the mistake isn’t what you declare — it’s how you bundle the result. A single fat.jar containing everything from 200 classes means any single class change forces the whole archive to rebuild. The task graph looks incremental, but the output is all-or-nothing. That’s not a broken cache; it’s a broken strategy. Better: output individual class files or modular jars, then let a separate packaging task combine them only when the final artifact is needed. The extra task adds some overhead, but the trade-off is that ninety-five percent of edits only rebuild one jar, not the monolith. That said, this refactor takes real effort — you might decide the pipe doesn’t need it if your full build is under two minutes. But if your team already jokes about “coffee break compiles,” the aggregate output is probably the culprit.

The Long-Term Cost of Incremental Build Drift

It Starts With One Ignored Annotation Processor

Six months ago, a team I worked with had build times around forty-five seconds. Not great, not terrible—they shipped daily, nobody complained. Yet by month seven, that same project took over four minutes per incremental change. Nobody ran clean deliberately; they just started noticing that Gradle's UP-TO-DATE checks seemed to stop caring. The root cause? One misconfigured annotation processor in a shared library. It had a non-deterministic output—timestamp metadata that changed every run—so Gradle correctly invalidated everything downstream. Every. Single. Time. The team blamed memory, blamed plugins, even blamed the CI machines. But the real culprit was subtle drift, invisible week to week, catastrophic over a quarter.

Silent Accumulation of Task Re-Execution

Here is the pattern that kills. A developer adds a custom task that writes a build artifact outside the Gradle output directory—say, into build/tmp/ but via absolute path. Next sprint, another developer modifies a plugin that reads that file. No one notices because the build still works. But Gradle's task graph no longer tracks that dependency. The result? Full re-execution of tasks that should be cacheable. I have seen projects where three small violations—a wrong output directory, a missing annotation processor declaration, a shared mutable property—added forty seconds to every incremental build. The seam blows out slowly, like a tire with a nail. You only spot the leak when someone runs a diff of task outcomes across two weeks and discovers that compileJava has been running fully every time since February.

“We thought we were doing incremental builds. We were actually running near-clean executions while Gradle pretended everything was fine.”

— lead engineer, mid-market Android team, after a three-month drift audit

How to Detect Drift Before It Becomes Normal

Most teams skip this: record your task graph outcomes as structured data. A simple script dumping gradle build --scan output to a JSON file every Friday lets you track the ratio of UP-TO-DATE vs rerun tasks. That ratio should remain flat for stable codebases. When it drops below 70% for unchanged source files, something is rotten. The catch is that build scans alone won't tell you why—you need to diff the task input properties across versions. I've started keeping a bookmarklet that compares two scans side by side; it has caught more annotation processor misconfigurations than any code review ever did. Quick reality check—if your team hasn't looked at a task graph in six weeks, you're almost certainly accumulating drift. That single misconfigured processor? It grew into five over the next three months. Not because engineers were sloppy, but because nobody had a feedback loop that screamed "your build is lying to you."

What you can do now: Run gradle build --info 2>&1 | grep -E "(UP-TO-DATE|EXECUTING|FROM-CACHE)" > task-outcomes.txt before your next deploy. Count the lines. If you see more EXECUTING than UP-TO-DATE for unchanged modules, you have drift. Fix the oldest violation first—usually an annotation processor that doesn't declare its outputs, or a task that writes timestamps into META-INF. That alone can reclaim two minutes per developer per hour. The build doesn't have to rot.

When You Should Skip Incremental Builds Altogether

Prototyping and rapid experimentation

You're rewriting a parser for the fifth time this afternoon. The types are wrong, the edge case for null inputs just exploded, and you need to see if the new approach even compiles before you lose the thread of your thinking. Incremental builds won't help you here—they will actually slow you down. The task graph, so carefully honed for production speed, now wastes seconds checking UP-TO-DATE markers on classes you already know are stale. I have watched engineers burn twenty minutes across an afternoon because they kept trusting incremental state that was, in fact, subtly poisoned by half-finished refactors. The better move? ./gradlew clean :my-module:compileKotlin. It's faster to rebuild from scratch than to let Gradle reconcile inputs you're deliberately thrashing. Quick reality check—if your edit cycle is under thirty seconds and you're changing type hierarchies or renaming packages, skip the incremental machinery entirely. It was designed for stability, not chaos.

CI environments with ephemeral agents

Your CI spins up a fresh container for every branch build. The build cache is mounted as a read-only volume from the previous run—or worse, it's empty because the agent was recycled. Incremental builds in this context are a mirage. The task graph tries to check file hashes against a local cache that doesn't exist, so every task runs from scratch anyway, but with the additional overhead of scanning inputs and comparing timestamps that are meaningless. Most teams skip this: they leave incremental builds enabled on CI because it feels prudent, then wonder why builds take forty seconds longer than a clean local run. The catch is that you actually accumulate cache misses faster when agents are ephemeral. I have seen three-person teams waste a collective six hours per week on CI builds that were effectively clean builds dressed up in incremental clothing. Disable incremental execution for CI tasks that produce artifacts—your deploy pipeline doesn't need the same input tracking your laptop does.

Reality check: name the development owner or stop.

Tasks that are inherently non-deterministic

Some tasks lie. They claim to be cacheable, they declare their inputs honestly, but something in their implementation produces different output given the same inputs. Code generators that embed timestamps. Annotation processors that read filesystem metadata. Tasks that depend on the order of files in a directory listing—which is not guaranteed across operating systems. Incremental builds break on these tasks the moment the local Git branch changes, because the output from last Tuesday doesn't match the output from today, even though the declared inputs are identical. The task graph thinks everything is fine. It skips re-execution. You get a stale artifact that compiles locally but fails on the next developer's machine. This is the worst kind of bug: silent, sporadic, and nearly impossible to reproduce.

“I spent two days chasing a build failure that only happened on my colleague's machine. Turned out our Dagger annotation processor was not idempotent across warming runs.”

— senior engineer, after disabling incremental annotation processing entirely

How do you catch these? Watch your build/reports/tasks/ for tasks that always show FROM-CACHE but produce different bytecode sizes on the same source. Or just mark them outputs.upToDateWhen { false } and take the two-second hit—it beats the debugging spiral. That sounds drastic, but I have applied this to exactly four tasks in a production build of seventy modules, and our weekly "unexplained red build" count dropped to zero. You don't need incremental builds for everything. You need them only for the things that actually are incremental.

Frequently Asked Questions About Task Graphs

Why does my build run tasks marked UP-TO-DATE?

You watch the console scroll past—task after task stamped UP-TO-DATE—yet the build still takes forty seconds. Something is running, and it isn't incremental. The most common culprit? A task that declares no outputs but depends on one that does. Gradle marks your compile task as cached, then re-runs a downstream task that inspects the entire classpath anyway. I have debugged this exact situation three times this year; each time the fix was adding a missing @OutputFile annotation on a custom plugin. The ten-second check: add --info to your build and look for tasks that say NO-SOURCE or EXECUTING instead of UP-TO-DATE. That seam is where your incremental profit disappears.

Another trap: timestamp mismatches. Gradle's up-to-date detection uses file hashes by default, but if your build mixes lastModified checks with content hashing, the task graph sees phantom changes. We fixed this by standardizing on org.gradle.caching.internal.origin across all custom tasks. Painful afternoon. Worth it.

Can I force a clean build without losing cache?

Short answer: yes, but most teams do it wrong. Running ./gradlew clean build nukes your entire .gradle cache directory—hours of compiled dependencies, gone. What you actually want is --rerun-tasks without clean. That flag forces every task to execute but preserves the build cache for future runs. The catch is it only works if your tasks are properly cacheable. If a task lacks declared inputs and outputs, --rerun-tasks still runs it—but won't save the result for later. I keep a one-liner in my aliases: gw build --rerun-tasks -x clean. That avoids the nuclear option. Most teams skip this—they hammer clean out of habit, then wonder why Monday mornings are rebuild marathons.

“A clean build should be a surgical tool, not a default reflex. You lose dependency cache once a day—that's 30 minutes a week you never get back.”

— lead engineer rewriting their team's CI pipeline after a three-hour outage

How do I trace which task input changed?

The Gradle build scan is your best friend here—but you don't need the enterprise edition. Run with --scan and open the timeline view. Look for tasks that flipped from FROM-CACHE to EXECUTING between two otherwise identical builds. That delta shows you exactly which input property changed. I have caught dependencies escaping through wildcard includes, annotation processors regenerating metadata every compile, and—my personal favorite—a timestamp in a generated file that updated merely because the CI agent's clock drifted. The trick is narrowing the window: run the same commit twice, compare the scans. If you see EXECUTING on a task that shouldn't change, inspect its @Input annotations. One team found a single @InputFiles directory that recursively scanned the entire project root—including build/ itself. Every build invalidated every task. The fix was one exclude filter. That kind of drift compounds silently. Chase it early.

Diagnose Your Build: Next Steps to Try Now

Run a build scan and look for 'EXECUTED' tasks

Open your last build scan. Not the green one you assume is fine—the one that felt just a little too slow. Filter by task outcome. Every orange 'EXECUTED' tile is a signal that Gradle decided it could not reuse cached work. Some of that's legitimate: tests ran, compilation happened. But if you see a task that only concatenates a few text files marked as EXECUTED when nothing changed, you have a leak. I once found a project where a custom Copy task ran every single build because someone had wired project.buildDir as an input—a path that literally changes every build. That one mismatch cost the team 14 seconds per invocation. Fourteen seconds, four times an hour, for six months. Quick reality check—scan the last three scans side by side. Any EXECUTED task that shows up in all three but whose inputs should be stable is now your first target.

Check task inputs with —info logging

Build scans hide the gory details. Drop into your terminal and run ./gradlew assemble --info. Pipe it to a file—this floods stdout fast. Hunt for lines containing 'Caching disabled for task'. Right after that, Gradle explains why. Sometimes it's a missing @InputFiles annotation; sometimes it's a timestamp snuck into the task's output path. Most teams skip this: they see the red EXECUTED in the scan and assume it's a Gradle bug. It's not a bug—it's your own code. The —info output will show you the exact property or file that forced re-execution. That hurts. Fixing it often means moving a timestamp into the task's outputs.cacheIf condition, or replacing a File input with a String that ignores directory changes. Not glamorous. But one line change can drop a 30-second task to zero.

“We spent three days optimizing our build pipeline. The real fix was a single unclosed iterator on a file tree.”

— Engineer, post-mortem comment on a team retrospective board

Experiment with task output relocation

Pick one task that stubbornly stays EXECUTED. Move its output directory outside build/ to a dedicated location—.cache/my-task-output for example. Run the build twice. If the second run finally shows UP-TO-DATE, your problem was cross-task output interference: two tasks writing into overlapping directory trees, invalidating each others' cached state. The catch is that relocating outputs can break IDE importers or CI artifact collectors that expect everything under build/. So do this as a diagnostic, not a permanent move. Another experiment: add outputs.upToDateWhen { true } to a non-critical task just to see if the task graph collapses. It will—and then you will know the task itself is safe to skip, but something else is poisoning its inputs. That narrows your search by half. Try these three experiments this week—one per build session. Your build should feel noticeably crisper by Friday. If not, hunt the 'FROM-CACHE' tasks instead; they're often the silent killer you never saw coming.