Tasks not cached

[tduehr]

Running dependency check on a large project takes quite a while. Gradle has build caching to alleviate this problem for tasks that have not changed. Dependency check does not properly declare its inputs or outputs to enable this. Further, things that could be cached (like dependency -> cpe resolutions) are not either.

Each dependencyCheckAnalyze task should declare its inputs and outputs so the plugin can properly advise Gradle for build caching. Cache misses may also be reduced by further breaking up the tasks into sub tasks (e.g., dependency resolution, and CVE lookup) that may be cached separately. This way, an update from NIST (for example) need not invalidate all cached tasks, only the CVE lookup.

The dependencyCheckAggregate task should depend on the Analyze tasks to leverage this caching and allow parallel processing. Only the aggregation step of this task cannot be parallelized.

[jeremylong]

Defining the inputs and outputs has been on my todo list for a while. I'll have to consider caching some of the other items listed.

[jeremylong]

One of the issues I've been trying to figure out is how one would make the incoming configuration an input (in terms of gradle's incremental build). What makes this more complex is the difference between the modern way of resolving dependencies as opposed to the legacy method.

[tduehr]

Use a flag file generated by the dB update task and the project’s ‘build.gradle’ file. Not the most accurate but good enough to start.

Possibly also an intermediate task that dumps a dependency tree somewhere.

[skjolber]

Any progress on this? We're now mostly running dependency checks in the build pipeline, because running locally is a bit too much time consuming.

[jeremylong]

@skjolber in all honesty I keep looking at this from time to time and while defining the @Output is easy trying to figure out what the best @Input is a challenge. Considering @classpath to capture the list of resolved dependencies:

  @Classpath
  @InputFiles
  FileCollection classpath 

We also have to consider the scan set which is going to need to consider the differences for an aggregate build vs the standard analyzer: https://github.com/jeremylong/dependency-check-gradle/blob/724dfab1cb3c863a1aaa9c2dc6130ec54476b17c/src/main/groovy/org/owasp/dependencycheck/gradle/tasks/AbstractAnalyze.groovy#L328-L335

Lastly, when implementing the caching we have to consider the timeliness of the datasources. By default ODC will update the h2 database every 4 hours. So technically, we should only cache build results for 4 hours.

[gmariotti]

Hi @jeremylong, any update on this issue?

[jeremylong]

nope - we accept PRs

[jeremylong]

Implementation to support this is in #473 and the draft PR #474. This is being tracked under #339 as gradle is forcing the implementation to use caching ;)

[runningcode]

There is a difference between configuration caching and build caching. 339 is about configuration caching, this issue is about build caching.
To be specific, this issue is about enabling task level caching for the dependency check tasks. You can enable caching by

1. marking all inputs correctly.
2. marking all output correctly.
3. marking the task as cacheable with the @CacheableTask annotation.

None of the above linked PRs do this.

[jeremylong]

The PRs do everything but mark the tasks with @CacheableTask. I'll create another PR for this.