Add `--dry-run` option to `mix test`

[whatyouhide]

For the whole discussion around this, see #14389.

The --dry-run option should compose with all other mix test option, but should render all setup callbacks and tests null.

This is going to be helpful for tools (and humans) who want to know a mix test "test plan" but without the burden of executing the tests.

[Nezteb]

I like this idea!

@whatyouhide How do you envision --dry-run would work with --stale in particular? I ask because I've been struggling with minimizing the number of tests being run for a given change in CI (or even locally). If a PR merges to main, it makes sense to run all tests for a large project. For each individual commit to a given PR (or before the PR merges), ideally not all tests are rerun every time.

I usually end up writing elaborate shell scripts to figure out which sets of tests to run, but I like the idea of being able to cache "stale test state" (for lack of a better term) so that you can figure out which tests are "stale" from commit to commit to minimize how often tests are run.

For reference:

• https://elixirforum.com/t/is-there-a-way-to-predetermine-which-tests-and-why-would-be-run-on-mix-test-stale/48325

• elixir/lib/mix/lib/mix/compilers/test.ex

  Lines 119 to 176 in 4fa2240

  	defp set_up_stale(matched_test_files, test_paths, opts) do
  	manifest = manifest()
  	modified = Mix.Utils.last_modified(manifest)
  	test_sources = read_manifest()
  	removed =
  	for {source, _} <- test_sources, source not in matched_test_files, do: source
  	test_helpers = Enum.map(test_paths, &Path.join(&1, "test_helper.exs"))
  	sources = [Mix.Project.config_mtime(), Mix.Project.project_file() | test_helpers]
  	force = opts[:force] || Mix.Utils.stale?(sources, [modified])
  	changed =
  	if force do
  	# Let's just require everything
  	matched_test_files
  	else
  	sources_mtimes = mtimes(test_sources)
  	# Otherwise let's start with the new sources
  	# Plus the sources that have changed in disk
  	for(
  	source <- matched_test_files,
  	not is_map_key(test_sources, source),
  	do: source
  	) ++
  	for(
  	{source, source(external: external)} <- test_sources,
  	times = Enum.map([source | external], &Map.fetch!(sources_mtimes, &1)),
  	Mix.Utils.stale?(times, [modified]),
  	do: source
  	)
  	end
  	stale = MapSet.new(changed -- removed)
  	sources = update_stale_sources(test_sources, removed, changed)
  	test_files_to_run =
  	sources
  	|> tests_with_changed_references()
  	|> MapSet.union(stale)
  	|> MapSet.to_list()
  	if test_files_to_run == [] do
  	write_manifest(sources)
  	{[], nil, nil}
  	else
  	{:ok, pid} = Agent.start_link(fn -> sources end)
  	cwd = File.cwd!()
  	parallel_require_callbacks = [
  	each_module: &each_module(pid, cwd, &1, &2, &3),
  	each_file: &each_file(pid, cwd, &1, &2)
  	]
  	{test_files_to_run, pid, parallel_require_callbacks}
  	end
  	end

[whatyouhide]

@Nezteb I’m not sure what you are asking here. Specifically, I’m not sure whether you're asking if --stale --dry-run would mark the stale tests as executed?

[Nezteb]

@whatyouhide Apologies for my rambling!

tl;dr yes I wonder if your proposed --dry-run flag will essentially work with --stale in a way that could be used in CI pipelines to figure out the minimal set of tests that should run for a given code change. For this, I would assume that the tests themselves wouldn't be able to execute in the same pass. Ideally the output would be somehow usable to determine diffs of stale test lists. For more context of what I'm talking about, I learned about this GH issue from a thread on the mailing list.

[whatyouhide]

@Nezteb yes, --dry-run would not execute tests. It would essentially print out tests that would run whatever the rest of the options to mix test are, without executing the tests. It would work with --stale, --exclude, --only, --failed, and so on.

[josevalim]

Closing in favor of #14499.