method: timestamp not running task if generated file is missing?

[vaizki]

New Tasker here, this must be some kind of FAQ but I can't find an answer...

So I have a task which extracts the first 512 bytes of an 12 GB AlmaLinux ISO image (MBR) for later rebuilding the image with new files. Because the source is so big, I used method: timestamp to check if the source has changed.. but if I run the task, then remove the generated file and run again, the task is skipped as up-to-date.. which makes no sense as the missing generated file is definitely not up-to-date.

Then if I remove the method: timestamp it is definitely taking the time to checksum the ISO and will run also if the generated file is missing.

I must be fundamentally misunderstanding something here because the two methods operate so differently?

tasks:
  ...
  extract_iso_mbr:
    deps: [ fetch_stock_iso, tmp_dir ]
    # hashing the whole ISO image every time is meh, just use timestamp to check
    method: timestamp
    sources:
      - "{{.ISO_DIR}}/{{.ALMA_ISO_FILE}}"
    generates:
      - "{{.TMP_DIR}}/isohdpfx.bin"
    cmds:
      - dd if={{.ISO_DIR}}/{{.ALMA_ISO_FILE}} bs=512 count=1 of={{.TMP_DIR}}/isohdpfx.bin

I discovered a workaround, that by adding a custom status: to check for the file it works:

    status:
      - test -f "{{.TMP_DIR}}/isohdpfx.bin"

.. but this seems superfluous? Also the docs in the schema for status states:

This overrides method, sources and generates.

.. so I thought you can't combine them but apparently it works fine, doing both the timestamp check on sources and my custom status check on the generated file. Are the docs out of date here? For the checksum method this extra check is not necessary.

Using version 3.44.0 if that makes a difference. And yes this task is so short it doesn't matter if it runs always but that's not "clean" and I will surely run into this issue again very soon. And yes I have 30 years of make background...

[trulede]

Its a bug with the timestamp method. Task does not actually check the generates items exist, rather, it cunningly 'globs" each generates item and only notices those files which exist, and the timestamp file itself. That means, if you delete a file, task will not notice and because the timestamp file, created when the source last changed, was considered for the check ... another cunning move ... it will appear that the task is up-to-date.

The glob function needs to be reworked. If a glob (i.e. one of the generate items) does not find at least one match, then it should error. That will then trigger the task to run.

task/internal/fingerprint/sources_timestamp.go

Line 35 in 281edfe

generates, err := Globs(t.Dir, t.Generates)

Problem is well known:
#2181
#2313

[trulede]

I've suggested a fix for the problem: #2317