Skip to content

Conversation

@Klafyvel
Copy link

This is related to #68. Patching TiffImages.jl to accept arbitrary IO subtype might be complex: they rely a lot on seek, which is not compatible with the CodecZlib output I am targeting.

An alternative is to decompress the file to disk and use a mmaped io, which is interesting anyway. So I modified the tiff transformer so it becomes able to do that. A new difficulty is that although the tests seem to pass, I am not able to test it for a real case due to an error that I am struggling to understand.

Given the following dataset:

[[molecules]]
uuid="fdfa6503-604b-47ef-b304-a4c0f4e0bb2c"
description = "Single molecules lighting up as a laser gets in resonance with their main electronic transition."

	[[molecules.storage]]
	driver = "web"
	checksum = "sha1:e12f9ac21226c140d8e936b694f36ba0dcc11a3a"
    url = "https://klafyvel.me/public_upload/long_cool_scan_wf_5uW_res_3p8K_2by2_10xGain.tif.gz"
    type = "IO"
	[[molecules.loader]]
    driver = "chain"
        [[molecules.loader.loaders]]
        driver = "gzip"
        [[molecules.loader.loaders]]
        driver = "io->file"
        [[molecules.loader.loaders]]
        driver = "tiff"
        mmap = true

I am getting this error:

ERROR: UnsatisfyableTransformer: There is no storage for "molecules" that can provide a Any.
 The defined storages are as follows:
   DataStorage{web}(IO) -> [IO, String, Vector{UInt8}, DataToolkitCore.FilePath, Any]

Stacktrace:
 [1] read1(dataset::DataToolkitCore.DataSet, as::Type)
   @ DataToolkitCore ~/.julia/packages/DataToolkitCore/B5vZ5/src/interaction/externals.jl:220
 [2] read(dataset::DataToolkitCore.DataSet, as::Type)
   @ DataToolkitCore ~/.julia/packages/DataToolkitCore/B5vZ5/src/interaction/externals.jl:151
 [3] read(dataset::DataToolkitCore.DataSet)
   @ DataToolkitCore ~/.julia/packages/DataToolkitCore/B5vZ5/src/interaction/externals.jl:176
 [4] macro expansion
   @ ~/.julia/packages/DataToolkitBase/iPXwX/src/DataToolkitBase.jl:23 [inlined]
 [5] top-level scope
   @ REPL[2]:1

I still need to understand if I did something wrong in my patch or if that's a bug.

@tecosaur tecosaur force-pushed the main branch 2 times, most recently from 9664136 to 423b571 Compare July 25, 2025 07:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant