Add flexible worker management, pools and onworkers

[oschulz]

Follow-up of #10

[oschulz]

@theHenks, you can give this a try now, I think.

[oschulz]

Example:

using ParallelProcessingTools

@always_everywhere begin
    using Distributed
    function mytask(runtime::Real = 2, args...)
        sleep(runtime)
        #@info "Hello from worker $(myid()), have to do $args."
        return args
    end
end

addworkers(SlurmRun(slurm_flags = `--ntasks=92 --cpus-per-task=2 --mem-per-cpu=8G`))

processing_task = @async begin
    @sync begin
        # Let's assume directories can be processed in parallel:
        for diridx in 1:20
            @async begin
                @info "Processing dir $diridx."
                # Let's assume files can be processed in parallel:
                @sync for fileidx in 1:30
                    @async begin
                        @info "Processing run $diridx, file $fileidx."
                        # Steps that have to be done sequentially:
                        for stepidx in 1:4
                            @info "Processing run $diridx, file $fileidx, step $stepidx."
                            result = on_free_worker(mytask, 2, fileidx, stepidx, label = "someaction-$diridx-$fileidx-$stepidx", tries = 2)
                            @assert result == (fileidx, stepidx)
                        end
                        @info "Done with run $diridx, file $fileidx."
                    end
                end
                @info "Done with dir $diridx."
            end
        end
    end
    @info "Processing sucessful."
end

[codecov]

Codecov Report

Attention: Patch coverage is 71.29300% with 242 lines in your changes are missing coverage. Please review.

Project coverage is 69.84%. Comparing base (eb1cd88) to head (4805bb6).

Files	Patch %	Lines
src/workerpool.jl	70.80%	47 Missing ⚠️
src/custom_cluster_managers.jl	55.68%	39 Missing ⚠️
src/runworkers.jl	63.44%	34 Missing ⚠️
src/display.jl	0.00%	28 Missing ⚠️
src/onworkers.jl	71.42%	28 Missing ⚠️
src/procinit.jl	82.17%	23 Missing ⚠️
src/slurm.jl	0.00%	17 Missing ⚠️
ext/ParallelProcessingToolsThreadPinningExt.jl	0.00%	12 Missing ⚠️
src/exceptions.jl	69.23%	4 Missing ⚠️
src/states.jl	92.15%	4 Missing ⚠️
... and 2 more

Additional details and impacted files

@@             Coverage Diff             @@
##             main      #11       +/-   ##
===========================================
+ Coverage   48.31%   69.84%   +21.53%     
===========================================
  Files          11       19        +8     
  Lines         563     1227      +664     
===========================================
+ Hits          272      857      +585     
- Misses        291      370       +79     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[oschulz]

Closed in favor of #13 .