Skip to content

Feature request: Allow separately specifying --nthreads and resource limits #63

@kazimuth

Description

@kazimuth

I'm using Dask to coordinate some ML jobs, which have internal parallelism. I'd like to restrict my cluster so that only one job can run on a node at a time, even though nodes have multiple CPUs. However, this is currently impossible, since the Kubernetes worker resource limit is used to customize --nthreads:
https://github.com/dask/helm-chart/blob/master/dask/templates/dask-worker-deployment.yaml#L37-L38

This also prevents giving nodes e.g. 0.5 or 1.5 cpus, because even though Kubernetes allows it Dask will get confused and crash (i.e. --nthreads fails to parse as an integer.)

This is moderately esoteric so if I should just fork the chart and use that I don't mind.

Metadata

Metadata

Assignees

No one assigned

    Labels

    chart/daskRelated to the dask chart

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions