In python, I have a task which I build like so:
# my_script.py
build_success = luigi.build( [MyTask()], workers=1, local_scheduler=False, scheduler_host='pod-A' )
This process runs in a kubernetes pod "pod-B" when I run kubectl exec -it pod-B -- python my_script.py
When I Ctrl+C out of the kubectl exec process, I expect MyTask() to stop as a result. When I do not provide scheduler_host and set local_scheduler=True this works as expected, but with a scheduler_host set, I am unable to stop MyTask from continuing to execute, despite having exited the kubectl process and the python process on pod-B exiting.
pod-A is running luigid.
Does anyone know how this is supposed to work, if this a bug or if I am doing something incorrect here? I can't even find where this process continues to run. We think the luigi daemon somehow absorbs the job and continues executing it despite the original executor stopping. I cannot find documentation about this specific interaction online.
I am on luigi version 3.5.2
In python, I have a task which I build like so:
This process runs in a kubernetes pod "pod-B" when I run
kubectl exec -it pod-B -- python my_script.pyWhen I
Ctrl+Cout of the kubectl exec process, I expectMyTask()to stop as a result. When I do not providescheduler_hostand setlocal_scheduler=Truethis works as expected, but with ascheduler_hostset, I am unable to stopMyTaskfrom continuing to execute, despite having exited the kubectl process and the python process onpod-Bexiting.pod-Ais runningluigid.Does anyone know how this is supposed to work, if this a bug or if I am doing something incorrect here? I can't even find where this process continues to run. We think the luigi daemon somehow absorbs the job and continues executing it despite the original executor stopping. I cannot find documentation about this specific interaction online.
I am on luigi version 3.5.2