mirai today works very well on HPC clusters when run on a login/submit node. Many improvements have been made over the last year or so to make this very user-friendly. Thanks to NNG, the scalability of computations is absolutely amazing !
In some cases however users would like to use a standalone server (say Posit Connect or a Shiny Server) to offload some computations onto a HPC cluster. One obvious way how this can be achieved is to make the Connect/Shiny server a submit node of the HPC cluster. Due to security implications this approach is not always desirable let alone allowed - ssh connections on the other hand are a classic and secure way of accessing an HPC cluster. Manual scripting of such an ssh ==> mirai@HPC approach may be feasible but definitely not for the faint-hearted.
It would be great to get a feature similar to what clustermq offers, the ability to transparently allow ssh and slurm to coexist within the framework and with little to no configuration need by a data scientist (cf. https://github.com/sol-eng/Workbench-HPC-Webinar/tree/main/remote-clustermq)
Happy to be an early tester of such a feature.
mirai today works very well on HPC clusters when run on a login/submit node. Many improvements have been made over the last year or so to make this very user-friendly. Thanks to NNG, the scalability of computations is absolutely amazing !
In some cases however users would like to use a standalone server (say Posit Connect or a Shiny Server) to offload some computations onto a HPC cluster. One obvious way how this can be achieved is to make the Connect/Shiny server a submit node of the HPC cluster. Due to security implications this approach is not always desirable let alone allowed - ssh connections on the other hand are a classic and secure way of accessing an HPC cluster. Manual scripting of such an ssh ==> mirai@HPC approach may be feasible but definitely not for the faint-hearted.
It would be great to get a feature similar to what
clustermqoffers, the ability to transparently allow ssh and slurm to coexist within the framework and with little to no configuration need by a data scientist (cf. https://github.com/sol-eng/Workbench-HPC-Webinar/tree/main/remote-clustermq)Happy to be an early tester of such a feature.