[FEATURE] Dispatch jobs by name, not by Nomad file

[sevagh]

Description

I have a use-case where I have a parametrized batch job in Nomad. I extended levant to add a command called levant dispatch, which takes the name of the job.

Help text:

sevagh:levant $ levant dispatch --help
Usage: levant dispatch [options] [JOBNAME]

  Dispatch an instance of a Nomad parametrized job

Arguments:

  JOBNAME name of Nomad parametrized job

General Options:

  -address=<http_address>
    The Nomad HTTP API address including port which Levant will use to make
    calls.

  -log-level=<level>
    Specify the verbosity level of Levant's logs. Valid values include DEBUG,
    INFO, and WARN, in decreasing order of verbosity. The default is INFO.

  -monitor
    Monitor the job until it completes.

  -timeout
    How long to wait ( in seconds ) for jobs to be successful before giving up.
    Only applies to batch type jobs for now. Default is 300 seconds.

I also borrow code heavily from this other PR which added monitoring for batch jobs: #85

Finally, I have a feature that outputs the stdout and stderr from the Nomad job:

stdout, err := l.nomad.AllocFS().Cat(alloc, "alloc/logs/build.stdout.0", nil)
if err != nil {
      logging.Error("levant/dispatch: error getting stdout: %v", err)
      return
}

All of my code uses standard Nomad APIs.

Basically, there's a lot of material in my custom branch that may or may not be valuable to contribute upstream:

1. Polishing and getting this PR (added in the ability to monitor batch jobs to completion - WIP #85) merged to add monitoring of batch/parametrized jobs
2. Adding a dispatch command to dispatch named jobs instead of submitting HCL files
3. Adding the ability to view stdout/stderr from the job

Are any of these interesting? All?

[jrasell]

hey @sevagh, thanks firstly for opening this issue request and the detail you have put into it. I am certainly keen to have a dispatch wrapper as detailed in #113 and coincidentally is something I was working on late last night just before you raised this issue. I am open to discussions around how to move forward as we both have done work on this, so I am happy if you wish to raise the PR for this feature and then have a review or I can push my code later today and get you to review that?

In terms of the log tracking, again this is something I have been working on but has a little complexity around it. I talked to @dansteen about the log tracking feature and I am keen (to which he agreed) to keep the ability to follow a jobs logs to a separate command. I prefer this approach as it keeps deployment pipelines very simple and easy to observe failures; deploy job > tail job logs and also keep the deployment command solely for deploying jobs.

I would be more than happy to discuss any of this further, and would like to in fact as it certainly helps the project and my thinking.

[sevagh]

I'll defer to you on the log processing, since the Nomad API exposes more nuanced methods (tail, etc.). In my case I just cat a file that I know exists.

I'll PR what I have for the dispatch command (draft/WIP). If there are valuable parts I can split them into separate PRs.

[jrasell]

Closing as duplicate of #113 and #112