run gather in background

[SeanMooney]

run_bg can only be used to run executables in the backgorund
not bash functions. This change adds a wait_for_bg_slot funciton
and refactors the sos report gathering to run in parallel up to
the concurrancy limit.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign fmount for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[bogdando]

/lgtm

[Akrog]

I think it would be good to have "why" the run_bg function is not appropriate in the commit message.

[gibizer]

I've opened a tracker Jira for this https://issues.redhat.com/browse/OSPRH-7830
The bug only appears if the strict error checking is enabled in bash, but we cannot enable that globally in the scripts due to other error handling shortcomings.

[Akrog]

Thanks @gibizer I think that context is needed in the commit message and the PR description.
From the existing content I thought there was a bad behavior of the script, which doesn't seem to be the case.
Though I agree with the intent of the PR to make changes to enable strict error checking.