[9.2] (backport #11105) [beatreceiver] support loadbalace:true option

[mergify]

What does this PR do?

This PR primarily supports output.loadbalance:true option on beatreceivers. It also adds support for input.[filestream].pipeline parameter by enabling logs_dynamic_pipeline setting on elasticsearch exporter. Related PR on beats where the support was added.

output.pipeline and output.pipelines has never worked with standalone elastic-agent and hence we do not support it in beatreceivers either.

Why is it important?

To achieve feature parity between beats and beatreceiver's output settings

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

None

How to test this PR locally

Related issues

• Relates [Beats receivers] Add exporter translation support for all currently unsupported Elasticsearch output features beats#46749

---

This is an automatic backport of pull request #11105 done by [Mergify](https://mergify.com).

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: f5d4c30

Failed CI Steps

• Win2025:sudo:default
• Win2025:sudo:default
• Win2025:sudo:default
• Win2025:sudo:fleet

History

cc @khushijain21

[mergify]

This pull request has not been merged yet. Could you please review and merge it @khushijain21? 🙏