Support custom logs correlation

[ps48]

Description

Many users of trace analytics have different ways of defining their logs index. Many of them do not want set an alias on their existing indices and fields. Hence, we are adding this experimental feature to enable them to add custom log source used for correlation of spans & services to logs.

Redirection to Explorer:

obs-correlated-logs.mov

Redirection to Discover:
NOTE: We see an error in discover to load data from specific indexes using PPL. This looks to be a discover only issue.

mds-correlated-logs.mov

Issues Resolved

#2141

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: Patch coverage is 47.91667% with 25 lines in your changes missing coverage. Please review.

Project coverage is 56.16%. Comparing base (5cfbba1) to head (8815884).
Report is 17 commits behind head on main.

Files with missing lines	Patch %	Lines
...nalytics/components/common/custom_index_flyout.tsx	45.45%	12 Missing ⚠️
...e_analytics/components/common/helper_functions.tsx	50.00%	9 Missing ⚠️
...ace_analytics/components/services/service_view.tsx	0.00%	2 Missing ⚠️
...analytics/components/traces/span_detail_flyout.tsx	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2375      +/-   ##
==========================================
- Coverage   56.53%   56.16%   -0.37%     
==========================================
  Files         393      396       +3     
  Lines       15574    15852     +278     
  Branches     4284     4384     +100     
==========================================
+ Hits         8804     8903      +99     
- Misses       6705     6878     +173     
- Partials       65       71       +6     

Flag	Coverage Δ
dashboards-observability	56.16% <47.91%> (-0.37%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[TackAdam]

Are we intentionally applying the log “Correlated log indicies” to links from “Data Prepper” or should it be isolated to only affecting Traces while under the “Custom Source” tab. It is currently applying to both which seems to be useful but just wanted to make sure that its intended.

[TackAdam]

“View associated logs” redirection seems broken under the new navigation when “Explorer” is not present as it cannot land on the page. Don’t think this was introduced by this PR but restricts the ability to test the functionality with new home enabled.

[ps48]

“View associated logs” redirection seems broken under the new navigation when “Explorer” is not present as it cannot land on the page. Don’t think this was introduced by this PR but restricts the ability to test the functionality with new home enabled.

Did you have MDS enabled and query_enhancements enabled? -> We need this to make sure these are enabled, when explorer is missing.

[paulstn]

Changes look good to me, could we get tests?

[ps48]

Note: We need to fixate of default timestamp in the next upcoming version.