feat: improve Application Insights logging and telemetry handling

[AjitPadhi-Microsoft]

Purpose

• This pull request introduces significant improvements to observability and monitoring in the Content Processing Solution Accelerator. The main focus is on integrating Azure Application Insights and OpenTelemetry tracing, enabling better tracking of pipeline stages and process metadata. Additionally, there are updates to infrastructure templates and dependency management to support these features.

Does this introduce a breaking change?

• Yes
• No

Golden Path Validation

• I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

• I have validated the deployment process successfully and all services are running as expected with this change.

What to Check

Verify that the following are valid

• ...

Other Information