fix(metrics): gracefully handle DogStatsd client creation failures

[sentry]

Fixes SNUBA-9Y5. The issue was that: DatadogMetricsBackend's threading.local() client access during concurrent ThreadPoolExecutor shutdown causes C-level SystemError.

• Import the logging module and initialize a logger instance.
• Modify the __client property to catch exceptions during DogStatsd client factory calls.
• Log a warning and return None if the client factory fails, preventing application crashes during thread cleanup.
• Update all metric reporting methods (increment, gauge, timing, distribution, event) to check if the DogStatsd client is None and fail silently if it is, avoiding errors when the client could not be initialized.

This fix was generated by Seer in Sentry, triggered by pierre.massat@sentry.io. 👁️ Run ID: 10536044

Not quite right? Click here to continue debugging with Seer.

Legal Boilerplate

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.

[sentry]

Bug: If __client_factory fails, self.__thread_state.client is not set. Subsequent metric calls in the same thread will repeatedly and unsuccessfully attempt to create the client, causing log spam.
_{Severity: CRITICAL}

Suggested Fix

In the except Exception block where the client factory failure is handled, explicitly set self.__thread_state.client = None. This ensures the attribute is set, preventing subsequent calls from re-triggering the client creation logic and the associated AttributeError.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: snuba/utils/metrics/backends/datadog.py#L46-L56

Potential issue: In the `__client` property, if the `self.__client_factory()` call fails
with an exception, the assignment to `self.__thread_state.client` does not complete.
Because the attribute remains unset, any subsequent call to a metric method within the
same thread will trigger another `AttributeError`. This leads to a cycle where the code
repeatedly attempts to create the client, fails, and logs a warning with a full stack
trace for every single metric invocation. This will cause significant log spam and
performance degradation in high-throughput paths.

_{Did we get this right? 👍 / 👎 to inform future reviews.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.}

[cursor]

Outer except only catches AttributeError, not SystemError

High Severity

The PR description states the issue is that threading.local() client access during ThreadPoolExecutor shutdown causes a C-level SystemError. However, the outer try/except on line 40-42 only catches AttributeError. If the SystemError occurs when reading self.__thread_state.client (line 41) — which is exactly the scenario described — it won't be caught, and the crash the PR intends to fix will still happen. The inner except Exception only protects the factory call path, not the attribute access path. The outer except needs to also catch Exception (or at least SystemError) to fully address the described issue.