Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SL-2831] Switch target info logs to debug #1490

Open
Jstein77 opened this issue Oct 30, 2024 · 0 comments
Open

[SL-2831] Switch target info logs to debug #1490

Jstein77 opened this issue Oct 30, 2024 · 0 comments
Assignees
Labels
High priority Created by Linear-GitHub Sync Medium Created by Linear-GitHub Sync Metricflow Created by Linear-GitHub Sync starter task Created by Linear-GitHub Sync
Milestone

Comments

@Jstein77
Copy link
Contributor

We have a lot of logger.info() calls inside of MetricFlow that flood our datadog logs. There are a couple of places where these are essentially printing object processing info, but are doing so inside of a method that's being invoked within a loop or recursive process. This means complex semantic manifests or queries against large numbers of metrics could end up spitting out thousands of these log lines, which are mainly useful for tracking progress during development.

This task is to find and update a small number of high volume logs of this type. You can find them by doing datadog searches for high volume logs (simply scrolling through a reasonable subset of the last day's worth of MT logs for MFS should give you some ideas), or just debugging some query and being like "whoa why are there 9 pages of this same message here?!". You can also find them by looking carefully at MetricFlow's codebase and realizing that they're invoked inside of some nested loop someplace. Things to look out for are nondescript info messages that are called inside of deep for loops in the DataflowPlanBuilder, or that are called in any visit_* method in a visitor pattern object.

Examples:

  1. The "Handling {node.ui_description}" info logs in the push down visitor can just be switched to debug.
  2. Filtering valid linkable element time reporting - this might've been a temporary thing we forgot to remove, actually, in which case we can get rid of the time tracking altogether

More broadly, if we get datadog tracing set up we can probably switch all timing logs in MetricFlow to debug, since the function span views should cover most of what we need there.

From SyncLinear.com | SL-2831

@Jstein77 Jstein77 added High priority Created by Linear-GitHub Sync Medium Created by Linear-GitHub Sync Metricflow Created by Linear-GitHub Sync starter task Created by Linear-GitHub Sync labels Oct 30, 2024
@Jstein77 Jstein77 added this to the v0.207.0 milestone Oct 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
High priority Created by Linear-GitHub Sync Medium Created by Linear-GitHub Sync Metricflow Created by Linear-GitHub Sync starter task Created by Linear-GitHub Sync
Projects
None yet
Development

No branches or pull requests

2 participants