Paulo/cli-telemetry

[rossirpaulo]

Pull Request Template

Added CLI telemetry to baml-cli

Summary by CodeRabbit

Release Notes

• New Features

  • Added telemetry collection to the Engine CLI. Telemetry can be disabled via the BAML_CLI_DISABLE_TELEMETRY environment variable.

• Documentation

  • Added reference documentation describing telemetry behavior, captured data, privacy guarantees, and disablement instructions.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
beps	Ready	Preview, Comment	Feb 23, 2026 5:31pm
promptfiddle	Ready	Preview, Comment	Feb 23, 2026 5:31pm

[coderabbitai]

📝 Walkthrough

Walkthrough

A PostHog-based telemetry system is added to the Engine CLI that captures command-start events with minimal data (command name, CLI version, caller info, CI context, project hash, machine ID). Events are sent asynchronously without affecting command execution and can be disabled via the BAML_CLI_DISABLE_TELEMETRY environment variable.

Changes

Cohort / File(s)	Summary
Telemetry Implementation engine/cli/src/lib.rs, engine/cli/src/telemetry.rs	New telemetry module added with command-start event capture. Implements project hash computation, CI provider detection, machine ID persistence, and async PostHog payload sending. Includes comprehensive tests for disablement, LSP command suppression, CI detection, and payload field validation. Module integrated into CLI run path with argv cloning and event invocation before command execution.
Telemetry Documentation docs/engine-cli-telemetry-implementation-plan.md, fern/03-reference/baml-cli/telemetry.mdx, fern/docs.yml	Implementation plan document outlining module structure and integration points. User-facing reference documentation describing captured data, exclusions, disablement mechanism, and privacy guarantees. Navigation entry added to docs.yml for the new telemetry reference section.

Sequence Diagram

sequenceDiagram
    participant CLI as Engine CLI
    participant Telemetry as telemetry module
    participant ProjectHash as Project Hash<br/>Computation
    participant CIDetect as CI Provider<br/>Detection
    participant MachineID as Machine ID<br/>Persistence
    participant PostHog as PostHog API

    CLI->>Telemetry: capture_command_started(argv, command, caller_type)
    
    alt Telemetry disabled or LSP command
        Telemetry-->>CLI: return early
    else Proceed with event capture
        Telemetry->>ProjectHash: compute_project_hash(--from, cwd)
        ProjectHash-->>Telemetry: project_hash, source
        
        Telemetry->>CIDetect: detect_ci_provider()
        CIDetect-->>Telemetry: ci_provider variant
        
        Telemetry->>MachineID: read_or_create_machine_id()
        MachineID-->>Telemetry: machine_id
        
        Telemetry->>Telemetry: build TelemetryEvent payload
        Note over Telemetry: Includes: command, subcommand, version,<br/>caller, CI info, project hash,<br/>machine/session IDs, OS/arch, argv length
        
        Telemetry->>PostHog: spawn_send_event (async HTTP POST)<br/>timeout: ~5s, fire-and-forget
        PostHog-->>Telemetry: event sent
        
        Telemetry-->>CLI: return (event sent async)
    end
    
    CLI->>CLI: continue command execution

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 23.81% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check	❓ Inconclusive	The title 'Paulo/cli-telemetry' is a branch name, not a descriptive pull request title. It does not clearly summarize the main change of implementing CLI telemetry for the baml-cli.	Rename the title to clearly describe the main change, such as 'Add CLI telemetry to baml-cli' or 'Implement PostHog-based telemetry for Engine CLI'.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

• 📝 Generate docstrings (stacked PR)
• 📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch paulo/cli-telemetry

Tip

Issue Planner is now in beta. Read the docs and try it out! Share your feedback on Discord.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

https://sage-backend-kf03lmxc8-baml.vercel.app

[github-actions]

🌿 Preview your docs: https://boundary-preview-f38c3db7-a93d-42fe-9c7b-be0cf47d26d8.docs.buildwithfern.com

[coderabbitai]

Actionable comments posted: 5

---

ℹ️ Review info

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 6133e3f and d172216.

📒 Files selected for processing (5)

• docs/engine-cli-telemetry-implementation-plan.md
• engine/cli/src/lib.rs
• engine/cli/src/telemetry.rs
• fern/03-reference/baml-cli/telemetry.mdx
• fern/docs.yml

[coderabbitai]

🧹 Nitpick | 🔵 Trivial

Thorough implementation plan — well-structured and aligned with the code.

The plan clearly documents scope, non-goals, event contract, privacy rules, and testing strategy. All described behaviors match the actual implementation in telemetry.rs.

One consideration: since this is committed as a file, it may drift from the implementation over time. Consider adding a note that the source of truth is the code itself, or moving this to a wiki/ADR with a link in the codebase.

🧰 Tools

🪛 LanguageTool

[style] ~261-~261: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...etry failure cannot alter exit code. 3. Telemetry cannot panic process path. ## 9. Priva...

(ENGLISH_WORD_REPEAT_BEGINNING_RULE)

---

[style] ~267-~267: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...filesystem path values are exported. 3. No sensitive env values are exported. 4. D...

(ENGLISH_WORD_REPEAT_BEGINNING_RULE)

[rossirpaulo]

will delete it instead

[coderabbitai]

@rossirpaulo, sounds good! Removing the implementation plan document works too.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Find the GeneratorOutputType enum definition
rg -n 'enum GeneratorOutputType' --type rust -A 50

Repository: BoundaryML/baml

Length of output: 4055

---

Add mapping for the Rust generator output type.

The map_caller_runtime function handles 7 of 8 variants from the GeneratorOutputType enum. The Rust variant is missing and will silently map to "unknown" in telemetry, reducing data accuracy. Additionally, the non-exhaustive match pattern using _ => "unknown" masks this issue—any future enum variants added will also fall through silently.

Replace the wildcard with an explicit mapping for Rust (e.g., => "rust") and consider using #[non_exhaustive] on the enum or refactoring to an exhaustive match to catch missing variants at compile time.

[coderabbitai]

🧹 Nitpick | 🔵 Trivial

Unnecessary set_machine_id_path_override in this test.

build_command_started_event takes machine_id as a parameter and does not call get_or_create_machine_id, so the override on Line 642 is never exercised. Consider removing it to avoid misleading future readers.

Proposed cleanup

         let _cwd_guard = CwdGuard::set(&temp_root);
 
-        let machine_path = temp_root.join("telemetry_machine_id");
-        set_machine_id_path_override(Some(machine_path));
-
         let sensitive_path = temp_root.join("private").join("sensitive").join("baml_src");

         );
-        set_machine_id_path_override(None);
 
         let value = serde_json::to_value(&payload).expect("payload should serialize");

Also applies to: 666-666

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Missing required frontmatter.

This .mdx file must start with frontmatter containing title and subtitle. As per coding guidelines: "Every .mdx file must start with frontmatter containing title and subtitle in the specified format."

Proposed fix

+---
+title: Telemetry
+subtitle: Learn how to manage CLI telemetry in baml-cli
+---
+
 `baml-cli` emits one telemetry event when a command starts, except for `lsp`.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Section headings should use sentence case.

Per coding guidelines, titles should have "the first word capitalized followed by lowercase unless it is a name." The following headings should be updated:

• ## Event Behavior → ## Event behavior
• ## What Is Captured → ## What is captured
• ## What Is Not Captured → ## What is not captured
• ## Disable Telemetry → ## Disable telemetry

As per coding guidelines: "Titles must always start with an uppercase letter, followed by lowercase letters unless it is a name."

Also applies to: 12-12, 25-25, 31-31