feat: LangSmith-based LLM cost tracking for Number Cruncher#212
Merged
Conversation
Guard prepare-commit-msg so the interactive commitizen / breaking-change prompts auto-skip when there is no TTY (CI/agents) or HUSKY_NONINTERACTIVE=1 is set; the subshell /dev/tty probe avoids aborting the hook under dash. commitlint still enforces message format, so nothing is weakened. Add root CLAUDE.md documenting the non-interactive commit/push workflow and checks, and tests/CLAUDE.md documenting the Jest ESM test-suite conventions.
BedrockChat never set usage_metadata on Anthropic responses, so every Bedrock run ingested into LangSmith with 0 tokens and null cost, and its inference-profile model ids don't match LangSmith's pricing table.
Tags every LLM trace root (agent respond, onConversationEvent, and the stop-time conversation summary) with conversationId and a costPhase of liveEvent or postEvent, so a later cost fetcher can attribute and split spend per conversation and per phase.
Fetches LLM cost per conversation from LangSmith, split into liveEvent and postEvent aggregates by each trace's costPhase tag, with settle polling and a combineCostAggregates helper for building a display total.
Adds a ConversationCost model and upserting service that store the two cost phases as separate sub-documents, since LangSmith retention is only ~2 weeks and the phases must stay disaggregated for later reporting.
Grants numberCruncher the allPublicTopics read grant so the dispatcher routes conversationStopped events to it. On a public event's stop, it fetches the event's LangSmith cost split by phase, persists both phases, and posts a card with the combined total.
Renders the combined LLM cost estimate, a during/after phase breakdown, and per-model/per-agent tables sourced from the combined total.
security/detect-unsafe-regex kept flagging normalizeBedrockModelName's regex, even simplified to two single-quantifier patterns. Split/index based parsing is equivalent and eslint-clean.
Unlike the existing agentDispatcher tests (which fake Agent.find and assertCanRead), this exercises the real capabilities.ts + access chain against real Mongo documents, so a wrong grant type would be caught here instead of only failing silently in production.
null total_cost (e.g. self-hosted vLLM/Ollama models with no pricing-table entry) was being coerced to 0 by numberField(), indistinguishable from a genuinely free call. Adds priced/hasUnpricedCalls tracking through the fetcher, persisted record, and Slack card, plus a code note on a possible future custom-pricing adapter for unpriced platforms.
Cost tracking is core functionality for every conversation, not something that should require the Number Cruncher Slack bot to be provisioned. Extract the compute-and-persist flow into a shared conversationCostTracking service and run it from a standalone conversationCost job scheduled on every stop, gated by the new ENABLE_CONVERSATION_COST_TRACKING flag (default true). - doStopConversation schedules the conversationCost job for every stopped conversation, public or private. - The job steps aside when an active Number Cruncher agent exists, since that agent runs the same flow itself (via onConversationEvent) so it can also post the Slack cost card; running both would duplicate the settle-poll and record. - The pending record now carries a real preliminary LangSmith estimate rather than a zeroed placeholder, falling back to zeros only when nothing has landed. - Skip the whole flow up front when LangSmith tracing is not configured (LANGSMITH_TRACING_V2 + key + project), instead of running a dead settle-poll. - Number Cruncher's onConversationEvent is reduced to the Slack-facing part: call trackConversationCost, then build and post the card. Update SETUP.md to describe the decoupled architecture and the new flag.
Confirming the Slack cost-card integration previously meant waiting for a real event to stop and for LangSmith to settle. Add a probe that exercises the exact outbound path without either: it reads NC's stored Slack credentials, validates the bot token (auth.test), renders a sample card with the real card renderer, and posts it to NC's channel (chat.postMessage). - Flags: --auth-only, --private, --dry-run, --conversation=<id>. - Slack API errors map to an actionable fix (invite the bot, correct the channel id, add chat:write, re-provision the token). - Exposed as `yarn check:number-cruncher-slack`. Document both this fast Slack-only check and the full end-to-end cost-calculation verification in SETUP.md.
The settle-poll declared "settled" as soon as two consecutive LangSmith reads showed the same non-zero count, with only a ~60s floor. A stable count seen that early can just mean a slow post-event agent (e.g. the Vibes Analyst recap) hasn't started spending yet, so the card could post an undercounted total. - fetchConversationCostWithSettle now takes a minimumWaitMs (default 120_000) and never returns a settled result before that wall-clock has elapsed; it still returns once the delay budget is exhausted. - Lower all conversationCost and numberCruncher progress logs from info to debug (pending/preliminary/settle/exhaust lines and the private-topic note); errors and warnings are unchanged.
jencompgeek
approved these changes
Jul 21, 2026
jencompgeek
left a comment
jencompgeek
left a comment
Collaborator
There was a problem hiding this comment.
Tested locally with manual matching of the various traces in Langsmith to data sent to Slack. Everything seems to be working great! Awesome addition!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is in this PR?
This PR adds LangSmith-based LLM cost tracking to conversations, wired through the
Number Cruncher (NC) Slack bot. Every conversation stop now produces a
ConversationCostrecord split into
liveEvent(spend while the conversation was running) andpostEvent(spend on after-the-fact work like the Vibes Analyst recap and conversation summary), and
NC posts a phase-split Slack summary card to its admin channel. Tracking also extends to
private-topic events, since real money is spent regardless of privacy, without giving any
other agent broader access to private topics.
Changes in the codebase
conversationIdand acostPhase(liveEvent/postEvent)run counts to stabilize (up to ~7 minutes, with a 2-minute minimum wait)
ConversationCostrecords withliveEvent/postEventtotals; apendingrecord (a real preliminary estimate, not zeroed) is written immediately on stop and
flipped to
completeonce the settle-poll resolvesconversationCostjob runson every stop, gated by
ENABLE_CONVERSATION_COST_TRACKING; NC only adds the Slackcard, and the standalone job detects an active NC and steps aside to avoid a duplicate
settle-poll
conversationCostSummarySlack card with the combined total and phase breakdownallTopicsread grant soliveEventcost is still recorded for private conversations; every other agent stays
allPublicTopics-onlyprivate-topic events; the persisted Mongo record keeps the real name, tagged
topicIsPrivate: truesurfaced to LangSmith for cost estimation
yarn check:number-cruncher-slacksetup-check script (auth-only, dry-run,private, and specific-conversation modes) and warns on adapter/conversation config
drift
channel_not_found,not_in_channel, etc.) from theweb-api client, including a private-channel invite hint
src/agents/numberCruncher/SETUP.mdto document the new cost-tracking flowand the setup-check script
Documentation and automated testing
Did you:
Testing this PR
SETUP.md:yarn check:number-cruncher-slack(Slack-only path, posts a sample cost card toNC's channel), then the full cost-calculation path if LangSmith is configured.
phase-split (
liveEvent/postEvent) cost card posts to NC's channel with the realevent name, and a
ConversationCostrecord moves frompendingtocompleteinMongo.
liveEventcost is still recorded, the Slack card and fallback text show "Privateevent", and the Mongo record keeps the real event name tagged
topicIsPrivate: true.Additional information
Bedrock regional discounts differ), so the card always labels them as estimates.
historical events will never produce a record or card.
main): noCritical/Important issues.
yarn test: 97/98 suites, 1554/1555 tests passing (sole failure,openapi-types-generate.test.ts, requires a running local dev server on port 3000 —pre-existing, unrelated to this branch).
yarn lint/yarn build: clean.