Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Not finished][Do not review] #6823

Draft
wants to merge 20 commits into
base: main
Choose a base branch
from

Conversation

AutoLTX
Copy link

@AutoLTX AutoLTX commented Feb 19, 2025

End-user friendly description of the problem this fixes or functionality that this introduces

  • Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

Give a summary of what the PR does, explaining any non-trivial design decisions


Link of any specific issues this addresses


# Add local metrics to observation
if self.state and self.state.local_metrics:
observation.llm_metrics = copy.deepcopy(self.state.local_metrics)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we added it, instead of observation, to the action? Somewhere

  • after we got it from the agent at line 689
action = self.agent.step(self.state)
  • and just before it's added to the event stream, like before this line 751
self.event_stream.add_event(action, action._source)  # type: ignore [attr-defined]

Then it would be saved in the stream, so the server would be able to read it in its dict it sends to frontend. 🤔

I think we have some difficult problem atm with sending extra bits of information to the frontend, I'm sorry about that. This idea might be one way to do it, and it should work for our purpose.

@enyst
Copy link
Collaborator

enyst commented Feb 22, 2025

@AutoLTX Thank you for the work on this!

Please don't worry about the fact that it's unfinished, it's fine, that's why it's a draft. Everyone does temporary stuff in a draft.

This PR ends up much more complex than we first thought, I apologize about that! We seem to have two difficulties here:

  • (backend) sending extra information to the frontend is non-obvious. There is no channel other than events themselves, and an error callback channel, I think.
  • (frontend) frontend just had or has a redesign, and frontend folks will need to check changes so that we don't introduce inconsistencies or issues with the rest.

What do you say if we split this PR in two, maybe three:

  • backend stuff, see that the info is actually sent; unit tests
  • frontend - display accumulated cost
  • frontend - page with details on cost

That may make it easier to keep in sync with the high activity here, and easier to review and approve.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants