Single Agent: Computer Use Integration

[f-trycua]

Computer Use Integration

Hi @markwallace-microsoft, @giles17 and @peterychang! I'd like to help with the Computer Use implementation mentioned in issue #377.

I maintain Cua, which handles cross-platform desktop automation on local virtual environments (Windows sandbox, macOS VMs via Apple Virtualization.Framework, Linux on Docker). It supports 100+ VLMs configurations (OpenAI Computer Use Preview, Anthropic Claude, OpenCUA, InternVL, GLM, UI-Tars, local Ollama) and provides primitives for mouse/keyboard control, screenshots, and shell commands.

windows-sandbox.mp4

Proposed Integration Approach:

I see Computer Use could fit into the Agent Framework in a few ways:

1. As a Tool (like MCP tools in Python: Add MCP tool support to Foundry chat client #595):

   from agent_framework import ChatAgent
   from agent_framework.tools import ComputerUseTool
   
   computer = ComputerUseTool(provider="windows", name="windows-sandbox")
   agent = ChatAgent(
       chat_client=OpenAIChatClient(),
       tools=[computer],
       instructions="You are a desktop automation assistant."
   )

2. As a Chat Client extension (following the pattern of web search in Python: Web search file search tools #395):

   from agent_framework.openai import OpenAIChatClient
   
   client = OpenAIChatClient(tools=["computer_use"])
   agent = ChatAgent(chat_client=client)

3. In the lab package (for experimentation before core integration):

   from agent_framework.lab.cua import CuaComputerTool
   from agent_framework import ChatAgent
   
   computer = CuaComputerTool()
   agent = ChatAgent(tools=[computer], ...)

Technical Details:

The Cua Computer class provides methods that map well to the Agent Framework tool pattern:

# Screenshot capture
screenshot = await computer.screenshot()

# Mouse actions
await computer.left_click(x, y)
await computer.double_click(x, y)
await computer.move_cursor(x, y)

# Keyboard actions
await computer.type_text("Hello")
await computer.press_key("enter")
await computer.hotkey("command", "c")

# Shell commands
result = await computer.run_command("ls -la")

The API design should align well since Cua is async-first and uses a tool-based architecture that could wrap as AIFunction.

Questions:

1. Which integration pattern would align best with the existing built-in tools architecture?
2. Should this follow the OpenAI Responses API pattern or be more generic to support multiple providers?
3. Would you prefer I start with:
   • A proof-of-concept PR showing the integration
   • An ADR (Architectural Decision Record) first
   • A lab package for experimentation
4. Are there specific compatibility requirements I should be aware of?

Happy to work on this if it's useful. Let me know what makes sense for next steps.

Additional Context:

• Cua GitHub: https://github.com/trycua/cua
• Cua Documentation: https://docs.trycua.com

[moonbox3]

Would this abstraction allow us to tailor the CUA tool to specific providers that offer it like OpenAI or Azure OpenAi?

[ekzhu]

Looks super interesting! Thanks @f-trycua for this proposal. Does this work with for example OpenAI's computer use tool (https://platform.openai.com/docs/guides/tools-computer-use) or something that is more like a local MCP server that can be used for any model.

[eavanvalkenburg]

@f-trycua I would consider building this out as a ChatMiddleware, that way you can create a class that has the middleware process function, and holds all the assets needed to make it work, it then also allows you to capture the responses, parse them from the native format of a provider and execute the action, returning the outcome/screen and calling next(context) again, that way you have full control over the execution and can revert back to the user whenever you need (capturing their response in the middleware before next if needed, you could even use the FunctionApprovalRequestContent to ask for that feedback and then getting a FunctionApprovalResponseContent back to act on, in the meantime, you're middleware control the actual executions against the model, so you can ensure a good experience! Let me know if you want to collaborate on that!

[f-trycua]

Would this abstraction allow us to tailor the CUA tool to specific providers that offer it like OpenAI or Azure OpenAi?

Yes @moonbox3 @ekzhu! The CuaAgentMiddleware supports any provider via the model parameter. Cua handles both LiteLLM-based providers (OpenAI, Azure, Anthropic) AND custom implementations for open-weight models with their own specs (OpenCUA, InternVL, UI-TARS), so Agent Framework gets 100+ models without needing to implement any provider-specific logic.

[f-trycua]

@eavanvalkenburg Thanks for the guidance! I've implemented CuaAgentMiddleware following the ChatMiddleware pattern you suggested and opened a draft PR to start the discussion: #1338

The middleware does exactly what you described:

• Uses the process function to hold all Cua assets (ComputerAgent, Computer)
• Captures responses from Cua's native format and transforms them to Agent Framework's ChatResponse
• Supports FunctionApprovalRequestContent/FunctionApprovalResponseContent for human-in-the-loop approval
• Sets context.terminate = True to control execution flow
• One key difference: instead of wrapping just the Computer tool, I wrapped Cua's ComputerAgent to leverage Cua's existing support for 100+ model configurations (OpenCUA, InternVL, UI-Tars, etc.) without reimplementing provider-specific parsers. This way Agent Framework benefits from Cua's ongoing model additions automatically.

Happy to hop on a call this week to discuss the approach and iterate on the design! Would love to collaborate on getting this right.