The problem of coordinate offset

[getupgogogo]

Hello! First of all, I would like to express my gratitude for your open-source project.
I would like to ask if you also encounter the problem of coordinate errors? I found that the process of using Agent S was not very precise. Especially when dealing with smaller buttons. If so, could you please tell me whether you are addressing this issue and if so, through what means? I sincerely hope to have the opportunity to communicate with all of you!

[Chase-Xuu]

Hi! I've been investigating this coordinate precision issue and wanted to share my analysis.

Root Cause Analysis

After digging through the codebase, I believe the coordinate offset problem stems from several factors:

1. Single-point variance: The grounding model outputs one (x,y) point per query. Even small model stochasticity causes different predictions across calls.

2. Resolution scaling amplification: When grounding_width ≠ screen_width, coordinate errors get multiplied. A 5px error at 1000px grounding resolution becomes ~10px at 1920px screen.

3. No element size awareness: The system treats a 200px button the same as a 15px icon — but 10px offset is fine for one and a complete miss for the other.

Proposed Solution: Multi-Sample Voting

I've implemented a robust coordinate generation approach that runs the grounding model multiple times and takes the median:

def generate_coords(self, ref_expr: str, obs: Dict, n_samples: int = 1) -> List[int]:
    # When n_samples > 1, run grounding multiple times and take median
    samples = []
    for i in range(n_samples):
        coords = self._single_grounding_call(ref_expr, obs)
        if coords:
            samples.append(coords)
    
    # Median is more robust to outliers
    x = int(np.median([s[0] for s in samples]))
    y = int(np.median([s[1] for s in samples]))
    return [x, y]

Benefits:

• Reduces variance significantly (statistical principle: std decreases by √n)
• Backward compatible (n_samples=1 by default)
• Simple implementation

Trade-offs:

• Nx latency when using multi-sample (can be parallelized in future work)
• Nx API cost (acceptable for improved accuracy)

I've also refactored the coordinate parsing to properly handle both integer and normalized (0-1) formats.

PR incoming!