Mouse Control

Control mouse input to interact with GUI applications. Mouse control enables clicking, moving, dragging, and scrolling operations.

Overview

Mouse control enables:

Clicking buttons and UI elements
Moving cursor to specific positions
Dragging and dropping elements
Scrolling windows and content

Desktop automation requires a template with desktop support. Ensure your sandbox has desktop capabilities enabled.

Clicking

Click at a specific position on the screen:

Python
JavaScript

from hopx_ai import Sandbox

sandbox = Sandbox.create(template="desktop")

# Single left click
sandbox.desktop.click(100, 100)

# Right click
sandbox.desktop.click(200, 200, button="right")

# Middle click
sandbox.desktop.click(300, 300, button="middle")

# Double click
sandbox.desktop.click(150, 150, clicks=2)

import { Sandbox } from '@hopx-ai/sdk';

const sandbox = await Sandbox.create({ template: 'desktop' });

// Single left click
await sandbox.desktop.mouseClick(100, 100);

// Right click
await sandbox.desktop.mouseClick(200, 200, { button: 'right' });

// Middle click
await sandbox.desktop.mouseClick(300, 300, { button: 'middle' });

// Double click
await sandbox.desktop.mouseClick(150, 150, { clicks: 2 });

Moving Cursor

Move the mouse cursor to a specific position:

Python
JavaScript

# Move cursor to position
sandbox.desktop.move(500, 300)

// Move cursor to position
await sandbox.desktop.mouseMove(500, 300);

Dragging

Drag from one position to another:

Python
JavaScript

# Drag from (100, 100) to (200, 200)
sandbox.desktop.drag(100, 100, 200, 200)

# Drag with right button
sandbox.desktop.drag(100, 100, 200, 200, button="right")

// Drag from (100, 100) to (200, 200)
await sandbox.desktop.mouseDrag(100, 100, 200, 200);

Scrolling

Scroll the mouse wheel:

Python
JavaScript

# Scroll down 5 clicks
sandbox.desktop.scroll(5, "down")

# Scroll up 3 clicks
sandbox.desktop.scroll(3, "up")

# Scroll left
sandbox.desktop.scroll(2, "left")

# Scroll right
sandbox.desktop.scroll(2, "right")

// Scroll (JavaScript SDK may have different API)
// Note: Check SDK documentation for exact scroll method signature
await sandbox.desktop.mouseScroll(500, 300, 5); // x, y, deltaY

Mouse Buttons

Available mouse buttons:

"left" - Left mouse button (default)
"right" - Right mouse button
"middle" - Middle mouse button (scroll wheel)

Complete Example

Automate a simple GUI interaction:

Python
JavaScript

from hopx_ai import Sandbox

sandbox = Sandbox.create(template="desktop")

try:
    # Move to button
    sandbox.desktop.move(400, 300)
    
    # Click button
    sandbox.desktop.click(400, 300)
    
    # Wait a moment (you might use time.sleep in real scenarios)
    
    # Drag a slider
    sandbox.desktop.drag(100, 200, 300, 200)
    
    # Scroll to see more content
    sandbox.desktop.scroll(10, "down")
    
finally:
    sandbox.kill()

import { Sandbox } from '@hopx-ai/sdk';

const sandbox = await Sandbox.create({ template: 'desktop' });

try {
  // Move to button
  await sandbox.desktop.mouseMove(400, 300);
  
  // Click button
  await sandbox.desktop.mouseClick(400, 300);
  
  // Wait a moment (you might use setTimeout in real scenarios)
  
  // Drag a slider
  await sandbox.desktop.mouseDrag(100, 200, 300, 200);
  
  // Scroll to see more content
  await sandbox.desktop.mouseScroll(500, 300, 10);
  
} finally {
  await sandbox.kill();
}

Coordinate System

Mouse coordinates use screen pixels:

Origin (0, 0): Top-left corner of the screen
X-axis: Increases from left to right
Y-axis: Increases from top to bottom

Use screenshots to identify exact coordinates of UI elements before automating interactions.

Keyboard Control - Control keyboard input
CLI Reference - Command-line interface for HopX
Screenshots - Capture screenshots to find coordinates
X11 Advanced - Find elements by text
SDK: sandbox.desktop.click() - Python SDK method

Next Steps

Learn about Keyboard Control for text input
Explore Screenshots to find element coordinates
Review VNC Server for remote desktop access

Home

Get Started

Sandboxes

Code Execution

Filesystem

Commands

Environment Variables

Templates

Desktop Automation

Terminal

Observability

Cache

Overview

Clicking

Moving Cursor

Dragging

Scrolling

Mouse Buttons

Complete Example

Coordinate System

Next Steps

​Overview

​Clicking

​Moving Cursor

​Dragging

​Scrolling

​Mouse Buttons

​Complete Example

​Coordinate System

​Related

​Next Steps

Overview

Clicking

Moving Cursor

Dragging

Scrolling

Mouse Buttons

Complete Example

Coordinate System

Related

Next Steps