Control mouse input to interact with GUI applications. Mouse control enables clicking, moving, dragging, and scrolling operations.
Overview
Mouse control enables:
- Clicking buttons and UI elements
- Moving cursor to specific positions
- Dragging and dropping elements
- Scrolling windows and content
Desktop automation requires a template with desktop support. Ensure your sandbox has desktop capabilities enabled.
Clicking
Click at a specific position on the screen:
from hopx_ai import Sandbox
sandbox = Sandbox.create(template="desktop")
# Single left click
sandbox.desktop.click(100, 100)
# Right click
sandbox.desktop.click(200, 200, button="right")
# Middle click
sandbox.desktop.click(300, 300, button="middle")
# Double click
sandbox.desktop.click(150, 150, clicks=2)
import { Sandbox } from '@hopx-ai/sdk';
const sandbox = await Sandbox.create({ template: 'desktop' });
// Single left click
await sandbox.desktop.mouseClick(100, 100);
// Right click
await sandbox.desktop.mouseClick(200, 200, { button: 'right' });
// Middle click
await sandbox.desktop.mouseClick(300, 300, { button: 'middle' });
// Double click
await sandbox.desktop.mouseClick(150, 150, { clicks: 2 });
Moving Cursor
Move the mouse cursor to a specific position:
# Move cursor to position
sandbox.desktop.move(500, 300)
// Move cursor to position
await sandbox.desktop.mouseMove(500, 300);
Dragging
Drag from one position to another:
# Drag from (100, 100) to (200, 200)
sandbox.desktop.drag(100, 100, 200, 200)
# Drag with right button
sandbox.desktop.drag(100, 100, 200, 200, button="right")
// Drag from (100, 100) to (200, 200)
await sandbox.desktop.mouseDrag(100, 100, 200, 200);
# Scroll down 5 clicks
sandbox.desktop.scroll(5, "down")
# Scroll up 3 clicks
sandbox.desktop.scroll(3, "up")
# Scroll left
sandbox.desktop.scroll(2, "left")
# Scroll right
sandbox.desktop.scroll(2, "right")
// Scroll (JavaScript SDK may have different API)
// Note: Check SDK documentation for exact scroll method signature
await sandbox.desktop.mouseScroll(500, 300, 5); // x, y, deltaY
"left" - Left mouse button (default)"right" - Right mouse button"middle" - Middle mouse button (scroll wheel)
Complete Example
Automate a simple GUI interaction:
from hopx_ai import Sandbox
sandbox = Sandbox.create(template="desktop")
try:
# Move to button
sandbox.desktop.move(400, 300)
# Click button
sandbox.desktop.click(400, 300)
# Wait a moment (you might use time.sleep in real scenarios)
# Drag a slider
sandbox.desktop.drag(100, 200, 300, 200)
# Scroll to see more content
sandbox.desktop.scroll(10, "down")
finally:
sandbox.kill()
import { Sandbox } from '@hopx-ai/sdk';
const sandbox = await Sandbox.create({ template: 'desktop' });
try {
// Move to button
await sandbox.desktop.mouseMove(400, 300);
// Click button
await sandbox.desktop.mouseClick(400, 300);
// Wait a moment (you might use setTimeout in real scenarios)
// Drag a slider
await sandbox.desktop.mouseDrag(100, 200, 300, 200);
// Scroll to see more content
await sandbox.desktop.mouseScroll(500, 300, 10);
} finally {
await sandbox.kill();
}
Coordinate System
Mouse coordinates use screen pixels:
- Origin (0, 0): Top-left corner of the screen
- X-axis: Increases from left to right
- Y-axis: Increases from top to bottom
Use screenshots to identify exact coordinates of UI elements before automating interactions.
Next Steps
