Documentation Index
Fetch the complete documentation index at: https://docs.hopx.ai/llms.txt
Use this file to discover all available pages before exploring further.
Control mouse input to interact with GUI applications. Mouse control enables clicking, moving, dragging, and scrolling operations.
Overview
Mouse control enables:
- Clicking buttons and UI elements
- Moving cursor to specific positions
- Dragging and dropping elements
- Scrolling windows and content
Desktop automation requires a template with desktop support. Ensure your sandbox has desktop capabilities enabled.
Clicking
Click at a specific position on the screen:
from hopx_ai import Sandbox
sandbox = Sandbox.create(template="desktop")
# Single left click
sandbox.desktop.click(100, 100)
# Right click
sandbox.desktop.click(200, 200, button="right")
# Middle click
sandbox.desktop.click(300, 300, button="middle")
# Double click
sandbox.desktop.click(150, 150, clicks=2)
import { Sandbox } from '@hopx-ai/sdk';
const sandbox = await Sandbox.create({ template: 'desktop' });
// Single left click
await sandbox.desktop.mouseClick(100, 100);
// Right click
await sandbox.desktop.mouseClick(200, 200, { button: 'right' });
// Middle click
await sandbox.desktop.mouseClick(300, 300, { button: 'middle' });
// Double click
await sandbox.desktop.mouseClick(150, 150, { clicks: 2 });
Moving Cursor
Move the mouse cursor to a specific position:
# Move cursor to position
sandbox.desktop.move(500, 300)
// Move cursor to position
await sandbox.desktop.mouseMove(500, 300);
Dragging
Drag from one position to another:
# Drag from (100, 100) to (200, 200)
sandbox.desktop.drag(100, 100, 200, 200)
# Drag with right button
sandbox.desktop.drag(100, 100, 200, 200, button="right")
// Drag from (100, 100) to (200, 200)
await sandbox.desktop.mouseDrag(100, 100, 200, 200);
# Scroll down 5 clicks
sandbox.desktop.scroll(5, "down")
# Scroll up 3 clicks
sandbox.desktop.scroll(3, "up")
# Scroll left
sandbox.desktop.scroll(2, "left")
# Scroll right
sandbox.desktop.scroll(2, "right")
// Scroll (JavaScript SDK may have different API)
// Note: Check SDK documentation for exact scroll method signature
await sandbox.desktop.mouseScroll(500, 300, 5); // x, y, deltaY
"left" - Left mouse button (default)"right" - Right mouse button"middle" - Middle mouse button (scroll wheel)
Complete Example
Automate a simple GUI interaction:
from hopx_ai import Sandbox
sandbox = Sandbox.create(template="desktop")
try:
# Move to button
sandbox.desktop.move(400, 300)
# Click button
sandbox.desktop.click(400, 300)
# Wait a moment (you might use time.sleep in real scenarios)
# Drag a slider
sandbox.desktop.drag(100, 200, 300, 200)
# Scroll to see more content
sandbox.desktop.scroll(10, "down")
finally:
sandbox.kill()
import { Sandbox } from '@hopx-ai/sdk';
const sandbox = await Sandbox.create({ template: 'desktop' });
try {
// Move to button
await sandbox.desktop.mouseMove(400, 300);
// Click button
await sandbox.desktop.mouseClick(400, 300);
// Wait a moment (you might use setTimeout in real scenarios)
// Drag a slider
await sandbox.desktop.mouseDrag(100, 200, 300, 200);
// Scroll to see more content
await sandbox.desktop.mouseScroll(500, 300, 10);
} finally {
await sandbox.kill();
}
Coordinate System
Mouse coordinates use screen pixels:
- Origin (0, 0): Top-left corner of the screen
- X-axis: Increases from left to right
- Y-axis: Increases from top to bottom
Use screenshots to identify exact coordinates of UI elements before automating interactions.
Next Steps
