Skip to main content
Control mouse input to interact with GUI applications. Mouse control enables clicking, moving, dragging, and scrolling operations.

​
Overview

Mouse control enables:
  • Clicking buttons and UI elements
  • Moving cursor to specific positions
  • Dragging and dropping elements
  • Scrolling windows and content
Desktop automation requires a template with desktop support. Ensure your sandbox has desktop capabilities enabled.

​
Clicking

Click at a specific position on the screen:
  • Python
  • JavaScript
from hopx_ai import Sandbox

sandbox = Sandbox.create(template="desktop")

# Single left click
sandbox.desktop.click(100, 100)

# Right click
sandbox.desktop.click(200, 200, button="right")

# Middle click
sandbox.desktop.click(300, 300, button="middle")

# Double click
sandbox.desktop.click(150, 150, clicks=2)

​
Moving Cursor

Move the mouse cursor to a specific position:
  • Python
  • JavaScript
# Move cursor to position
sandbox.desktop.move(500, 300)

​
Dragging

Drag from one position to another:
  • Python
  • JavaScript
# Drag from (100, 100) to (200, 200)
sandbox.desktop.drag(100, 100, 200, 200)

# Drag with right button
sandbox.desktop.drag(100, 100, 200, 200, button="right")

​
Scrolling

Scroll the mouse wheel:
  • Python
  • JavaScript
# Scroll down 5 clicks
sandbox.desktop.scroll(5, "down")

# Scroll up 3 clicks
sandbox.desktop.scroll(3, "up")

# Scroll left
sandbox.desktop.scroll(2, "left")

# Scroll right
sandbox.desktop.scroll(2, "right")

​
Mouse Buttons

Available mouse buttons:
  • "left" - Left mouse button (default)
  • "right" - Right mouse button
  • "middle" - Middle mouse button (scroll wheel)

​
Complete Example

Automate a simple GUI interaction:
  • Python
  • JavaScript
from hopx_ai import Sandbox

sandbox = Sandbox.create(template="desktop")

try:
    # Move to button
    sandbox.desktop.move(400, 300)
    
    # Click button
    sandbox.desktop.click(400, 300)
    
    # Wait a moment (you might use time.sleep in real scenarios)
    
    # Drag a slider
    sandbox.desktop.drag(100, 200, 300, 200)
    
    # Scroll to see more content
    sandbox.desktop.scroll(10, "down")
    
finally:
    sandbox.kill()

​
Coordinate System

Mouse coordinates use screen pixels:
  • Origin (0, 0): Top-left corner of the screen
  • X-axis: Increases from left to right
  • Y-axis: Increases from top to bottom
Use screenshots to identify exact coordinates of UI elements before automating interactions.

​
Next Steps