Skip to main content
Version: 0.3.0
Last Verified: 2025-01-27
Package: hopx-ai on PyPI

​
Overview

The Desktop resource provides desktop automation capabilities for sandboxes with GUI support. Use this resource for GUI automation, taking screenshots, controlling mouse/keyboard, managing windows, and accessing VNC.
Desktop features require a desktop-enabled template. Not all templates support desktop operations.

​
Access

from hopx_ai import Sandbox

sandbox = Sandbox.create(template="desktop-template")
desktop = sandbox.desktop  # Lazy-loaded

​
VNC Methods

​
start_vnc

Start VNC server. 
desktop.start_vnc(display=1, password=None) -> VNCInfo
Parameters:
  • display (int, optional): Display number (default: 1)
  • password (str, optional): VNC password
Returns: VNCInfo - VNC server information with url, port, password Example: 
vnc = sandbox.desktop.start_vnc(password="mypassword")
print(f"VNC URL: {vnc.url}")
Expected Output: 
VNC URL: https://vnc-sandbox_abc123xyz.hopx.dev

​
stop_vnc

Stop VNC server. 
desktop.stop_vnc() -> None
Expected Output: 
VNC server stopped successfully

​
get_vnc_status

Get VNC server status. 
desktop.get_vnc_status() -> VNCInfo
Expected Output: 
VNC server is running on port 5901

​
get_vnc_url

Get VNC URL (convenience method). 
desktop.get_vnc_url() -> str
Expected Output: 
https://vnc-sandbox_abc123xyz.hopx.dev

​
Mouse Methods

​
click

Click at position. 
desktop.click(x, y, button='left', clicks=1) -> None
Parameters:
  • x (int): X coordinate
  • y (int): Y coordinate
  • button (str, optional): Mouse button ('left', 'right', 'middle') (default: 'left')
  • clicks (int, optional): Number of clicks (default: 1)
Expected Output: 
Mouse clicked at (x, y)

​
move

Move mouse cursor to position. 
desktop.move(x, y) -> None
Expected Output: 
Mouse moved to (x, y)

​
drag

Drag from one position to another. 
desktop.drag(from_x, from_y, to_x, to_y, button='left') -> None
Expected Output: 
Drag completed from (from_x, from_y) to (to_x, to_y)

​
scroll

Scroll mouse wheel. 
desktop.scroll(amount, direction='down') -> None
Parameters:
  • amount (int): Scroll amount
  • direction (str, optional): 'up' or 'down' (default: 'down')
Expected Output: 
Scrolled {amount} units {direction}

​
Keyboard Methods

​
type

Type text. 
desktop.type(text, delay_ms=10) -> None
Parameters:
  • text (str): Text to type
  • delay_ms (int, optional): Delay between keystrokes in milliseconds (default: 10)
Expected Output: 
Text typed: {text}

​
press

Press a key. 
desktop.press(key) -> None
Parameters:
  • key (str): Key name (e.g., 'Enter', 'Escape', 'Tab')
Expected Output: 
Key pressed: {key}

​
combination

Press key combination. 
desktop.combination(modifiers, key) -> None
Parameters:
  • modifiers (List[str]): Modifier keys (e.g., ['Ctrl', 'Alt'])
  • key (str): Key to press
Example: 
desktop.combination(['Ctrl', 'C'], 'c')  # Ctrl+C
Expected Output: 
Key combination executed: Ctrl+C

​
Clipboard Methods

​
set_clipboard

Set clipboard content. 
desktop.set_clipboard(text) -> None
Expected Output: 
Clipboard set successfully

​
get_clipboard

Get clipboard content. 
desktop.get_clipboard() -> str
Expected Output: 
Clipboard content retrieved

​
Screenshot Methods

​
screenshot

Capture full screen screenshot. 
desktop.screenshot() -> bytes
Returns: bytes - Screenshot image as PNG bytes Example: 
screenshot = sandbox.desktop.screenshot()
with open("screenshot.png", "wb") as f:
    f.write(screenshot)
Expected Output: 
Screenshot saved to screenshot.png (PNG bytes)

​
screenshot_region

Capture screenshot of specific region. 
desktop.screenshot_region(x, y, width, height) -> bytes
Parameters:
  • x (int): X coordinate of top-left corner
  • y (int): Y coordinate of top-left corner
  • width (int): Region width
  • height (int): Region height
Expected Output: 
Screenshot captured (PNG bytes, {width}x{height} region)

​
Window Methods

​
get_windows

Get list of all windows. 
desktop.get_windows() -> List[WindowInfo]
Returns: List[WindowInfo] - List of window information objects Expected Output: 
[WindowInfo(id='window_1', title='Application', ...), WindowInfo(id='window_2', ...)]

​
focus_window

Focus (activate) window. 
desktop.focus_window(window_id) -> None
Expected Output: 
Window focused successfully

​
close_window

Close window. 
desktop.close_window(window_id) -> None
Expected Output: 
Window closed successfully

​
Examples

​
Example 1: Taking Screenshots

from hopx_ai import Sandbox

sandbox = Sandbox.create(template="desktop-template")

# Take full screen screenshot
screenshot = sandbox.desktop.screenshot()
with open("fullscreen.png", "wb") as f:
    f.write(screenshot)

# Take region screenshot
region = sandbox.desktop.screenshot_region(0, 0, 800, 600)
with open("region.png", "wb") as f:
    f.write(region)

sandbox.kill()
Expected Output: 
Screenshots saved: fullscreen.png, region.png

​
Example 2: Mouse and Keyboard Automation

from hopx_ai import Sandbox

sandbox = Sandbox.create(template="desktop-template")

# Click at position
sandbox.desktop.click(100, 200)

# Type text
sandbox.desktop.type("Hello, World!")

# Press Enter
sandbox.desktop.press("Enter")

# Drag and drop
sandbox.desktop.drag(100, 100, 300, 300)

sandbox.kill()
Expected Output: 
Mouse clicked at (100, 200)
Text typed: Hello, World!
Enter key pressed
Drag completed from (100, 100) to (300, 300)
  • Sandbox - Main sandbox class
  • Models - Data models including VNCInfo, WindowInfo

​
See Also

​
Next Steps