Desktop Automation

Version: 0.3.0
Last Verified: 2025-01-27
Package: hopx-ai on PyPI

Overview

The Desktop resource provides desktop automation capabilities for sandboxes with GUI support. Use this resource for GUI automation, taking screenshots, controlling mouse/keyboard, managing windows, and accessing VNC.

Desktop features require a desktop-enabled template. Not all templates support desktop operations.

Access

from hopx_ai import Sandbox

sandbox = Sandbox.create(template="desktop-template")
desktop = sandbox.desktop  # Lazy-loaded

VNC Methods

`start_vnc`

Start VNC server.

desktop.start_vnc(display=1, password=None) -> VNCInfo

Parameters:

display (int, optional): Display number (default: 1)
password (str, optional): VNC password

Returns: VNCInfo - VNC server information with url, port, password Example:

vnc = sandbox.desktop.start_vnc(password="mypassword")
print(f"VNC URL: {vnc.url}")

Expected Output:

VNC URL: https://vnc-sandbox_abc123xyz.hopx.dev

`stop_vnc`

Stop VNC server.

desktop.stop_vnc() -> None

Expected Output:

VNC server stopped successfully

`get_vnc_status`

Get VNC server status.

desktop.get_vnc_status() -> VNCInfo

Expected Output:

VNC server is running on port 5901

`get_vnc_url`

Get VNC URL (convenience method).

desktop.get_vnc_url() -> str

Expected Output:

https://vnc-sandbox_abc123xyz.hopx.dev

Mouse Methods

`click`

Click at position.

desktop.click(x, y, button='left', clicks=1) -> None

Parameters:

x (int): X coordinate
y (int): Y coordinate
button (str, optional): Mouse button ('left', 'right', 'middle') (default: 'left')
clicks (int, optional): Number of clicks (default: 1)

Expected Output:

Mouse clicked at (x, y)

`move`

Move mouse cursor to position.

desktop.move(x, y) -> None

Expected Output:

Mouse moved to (x, y)

`drag`

Drag from one position to another.

desktop.drag(from_x, from_y, to_x, to_y, button='left') -> None

Expected Output:

Drag completed from (from_x, from_y) to (to_x, to_y)

`scroll`

Scroll mouse wheel.

desktop.scroll(amount, direction='down') -> None

Parameters:

amount (int): Scroll amount
direction (str, optional): 'up' or 'down' (default: 'down')

Expected Output:

Scrolled {amount} units {direction}

Keyboard Methods

`type`

Type text.

desktop.type(text, delay_ms=10) -> None

Parameters:

text (str): Text to type
delay_ms (int, optional): Delay between keystrokes in milliseconds (default: 10)

Expected Output:

Text typed: {text}

`press`

Press a key.

desktop.press(key) -> None

Parameters:

key (str): Key name (e.g., 'Enter', 'Escape', 'Tab')

Expected Output:

Key pressed: {key}

`combination`

Press key combination.

desktop.combination(modifiers, key) -> None

Parameters:

modifiers (List[str]): Modifier keys (e.g., ['Ctrl', 'Alt'])
key (str): Key to press

Example:

desktop.combination(['Ctrl', 'C'], 'c')  # Ctrl+C

Expected Output:

Key combination executed: Ctrl+C

Clipboard Methods

`set_clipboard`

Set clipboard content.

desktop.set_clipboard(text) -> None

Expected Output:

Clipboard set successfully

`get_clipboard`

Get clipboard content.

desktop.get_clipboard() -> str

Expected Output:

Clipboard content retrieved

Screenshot Methods

`screenshot`

Capture full screen screenshot.

desktop.screenshot() -> bytes

Returns: bytes - Screenshot image as PNG bytes Example:

screenshot = sandbox.desktop.screenshot()
with open("screenshot.png", "wb") as f:
    f.write(screenshot)

Expected Output:

Screenshot saved to screenshot.png (PNG bytes)

`screenshot_region`

Capture screenshot of specific region.

desktop.screenshot_region(x, y, width, height) -> bytes

Parameters:

x (int): X coordinate of top-left corner
y (int): Y coordinate of top-left corner
width (int): Region width
height (int): Region height

Expected Output:

Screenshot captured (PNG bytes, {width}x{height} region)

Window Methods

`get_windows`

Get list of all windows.

desktop.get_windows() -> List[WindowInfo]

Returns: List[WindowInfo] - List of window information objects Expected Output:

[WindowInfo(id='window_1', title='Application', ...), WindowInfo(id='window_2', ...)]

`focus_window`

Focus (activate) window.

desktop.focus_window(window_id) -> None

Expected Output:

Window focused successfully

`close_window`

Close window.

desktop.close_window(window_id) -> None

Expected Output:

Window closed successfully

Examples

Example 1: Taking Screenshots

from hopx_ai import Sandbox

sandbox = Sandbox.create(template="desktop-template")

# Take full screen screenshot
screenshot = sandbox.desktop.screenshot()
with open("fullscreen.png", "wb") as f:
    f.write(screenshot)

# Take region screenshot
region = sandbox.desktop.screenshot_region(0, 0, 800, 600)
with open("region.png", "wb") as f:
    f.write(region)

sandbox.kill()

Expected Output:

Screenshots saved: fullscreen.png, region.png

Example 2: Mouse and Keyboard Automation

from hopx_ai import Sandbox

sandbox = Sandbox.create(template="desktop-template")

# Click at position
sandbox.desktop.click(100, 200)

# Type text
sandbox.desktop.type("Hello, World!")

# Press Enter
sandbox.desktop.press("Enter")

# Drag and drop
sandbox.desktop.drag(100, 100, 300, 300)

sandbox.kill()

Expected Output:

Mouse clicked at (100, 200)
Text typed: Hello, World!
Enter key pressed
Drag completed from (100, 100) to (300, 300)

Sandbox - Main sandbox class
Models - Data models including VNCInfo, WindowInfo

Next Steps

Learn about VNC Server for remote desktop access
Explore Screenshots for visual verification
Review Mouse Control and Keyboard Control for GUI automation
CLI System Commands - System operations from CLI

Python SDK

JavaScript SDK

Overview

Access

VNC Methods

`start_vnc`

`stop_vnc`

`get_vnc_status`

`get_vnc_url`

Mouse Methods

`click`

`move`

`drag`

`scroll`

Keyboard Methods

`type`

`press`

`combination`

Clipboard Methods

`set_clipboard`

`get_clipboard`

Screenshot Methods

`screenshot`

`screenshot_region`

Window Methods

`get_windows`

`focus_window`

`close_window`

Examples

Example 1: Taking Screenshots

Example 2: Mouse and Keyboard Automation

See Also

Next Steps

​Overview

​Access

​VNC Methods

​start_vnc

​stop_vnc

​get_vnc_status

​get_vnc_url

​Mouse Methods

​click

​move

​drag

​scroll

​Keyboard Methods

​type

​press

​combination

​Clipboard Methods

​set_clipboard

​get_clipboard

​Screenshot Methods

​screenshot

​screenshot_region

​Window Methods

​get_windows

​focus_window

​close_window

​Examples

​Example 1: Taking Screenshots

​Example 2: Mouse and Keyboard Automation

​Related Classes

​See Also

​Related

​Next Steps

Overview

Access

VNC Methods

`start_vnc`

`stop_vnc`

`get_vnc_status`

`get_vnc_url`

Mouse Methods

`click`

`move`

`drag`

`scroll`

Keyboard Methods

`type`

`press`

`combination`

Clipboard Methods

`set_clipboard`

`get_clipboard`

Screenshot Methods

`screenshot`

`screenshot_region`

Window Methods

`get_windows`

`focus_window`

`close_window`

Examples

Example 1: Taking Screenshots

Example 2: Mouse and Keyboard Automation

Related Classes

See Also

Related

Next Steps