Desktop Automation

Automate desktop interactions in HopX sandboxes with VNC, screenshots, mouse and keyboard control, and window management. This hub covers all desktop automation features.

​

Related

• CLI Reference - Command-line interface

​

What You’ll Learn

• How to set up and use VNC servers
• How to capture screenshots and screen recordings
• How to control mouse and keyboard
• How to manage windows and displays
• How to use clipboard and advanced X11 features

​

Desktop Automation Overview

​

Core Concepts

​

VNC Server

Set up and connect to VNC servers for remote desktop access.

• VNC Server - Set up VNC servers
• SDK: Python: sandbox.desktop.vnc | JavaScript: sandbox.desktop.vnc
• API: VNC server endpoints

​

Screenshots

Capture screenshots and screen recordings from sandboxes.

• Screenshots - Capture screenshots
• Screen Recording - Record screen activity
• SDK: Python: sandbox.desktop.screenshot() | JavaScript: sandbox.desktop.screenshot()
• API: Screenshot endpoints

​

Mouse Control

Control mouse movements, clicks, and gestures.

• Mouse Control - Control mouse interactions
• SDK: Python: sandbox.desktop.mouse | JavaScript: sandbox.desktop.mouse
• API: Mouse control endpoints

​

Keyboard Control

Send keyboard input and key combinations.

• Keyboard Control - Control keyboard input
• SDK: Python: sandbox.desktop.keyboard | JavaScript: sandbox.desktop.keyboard
• API: Keyboard control endpoints

​

Window Management

Manage windows, displays, and desktop layout.

• Window Management - Manage windows
• Display Management - Manage displays
• SDK: Python: sandbox.desktop.windows | JavaScript: sandbox.desktop.windows
• API: Window management endpoints

​

Advanced X11

Advanced X11 features for desktop automation.

• Advanced X11 - Advanced X11 features
• SDK: Python: sandbox.desktop.x11 | JavaScript: sandbox.desktop.x11
• API: X11 endpoints

​

Clipboard

Access and manipulate clipboard contents.

• Clipboard - Clipboard operations
• SDK: Python: sandbox.desktop.clipboard | JavaScript: sandbox.desktop.clipboard
• API: Clipboard endpoints

​

SDK Reference

​

Python SDK

Complete reference for Python desktop automation.

• Desktop Module - Complete desktop automation API
  • desktop.vnc - VNC server
  • desktop.screenshot() - Screenshots
  • desktop.mouse - Mouse control
  • desktop.keyboard - Keyboard control
  • desktop.windows - Window management
  • desktop.clipboard - Clipboard
  • desktop.x11 - Advanced X11

​

JavaScript SDK

Complete reference for JavaScript/TypeScript desktop automation.

• Desktop Module - Complete desktop automation API
  • desktop.vnc - VNC server
  • desktop.screenshot() - Screenshots
  • desktop.mouse - Mouse control
  • desktop.keyboard - Keyboard control
  • desktop.windows - Window management
  • desktop.clipboard - Clipboard
  • desktop.x11 - Advanced X11

​

API Reference

​

VM Agent API

REST API endpoints for desktop automation.

• VM Agent API Overview - API overview
• Desktop automation endpoints (VNC, screenshots, mouse, keyboard, windows)

​

Quick Start

If you’re new to desktop automation, start here:

1. VNC Server - Set up VNC access
2. Screenshots - Capture your first screenshot
3. Mouse Control - Control mouse interactions

​

Related Topics

• Sandboxes - Create sandboxes for desktop automation
• Code Execution - Execute code with desktop access
• Terminal - Terminal access alongside desktop
• Observability - Monitor desktop performance

​

Next Steps

• Learn about VNC Server for remote desktop access
• Explore Screenshots for visual capture
• Review Mouse Control and Keyboard Control for automation
• Check out Advanced X11 for advanced features