
AI Desktop Agent
The Problem We Designed Around
Desktop work contains ambiguity. Files may have unexpected names. A browser page may fail to load. A command may return an error. Some actions are straightforward to reverse; others affect external systems or remove information. A useful agent therefore needs more than a plan. It needs a controlled execution path, visible state, and a way to establish whether the requested result actually occurred.
For AJOX, those requirements shaped the architecture before additional capabilities were added.
What We Built
AJOX turns voice or text requests into a structured plan and executes it through registered tools. Users can follow each step, review pending approvals, and cancel the task from one desktop interface.
Sensitive approvals are tied to the task and step. Results are checked before AJOX reports completion; a failed verification remains failed.

Inside AJOX
From Intent to Verified Execution.
A structured execution pipeline backed by modular engineering, controlled tool access, and outcome verification. Every component has a defined responsibility in the task lifecycle.
01 / Execution
Execution Workflow
From the initial request to a verified result.
Request
The user provides a spoken or typed goal.
Plan
The planner breaks the goal into structured steps.
Policy check
Each step is matched to a registered action and checked against the applicable policy.
Approval
Sensitive actions pause for approval where required.
Execute
The executor runs the selected action module.
Verify
The system checks available evidence that the action produced the intended result.
Report
The task reports its result, including failures or incomplete steps.
02 / Engineering
Engineering Principles
The building blocks behind controlled execution.
Registered actions
Desktop, browser, and shell tasks run through a defined action layer.
Engineering foundation
Approval for each sensitive step
Permission is tied to the specific task and step being approved.
Engineering foundation
Local-first task state
Memory and task state stay on-device. Live voice uses Gemini Live.
Engineering foundation
03 / System Architecture
Technical Implementation
A modular, local-first architecture for orchestrating tools, managing execution, and validating task outcomes.
Technology Stack
AJOX combines a Python-based agent backend, FastAPI service, Next.js interface, and Tauri desktop shell. Gemini Live powers real-time voice interaction, registered tools provide controlled access to supported file, application, browser, and shell operations, and Playwright supports browser automation.
Execution & Verification
WebSocket updates expose task progress to the interface. The planner maps requests to registered actions, while unsupported operations are rejected. Policy checks determine whether an action can proceed, requires approval, or must be blocked. The executor runs approved modules, and verification checks the resulting state where supported.
Deployment & Readiness
Release validationAJOX is designed around local-first task state and memory, but it is not fully offline: real-time voice interaction depends on an external model service. Signed installer validation and broader hardware-based voice testing remain release-validation tasks before broader production-readiness claims can be made.







































