The Neuro-Symbolic Computer Use Agent

Computer use is the most promising frontier in AI agents. An agent that can see a screen, understand what's on it, and click, type, and scroll like a human can automate any software—no API integration required. But the current approach has a fundamental flaw: it puts an LLM in the driver's seat.

Every screenshot the agent takes gets sent to a foundation model. The model looks at the image, decides what to do, and returns an action. Then the agent takes another screenshot, sends it to the model again, and repeats. A single task that takes thirty actions means thirty API round-trips. At Claude Opus pricing, that's $0.50–$2.25 per task. At scale—thousands of tickets, orders, and workflows per day—the cost is prohibitive.

Worse, the model is a black box. When it clicks the wrong button, you can't ask it to show its reasoning. It doesn't have reasoning. It has token probabilities. It predicted that the next likely action was “click at (847, 312)” the same way it predicts the next word in a sentence. There's no formal chain of logic you can audit, no rule you can update, no hypothesis it tested before acting.

We built a different kind of computer use agent. One where the brain is not an LLM but a neuro-symbolic reasoning engine—a system that combines neural perception (OCR, UI detection) with symbolic logic (rules, programs, proofs) through a recursive feedback loop. It runs locally, costs nothing per inference, and can explain every decision it makes.

The black-box problem

StateSet's AI agents initially operated on a standard RAG pipeline: retrieve documents and rules from a vector store, construct a prompt, call a fine-tuned LLM to generate output, deliver responses to customers or downstream workflows. This architecture works for conversation but struggles with reasoning.

The LLM “imitates” correct behavior. It does not execute logic. It predicts the next likely token; it does not evaluate the truth of a rule. When an incorrect answer is produced, you cannot identify whether retrieval surfaced the wrong rule, whether the model misinterpreted the rule, whether fine-tuning weights encouraged a harmful pattern, or whether temperature injected randomness. There is no inspectable proof.

Correcting business logic requires writing more documents for RAG or fine-tuning the model again. This is analogous to teaching arithmetic by showing more examples rather than implementing algebraic operators.

What we wanted instead

We wanted a computer use agent with three properties:

1. Zero marginal cost—no API calls, no tokens, no per-task billing. The reasoning engine runs locally.
2. Full explainability—every action traced to an explicit reasoning chain: what the agent perceived, what gaps it identified, what hypothesis it formed, why it chose that action, and whether its prediction was confirmed.
3. Self-improvement—when the agent makes a mistake, it doesn't blindly retry. It explains the failure causally, generates a corrective hypothesis (abduction), and stores the evidence-scored pattern so it doesn't make the same mistake again.

This led us to build the NSR Computer Use Agent: a system where a Rust-based Neuro-Symbolic Recursive machine replaces the LLM as the brain of the computer use loop.

The agent loop

The NSR computer use agent implements Kahneman's System 2—slow, deliberative, step-by-step reasoning where every decision has an explicit justification, predictions are formulated before acting, and failures are explained causally rather than retried blindly.

The standard computer use loop is: screenshot → LLM → action → repeat. Our loop is fundamentally different:

Every step is recorded in a ReasoningChain—an ordered, justifiable sequence of inference steps that can be serialized and inspected after the fact. The chain logs: what was perceived, what gap was identified, what hypothesis was formed, what action was selected (and why), whether the prediction was confirmed, and if not, what the causal explanation was.

Vision: seeing the screen without an LLM

The first challenge in removing the LLM from the loop is perception. An LLM-powered agent sends a screenshot to the model and gets back a description of what's on screen. Without the LLM, you need an alternative vision pipeline.

Our VisionPipeline processes each screenshot into a structured ScreenState:

The UI detector is bootstrapped from screenshots captured during standard CUA operation and trained with a simple pipeline:

# Bootstrap dataset from captured screenshots
python scripts/nsr_bootstrap_ui_dataset.py \
  --source-dir screenshots/NSR --out datasets/ui-detector \
  --limit 120 --clean

# Train YOLO detector and export to ONNX
python scripts/nsr_train_ui_detector.py \
  --data datasets/ui-detector/data.yaml \
  --epochs 25 --imgsz 640 --batch 8 \
  --output models/nsr/ui_detector_v1.onnx

# Validate the ONNX model
python scripts/nsr_check_onnx_detector.py \
  --model models/nsr/ui_detector_v1.onnx

Grounding: from pixels to symbols

The ScreenGrounder converts the raw ScreenState into Grounded Symbol System (GSS) inputs that the Rust NSR machine can reason over. Each UI element becomes a composite grounded input encoding its type, label, normalized position, dimensions, confidence, and interactivity:

# Each detected UI element becomes a GSS node:
nsr.GroundedInput.composite([
    nsr.GroundedInput.text("button"),         # Element type
    nsr.GroundedInput.text("Close Ticket"),   # Label from OCR
    nsr.GroundedInput.number(0.73),           # Center X (normalized)
    nsr.GroundedInput.number(0.21),           # Center Y (normalized)
    nsr.GroundedInput.number(0.12),           # Width (normalized)
    nsr.GroundedInput.number(0.04),           # Height (normalized)
    nsr.GroundedInput.number(0.94),           # Detection confidence
    nsr.GroundedInput.number(1.0),            # Interactive = true
])

This is the critical bridge. The GSS representation lets the symbolic reasoning engine work with the same UI elements a human would see—buttons, fields, links, dialogs—without needing a language model to interpret the screenshot. The grounding is geometric and deterministic, not probabilistic.

The NSR machine: where reasoning happens

At the core of the agent is the Rust NSR Machine, accessed through PyO3 bindings. This is a genuine neuro-symbolic system—not a rule engine with an LLM bolted on, but a unified architecture where neural and symbolic components co-evolve through a recursive feedback loop.

The defining characteristic: components are not independent; they co-evolve through the feedback loop. This is not a traditional hybrid system where neural features feed into a separate symbolic engine. The perception model, parser, and program library are all refined jointly through the deduction-abduction training cycle.

The deduction-abduction loop

This is the core innovation. When the standard agent makes a mistake, it simply retries—same model, same approach, slightly different tokens. Our agent implements a fundamentally different error-recovery mechanism inspired by the ICLR 2024 NSR paper:

System 2 reasoning primitives

The engine maintains a bounded WorkingMemory (capacity: 7 slots, per Miller's number) that holds the current deliberation state. Every inference cycle produces a ReasoningChain—an ordered sequence of typed steps:

[System2] Close resolved ticket:
  PERCEIVE → MEANS_ENDS → HYPOTHESIZE → SELECT → VERIFY → LEARN
  (142ms)

Steps:
  1. PERCEIVE: Detected 14 UI elements, 23 text regions
     Confidence: 0.87 | Grounding: strict (OCR=tesseract, detector=ONNX)

  2. MEANS_ENDS: Gap identified — element_not_clicked
     Current: "Close" button not interacted with
     Desired: Execute click on "Close" button
     Operator: click | Confidence: 0.92

  3. HYPOTHESIZE: "If I click Close at (847, 312),
     the ticket status should change to Closed"
     Predicted changes: ["dialog closes", "status updates"]

  4. SELECT: click(847, 312)
     Justification: Highest confidence match for goal keyword
     Alternative: click(903, 312) — "Cancel" button (rejected)

  5. VERIFY: Prediction CONFIRMED
     Screen changed: true | Local changed: true
     Visual proof: 0.2% pixel change threshold met

  6. LEARN: Pattern stored with evidence score 0.92
     Key: "close_ticket_button_helpdesk"

This is the fundamental difference from an LLM-powered agent. You can inspect the full deliberation chain. You can see exactly what the agent perceived, what gap it identified, why it chose that specific action over alternatives, and whether its prediction held. If something goes wrong, the trace tells you where and why.

Means-ends analysis

Classic AI planning: compare current state with goal, identify gaps, find operators that close them. The MeansEndsAnalysis module maps gap types to action operators:

This replaces the LLM's implicit “I see a button so I'll click it” with an explicit gap analysis: “The goal requires the ticket to be closed. The current state shows a Close button that has not been interacted with. The operator that closes this gap is click. The target coordinates are (847, 312) with confidence 0.92.”

What makes this a true NSR system

Many systems call themselves “neuro-symbolic” when they simply pipe neural network output into a rule engine. That's a hybrid, not a neuro-symbolic recursive system. The distinction matters. A true NSR satisfies six pillars:

Advanced cognitive modules

The NSR machine includes 10+ modules that push beyond basic neuro-symbolic reasoning:

Metacognitive adaptation (dual-process)

Not every action requires the same depth of reasoning. The metacognitive controller implements a dual-process model:

• Fast path (confidence > 0.9)—direct program execution. The agent has seen this exact screen state before and knows what to do.
• Medium path (0.6–0.9)—MCTS-guided adaptation. The agent recognizes the general pattern but needs to search for the right variant.
• Slow path (confidence < 0.6)—full deduction-abduction search. Novel screen state requiring deliberate hypothesis generation and testing.

This mirrors human cognition: you don't carefully deliberate about clicking “OK” on a dialog you've seen a hundred times, but you do reason carefully when encountering an unfamiliar interface.

Running the agent

The NSR agent is a first-class execution mode of the StateSet Computer Use Agent, activated with a single flag:

# Standard CUA (uses Claude API — $0.50-$2.25/task)
python main.py --instruction "close resolved support tickets"

# NSR mode (local reasoning — $0.00/task)
python main.py --nsr --instruction "close resolved support tickets"

# NSR with continuous learning
python main.py --nsr --nsr-continuous --nsr-runtime-seconds 300 \
  --instruction "close resolved support tickets"

# NSR in strict grounding mode with custom UI detector
NSR_GROUNDING_MODE=strict NSR_OCR_ENGINE=tesseract \
NSR_UI_MODEL_PATH=models/nsr/ui_detector_v1.onnx \
python main.py --nsr --instruction "close resolved support tickets"

When --nsr is set, the entire standard CUA sampling loop is bypassed. No Claude API calls are made. All reasoning runs locally through the Rust NSR machine. The same ComputerTool and BashTool are used for action execution—the only thing that changes is the brain.

Benchmark results

We validated the NSR agent with a generalized benchmark covering multiple task profiles:

NSR vs. LLM-powered CUA

Production resilience

An agent that runs continuously across desktop applications needs fault tolerance that most AI demos never consider:

Configuration: fully environment-driven

Every aspect of the agent is configurable via environment variables with documented defaults. No code changes required to tune behavior:

# Vision
NSR_GROUNDING_MODE=strict          # strict | degraded
NSR_OCR_ENGINE=auto                # paddleocr | easyocr | tesseract | fallback
NSR_UI_MODEL_PATH=models/nsr/ui_detector_v1.onnx
NSR_UI_DETECTION_THRESHOLD=0.5

# Reasoning
NSR_EMBEDDING_DIM=64               # Symbol embedding dimensions
NSR_BEAM_WIDTH=8                   # Abduction search width
NSR_MCTS_SIMULATIONS=200           # Hypothesis exploration budget
NSR_MAX_PROGRAM_DEPTH=10           # Max action program depth
NSR_ENABLE_GRAPH_OF_THOUGHTS=1     # DAG reasoning
NSR_ENABLE_TIME_TRAVEL=1           # Counterfactual replanning
NSR_GSS_INFER_BUDGET_MS=1500       # Circuit breaker timeout

# Learning
NSR_ENABLE_LIBRARY_LEARNING=1      # DreamCoder-style abstraction
NSR_ENABLE_KNOWLEDGE_GRAPH=1       # Persistent pattern memory
NSR_ENABLE_CONTINUOUS_LEARNING=1   # Online trajectory learning
NSR_LIBRARY_LEARNING_INTERVAL=3    # Run every N episodes

The Rust engine

The NSR machine itself is 196 Rust source files accessed through PyO3 bindings. We chose Rust for the same reasons it matters in production commerce:

• Latency—symbolic inference runs in microseconds. 50–200ms total including perception, parsing, and program evaluation. No GC pauses, no runtime overhead.
• Safety—the type system and ownership model eliminate undefined behavior at compile time. Financial calculations in commerce workflows cannot have memory corruption or data races.
• Concurrency—the engine runs on Tokio with async/await throughout. Graph-of-Thoughts branches are evaluated in parallel. Beam search over symbol alternatives runs concurrently.

The Rust engine is deployed at nsr.stateset.cloud.stateset.app as a REST API (Axum-based, OpenAPI documented) and also available as an embedded library via PyO3 for the computer use agent integration.

What this means for commerce operations

The practical consequence is straightforward: any task a human performs by looking at a screen and clicking can now be automated with zero per-task cost, full auditability, and continuous self-improvement.

In commerce, that means:

• Support ticket resolution—the agent navigates your helpdesk, reads tickets, applies policies, and closes them. With reasoning traces for every action.
• Inventory management—the agent opens Shopify admin, checks stock levels, creates reorder alerts, and updates quantities. No Shopify API integration needed.
• Return processing—policy evaluation happens in the symbolic layer (deterministic, auditable). Execution happens via computer use tools. The customer gets the right answer every time.
• Cross-system workflows—the agent that starts in Zendesk can navigate to NetSuite, then to Shopify, then back. No integration middleware. It just operates each application the same way a human would.

Open questions

• Generalization across UIs—the current UI detector is trained on specific application screenshots. We are working on a foundation detector that generalizes across arbitrary desktop applications without domain-specific training.
• Federated learning—when multiple brands run NSR agents against similar applications, can the system share learned patterns without leaking proprietary data? We are exploring privacy-preserving knowledge transfer.
• Hybrid escalation—some tasks genuinely benefit from LLM reasoning (novel, linguistically complex queries). The ideal system uses NSR for deterministic workflows and escalates to an LLM only when symbolic reasoning is insufficient. We are building this handoff mechanism.
• Multi-agent coordination—the current architecture runs one NSR agent per task. We are designing a fleet coordinator where NSR agents can pass context to each other mid-workflow.

References

1. Qiu et al. (2024). “Neural-Symbolic Recursive Machine for Systematic Generalization.” ICLR 2024.
2. Ellis et al. (2021). “DreamCoder: Bootstrapping Inductive Program Synthesis with Wake-Sleep Library Learning.” PLDI 2021.
3. Besta et al. (2023). “Graph of Thoughts: Solving Elaborate Problems with Large Language Models.”
4. Plate (1995). “Holographic Reduced Representations.” IEEE Transactions on Neural Networks.
5. Rocktäschel & Riedel (2017). “End-to-end Differentiable Proving.” NeurIPS 2017.
6. Kirkpatrick et al. (2017). “Overcoming catastrophic forgetting in neural networks.” PNAS.
7. Kahneman (2011). “Thinking, Fast and Slow.” Farrar, Straus and Giroux.

The question is not whether agents can operate software autonomously. It is whether you need to pay a foundation model every time they do.