← All articles

AI agents / security

Beyond the Screenshot Loop: What Tesla's Digital Optimus Signals for Computer-Use Agent Architecture and Security

Disclosure: this article discusses Grok Bot, a product of xAI/SpaceX (section 3). The author's AI assistant is Grok Bot. The analysis relies on public documentation and statements only.

Almost every commercial "computer use" agent works in discrete steps: screenshot, ask a vision-language model (VLM) what to do, execute the click or keystroke, repeat (Anthropic) [1]. Tesla and SpaceX's xAI say they are building something different: an agent that consumes continuous screen video and emits mouse and keyboard controls the way a driving policy emits steering (Musk) [2]. If that works, it changes agents' latency, cost and data flow, but not the security problems that matter most to enterprises. We label our own inference.

Sources for this section: [1] Anthropic, Computer use tool. [2] Musk on X, 11 March 2026.

1. How today's screenshot loop works, and where it hurts

The vendor APIs share one pattern: the model returns actions, your application executes them and returns a fresh screenshot (Anthropic [1], OpenAI [2], Google [3]). Anthropic says latency "might be too slow compared to regular human-directed computer actions", puts a screenshot at roughly 1,000 to 1,800 input tokens and adds about 4,500 tokens of toolset overhead per request (Anthropic) [1].

Research quantifies the blind spot. A June 2026 paper describes agents taking roughly one screenshot every 3 to 5 seconds, blind to transient UI between frames; sampling at about 3 Hz and narrating changes improved dynamic-task success by 17 to 48 percentage points without retraining, though model decisions still take 1 to 5 seconds (arXiv 2606.29472) [4]. Another paper blames "correct but late" failures on decoding latency and pre-computes conditional policy trees, lifting success in a contested decision window from 0.50 to 0.79 (AAPT) [5].

Our reading: the snapshot loop is weak on time-critical screens; the field is attacking that with scaffolding on existing models and, in Tesla's case, purpose-built low-latency policies.

Sources for this section: [1] Anthropic, Computer use tool. [2] OpenAI, Computer use guide. [3] Google, Gemini 2.5 Computer Use. [4] Agent-Computer Observation Interfaces. [5] Why Are GUI Agents Correct but Late?

2. What Tesla and SpaceX have actually said

  • 11 March 2026. Musk posted that "Macrohard or Digital Optimus" is a joint xAI-Tesla project in which Grok is the "master conductor/navigator", directing an agent that processes "the past 5 secs of real-time computer screen video and keyboard/mouse actions", running on "the super low cost Tesla AI4 ($650)" (post) [1]. On 12 March he added that it works in AI4-equipped cars (post) [2]. Teslarati reports an availability target of about six months (Teslarati) [3].
  • 22 July, Q2 call. Musk called it "photons in, controls out": "it's not like screenshots type of things, it's real-time video and high frame rate", with a larger model acting as "manager" (call transcript) [4]. Tesla's written Q2 update does not mention it (in the text we extracted), so the architecture is on record from the call, not filings (update) [5].
  • Who is the manager? Musk called "SpaceX" the "big model" manager and later said "Grok helping drive Digital Optimus" (transcript) [4]. That is consistent: SpaceX acquired xAI on 2 February 2026, so Grok now sits inside SpaceX (CNN) [6]. Tesla's general counsel said the companies deepened their relationship "through an investment and a framework agreement" covering "Terafab and Digital Optimus" (transcript) [4]. Still unstated: whether the manager must be a Grok model.
  • 12 August. Replying to a user who wrote "Grok Bot IS Digital Optimus", Musk answered: "Digital Optimus is different: real-time video human emulation" (reply; parent post) [7]. They are separate products (section 3).
  • Hiring. A Tesla posting (via an aggregator) says the approach "is modeled after real-time control policies, rather than screenshot-based VLM agents", targets "edge device deployment", and requires "steerable and interpretable" systems (listing) [8]. A 7 October role covers web, professional apps and AAA games (listing) [9].
  • 10 October. Musk said it "can play about halfway through the Diablo campaign so far just by looking at the screen like a human" (post) [10]. Self-reported; no published evaluation.
  • Trademark. The USPTO record for MEGAPOD (serial 99893717, intent-to-use, filed 18 June 2026) covers modular AI computing hardware and awaits examination (USPTO) [11]. It shows intent, not a product.

Not established by any source we found: a public demo or beta (the six-month target fell around September; no launch seen by 10 October), independent benchmarks, a security or permission model, pricing, and what the edge policy sends to the planner.

Sources for this section: [1] Musk, 11 Mar. [2] Musk, 12 Mar. [3] Teslarati, What is Digital Optimus? [4] Tesla Q2 2026 call transcript (Motley Fool). [5] Tesla Q2 2026 update. [6] CNN, SpaceX acquires xAI. [7] Musk, 12 Aug reply and parent post. [8] Tesla job listing (aggregator). [9] Realtime Games listing (aggregator). [10] Musk, 10 Oct. [11] USPTO, MEGAPOD.

3. A contrast case: Grok Bot is not Digital Optimus

Readers meeting both names will conflate them. Grok Bot runs each member's agent on Cursor-operated hosted cloud computers and acts as that signed-in member, never holding more access than the person (Grok Bot docs) [1]; SpaceX has bought Cursor (CNBC) [2]. The docs describe network allow-lists, per-action approvals, an independent Auto Review model that screens risky actions including computer use, and, on Enterprise plans only, audit logs and 90-day Action Recording; they concede controls "reduce, but do not eliminate" prompt-injection risk (docs) [1].

The docs do not say how the agent perceives the screen, so we draw no conclusion on screenshots versus video. Grok Bot is a useful reference for what a documented permission model looks like, which Digital Optimus has not yet published. The only link we found between the two is Musk's 12 August reply saying they differ (reply) [3].

Sources for this section: [1] xAI, Grok Bot security. [2] CNBC, SpaceX $40B Nvidia financing (notes SpaceX "later bought" Cursor). [3] Musk, 12 Aug.

4. The hardware constraint is the interesting part

Tesla has not published an official TOPS figure for AI4. Third-party analysis puts the HW4 computer at about 100 to 150 INT8 TOPS with roughly 384 GB/s of GDDR6 bandwidth (Electrek) [1]. On the Q1 2026 call Musk called memory bandwidth "the choke point" for inference as planned "AI4 Plus" hardware doubled memory to 32 GB per SoC (Electrek) [2].

A back-of-envelope calculation, ours rather than Tesla's: decoding streams the active weights from memory for every token, so 384 GB/s caps an 8 GB model at roughly 48 tokens per second before batching or sparsity tricks. A frontier-scale VLM cannot be the real-time policy on that budget. This fits a two-tier split, a small, fast policy for reflexes and a large model planning elsewhere, and implies some screen-derived context probably leaves the device; Tesla has not said what is sent.

Tesla's recent decisions agree. On 1 October Musk said AI5 RAM was cut to 72 GB and AI6 to 144 GB to get "enough volume for Optimus production", expecting a "negligible effect" because "memory bandwidth is a bigger limiting factor than total memory storage" (post) [3]. The next day he raised AI5 to 96 GB so Tesla would not be "the only company using the min RAM version of LP5" (archive; Wccftech) [4][5]. "Negligible" is Musk's untested assertion; the takeaway is that memory supply constrains edge agents in 2026 and Tesla treats bandwidth as the governing metric.

On the Q2 call Musk hoped for AI5 volume production "around the middle of next year", initially in Optimus (a hope, not a commitment), so an agent tuned to AI4 should not expect AI5 benefits soon. He also described Megapods holding "a large number of AI4+x86 combos" ("Digital Optimus in a box") that "can be placed anywhere in the country or outside the country", citing about 7 GW of power at Superchargers (transcript) [6]. Our reading, not Tesla's: if customers' agents run there, the edge is vendor-operated and distributed, not on-premises, which matters for data residency and sub-processor reviews.

Sources for this section: [1] Electrek, AI4 vs. Nvidia Thor (third-party estimates). [2] Electrek, HW4 Plus. [3] Musk, 1 Oct. [4] Musk, 2 Oct (archive). [5] Wccftech, AI5 memory report. [6] Tesla Q2 2026 call transcript.

5. What real-time video would and would not fix

OSWorld 2.0 is the best public evidence on where agents fail: 108 long-horizon workflows, median human time about 1.6 hours. The top configuration completes 20.6%, GPT-5.5 plateaus near 14%, and no model exceeds 10% in the 137 to 163 minute bin. Agents fail by losing track of constraints, missing mid-task information, guessing rather than asking and skipping verification, not by struggling with basic GUI control. Only 6 of 108 tasks (5.6%) are tagged "streaming interaction", where the screen changes between observation and action (OSWorld 2.0) [1]. Grounding is another bottleneck: targets in professional high-resolution software average about 0.07% of the screen (ScreenSpot-Pro) [2].

Our assessment: continuous perception targets a narrow, real class of problems (games, video, animated UI, calls, transient dialogs). But non-streaming systems also play games: UI-TARS-2 reports about 60% of human-level performance across a 15-game suite (UI-TARS-2) [3]. For back-office workflows, the limits are state tracking and judgment, which a faster eye does not supply. Test "real time" on your workflows; it is not a general reliability upgrade.

Sources for this section: [1] OSWorld 2.0 (arXiv 2606.29537). [2] ScreenSpot-Pro. [3] UI-TARS-2.

6. Security: what changes and what does not

The visual channel stays an injection channel. Anthropic warns Claude may follow instructions found on screen, and OpenAI says to treat screen content as untrusted (Anthropic [1]; OpenAI [2]). VPI-Bench found computer-use agents deceived at rates up to 51% by visually embedded prompts, with system-prompt defences showing no significant overall effect (VPI-Bench) [3]. The tests used older Claude models, but the mechanism holds: if an agent reads pixels, an attacker only needs content on screen, and a video agent reads more pixels, more often.

Vendor progress is real but conditional. Anthropic reports better robustness yet states that "a 1% attack success rate... still represents meaningful risk" (Anthropic) [4]. OpenAI describes continuous hardening with an automated attacker (OpenAI) [5]. NIST CAISI shows why headline rates mislead: model-specific attacks raised success from 11% to 81%, and 25 repeated attempts raised average success from 57% to 80% (NIST) [6].

Excessive agency is the controlling risk. OWASP's 2025 Top 10 lists prompt injection (LLM01) and excessive agency (LLM06) (OWASP) [7]. An agent usually holds a human's permissions, so a hijack inherits them; joint CISA, NSA and partner guidance (May 2026) treats agents as least-privilege identities with accountable logging (CISA) [8].

Our inference for the video case (no published findings). A streaming agent can react to content visible for under a second, which a snapshot sampler or a reviewer might miss. Auditability gets harder: the guidance warns agent logs are large and hard to analyse (CISA) [8], and a policy emitting controls at frame rate has no natural step-by-step trail; Tesla's "interpretable" requirement is a goal, not a delivered capability. And speed shrinks the window for human intervention, so confirmations must be enforced outside the model, as VPI-Bench's authors also suggest (VPI-Bench) [3].

Routing changes the data path. On 7 October Musk said SpaceX "will use the best back end model for any given task, including Claude Opus 5.5, MidJourney, Suno and other leading APIs" in Grok Bot (post) [9]. The product docs say model selection is vendor-managed with "no fixed vendor set guaranteed", billing follows the serving model, and an Enterprise allow-list's "enforcement is not guaranteed"; customers whose contracts limit sub-processors are told to contact their account team first (docs) [10]. One product name can thus mean several sub-processors that change over time. Routing by task echoes Tesla's edge/planner split, so the buyer's question is less "screenshots or video?" than "which tier sees my data, and where does it run?". That Digital Optimus will adopt similar routing is our hypothesis; no source says so.

Privacy and regulation. Anthropic notes computer use is client-side, with screenshots stored in your environment and zero-data-retention eligibility, a control an opaque edge pipeline may not offer (Anthropic) [1]. For agents watching employees' screens, the Article 29 Working Party stresses proportionality and transparency (WP29) [11], and the UK ICO lists new technologies and systematic monitoring as DPIA triggers (UK guidance, persuasive in the EU) (ICO) [12]. Do a DPIA and consult employee representatives before a pilot. On the EU AI Act, a Freshfields summary of the final Digital Omnibus says stand-alone high-risk obligations are deferred to 2 December 2027, while the Article 50 duty to tell users they are dealing with AI applies from 2 August 2026 (Freshfields) [13]. A general agent is not high-risk merely by existing, but can become so for listed purposes such as employment decisions; take legal advice.

Sources for this section: [1] Anthropic, Computer use tool. [2] OpenAI, Computer use guide. [3] VPI-Bench. [4] Anthropic, Prompt-injection defenses. [5] OpenAI, Hardening Atlas. [6] NIST CAISI, Agent hijacking evaluations. [7] OWASP LLM Top 10 2025. [8] CISA et al., Careful adoption of agentic AI. [9] Musk, 7 Oct. [10] xAI, Grok Bot security. [11] WP29 Opinion 2/2017. [12] ICO, When do we need a DPIA?. [13] Freshfields, EU AI Act unpacked #34.

7. Concentration and counterparty risk

The Digital Optimus stack depends on closely linked companies: Grok under SpaceX, AI4/AI5 silicon at Tesla and a planned in-house fab. On 7 October Musk said "we will build and run the fab", with TSMC at most subleasing part of Terafab (post) [1]; that is intent, not delivery. SpaceX is reportedly in preliminary talks with Apollo and banks on financing a $40 billion Nvidia GPU purchase, per an anonymous source and unconfirmed by the parties (CNBC) [2]. SpaceX also leases compute to Anthropic; its filing allows either party to terminate on 90 days' notice, while Musk described a 180-day lease with 90-day cancellation thereafter (TechCrunch) [3].

Our inference: buyers should avoid designs where one vendor's agent, models and hardware are inseparable, and keep the ability to swap the planner or the executor.

Sources for this section: [1] Musk, 7 Oct (Terafab). [2] CNBC, Apollo, banks in talks on SpaceX's $40B Nvidia purchase (FT reported first). [3] TechCrunch, How long is Anthropic's lease with SpaceX?

8. What organisations should do now

  1. Design for observation-agnostic control. Permissions, logging and approvals should work whether the model sees snapshots or video.
  2. Prefer APIs and connectors; keep GUI control as a fallback.
  3. Isolate and de-privilege. Dedicated VM, separate low-privilege identity, short-lived credentials, domain allow-list, as vendors advise (Anthropic [1], OpenAI [2]).
  4. Enforce approvals outside the model. Gate payments, outbound data, deletions and permission changes; add step, time and cost budgets and a kill switch. Even Grok Bot's reviewer model "does not review every side effect" (docs) [3].
  5. Record what you can replay. Keep screenshots or sampled video, parsed actions and state checks, with retention agreed with your DPO.
  6. Red-team the pixels, repeatedly. Test injected on-screen text and pop-ups with VPI-Bench-style cases over many attempts (VPI-Bench [4], NIST [5]).
  7. Benchmark on your tasks. Treat OSWorld 2.0 (about one in five long workflows completed) as a baseline (OSWorld 2.0) [6].
  8. Decide data locality explicitly. Ask any two-tier vendor what the planner receives and where it runs.
  9. Fix model providers in the contract. Require a contractually enforced provider list (not a best-effort allow-list), notice before the serving mix changes, and per-request logging of which model handled what (docs) [3]. Also cover sub-processors, retention, audit access, incident notification and rollback.
  10. Plan the exit. Keep planner and executor swappable (section 7).
  11. Wait for evidence, not announcements: unedited demos, independent benchmarks, a security whitepaper and a documented permission model.

Sources for this section: [1] Anthropic, Computer use tool. [2] OpenAI, Computer use guide. [3] xAI, Grok Bot security. [4] VPI-Bench. [5] NIST CAISI. [6] OSWorld 2.0.

Bottom line

On the record, Tesla and SpaceX have described a two-tier design (a fast video-in, controls-out policy on AI4 plus a large planner), are hiring for it, report game-playing progress and have filed a trademark for the hardware. Whether it works on enterprise workflows, when it ships, what it costs, which models it routes to and how it is secured are open. The lesson holds regardless: the interface to a computer is an attack surface whether delivered as snapshots or video, and the controls that matter sit around the model.

Analysis, not legal or investment advice. Sources sit beside the claims they support; aggregator and third-party estimates are marked as such.