ScreenMirror Control API

A client keeps one WebSocket open to the relay. It may open a second one just for input.

Open a session

1. Open /view. A relay with accounts sends auth-required.
2. Sign in with auth. You receive auth-ok, then hello and host-list.
3. Open a machine with select and its number. unselect takes you back to the list.
4. Optional: open /ctl and send ctl-attach with the ctlKey from hello, within 10 seconds.
5. Send control messages.

People sign in with their account. There are no API keys for people. What you may do comes from your account's session at the relay.

Replies

Access tokens last 60 minutes. The refresh token renews them without a password, also on sibling relays that share the account database.

Every message is one JSON object in one WebSocket text frame.

• Put type first. Hosts look for the literal text "type":"…", and some expect it at the very start of a request.
• Keep objects flat. The host finds fields by name. A nested object that reuses a field name can be read as the outer one.
• Number your requests. A message that expects an answer carries a numeric reqId. The reply repeats it, and the relay delivers that reply to you alone.
• Expect new fields. Replies gain fields over time. Ignore the ones you don't know.

A host can run with remote input switched off. It still streams and answers questions, but it injects nothing. Here is what happens to each kind of message.

Pointer positions are fractions of the picture, from 0 to 1. They are never pixels.

• Divide by the picture's size: x = px / w and y = py / h. A screenshot's first reply gives you w and h.
• Round to five decimal places.
• Stay below 1 by clamping to 0.99999. Exactly 1 is one pixel past the edge, or the first pixel of the next screen.
• Outside the picture, send nothing.
• Take screenshots of the same screens the stream shows you, so the fractions mean the same thing.

How the host places the pointer

It checks these cases in order.

1. One stream per screen. When the host encodes each screen on its own, the fraction covers that whole screen. The relay adds mon for you.
2. A cropped view. If you were showing part of the screen, the fraction is inside that crop. Send full: 1 if you already converted it to the whole screen.
3. A merged wall. The fraction covers every screen, and the pointer crosses from one to the next.
4. A single screen. The fraction covers that screen.

Moves, presses, releases and scrolls. One message becomes one input event on the machine.

Behaviour you can rely on

• An unknown button does nothing. It never falls back to a left click.
• Clicks and wheel steps move the pointer first, so they land where you aimed.
• Your first mouse or key marks you as driving, and other viewers see it.
• If another operator used input in the last 5 seconds, the relay removes m. One person's Ctrl never changes another person's click.
• When the host offers a UDP path, the relay sends your moves over it. You don't need to do anything.

Presses and releases keys. Send a down and then an up for every key.

How the host picks the key

1. By position, with sc. For letters, digits, punctuation and AltRight. The host presses that physical key and its own layout picks the character, so KeyQ types "a" on a French machine.
2. By name, with code. For keys that mean the same on every layout: Enter, arrows, F-keys and modifiers.
3. By character, with code: "" and text. Skips the layout entirely, so accents and any script arrive exactly as sent.

Behaviour you can rely on

• A release undoes its own press. If you pressed by position, the matching up releases that position, whatever the up says.
• A line feed or tab in text is sent as the Enter or Tab key. Most Windows programs don't treat a raw line feed as Enter.
• sas raises Ctrl+Alt+Del through the host's service. It can't be pressed as three keys.
• Keys you leave down are released when the stream stops, or when the relay tells the host to let go.
• The relay adds vid to every key so the host can tell viewers apart. Don't send it yourself.

There is no message for a block of text. Type it as key messages, one character at a time. The viewer's auto-type works the same way.

• Walk the text by code point, so an emoji is one step and a surrogate pair stays whole.
• Send each character as code: "" with text: a down, then an up.
• Send line breaks and tabs as the Enter and Tab keys.
• Release Shift and Ctrl before the first character. Don't send a lone Alt or Win release, because it opens a menu or Start.
• Leave at least 40 ms between presses.
• Stop when you lose Control, or when the machine reports someone typing at it (local-input).

The viewer's pacing

Average time per key. "Like a person" holds each key about 90 ms, keeps the rhythm of recorded typing and makes occasional corrected typos (1.0 to 2.6%).

Sets the machine's clipboard text. The machine sends you its clipboard when it changes.

From the machine you receive clipboard with text, or clipboard-files with a files list. Your own write is not echoed back.

Files are offered, not pushed. You advertise them, and the machine asks for the bytes when someone pastes.

Takes a lossless PNG of one or more screens and returns it in base64 slices. The first reply carries the size, and you pull the rest.

shot

shot-get

Reply: fs-data

Limits

• The host keeps a picture for 10 minutes, holding the newest 3 and at most 64 MB in all.
• Up to 4 screenshot or reading jobs wait at once. More are refused with "too many screenshots at once".
• A shot always covers whole screens. len limits bytes on the wire only.
• On a host running as a service, sign-in and UAC screens are captured too.

Reads the words in a region of a picture the host is holding. The picture stays on the machine, and only the words come back.

The reply

An fs-data with an ocr object. Each line has its text in t and its words in w. A word is [text, x, y, width, height], measured from the region's top-left corner. Add the region's x and y to get picture pixels.

Two more forms

• {"type":"shot-ocr","reqId":10,"langs":1} lists the languages the machine can read.
• {"type":"shot-ocr","reqId":11,"add":"ja-JP"} installs a language through Windows Update, which can take up to 30 minutes. Never allowed on an agent grant.

Take one picture, then ask about parts of it. A full 4K PNG is megabytes; the words in a dialog are a few hundred bytes.

1. Send shot with len: 4096. Keep id, w and h from the first reply and ignore its bytes. That costs about 5.5 KB.
2. Send shot-ocr against that id for each region you care about. No new capture happens.
3. Pull PNG slices only when you need pixels. PNG stores rows top to bottom, so the first slices decode to the top of the picture.

Tells the relay you are driving the machine, or that you have stopped. Every viewer of the machine sees who is driving.

• You rarely need it. Your first mouse or key turns it on.
• If someone holds the machine exclusively, the relay answers control-revoked with their name in by.

Holds the machine for you alone. The relay drops every other operator's input until you release it or your connection ends.

• Ignored while someone else holds the lock.
• Ignored while a top-tier administrator's Solo blocks other operators.
• An agent's lock ends with its visit. Ask the people watching before you take it, and release it when you are done.

Blocks input at the machine, from other operators, or both. Solo is a setting of the machine, not of your session. It survives a restart and holds while nobody is watching.

• To change local or remote, turn Solo off and on again.
• A top-tier administrator releases it, or the chord typed at the machine.
• Ctrl+Alt+Del always works at the machine.

Blanks the machine's own monitors while your stream keeps showing the real desktop. The person at the machine sees a black screen.

• It lasts for the session. When the last viewer leaves, the monitors come back, so nobody is left in front of a black screen with no one on the other end.
• Refused while another operator holds the machine exclusively.

Sends your typing to one window without moving the focus. A person at the machine can keep typing somewhere else at the same time.

Hover the field you want, then send on: 1. The host answers key-route-state with on, your vid, and a reason when it refuses.

It only redirects your own keys and never blocks anyone, so any viewer may use it.

Runs one command on the machine and returns its exit code and output.

Reply: exec-result

• Windows runs it with cmd.exe /c. Call PowerShell yourself, as in powershell -NoProfile -Command "…". Linux runs it with /bin/sh -c.
• The host stops a command after 10 seconds and keeps the first 512 KB of output.
• Commands run on their own worker, so a slow one never holds up input. When the queue is full, the reply says the host is busy.
• Remote sign-out and restart are commands too.

These change the machine's state without injecting any input, so they work on view-only machines too.

An AI agent has no account. It asks to visit one machine, and a person watching that machine says yes or no. Nothing gets in without that press, unless an owner approved the key in advance.

1. Open /view. Don't sign in.
2. Send join-begin with the relay's join key and the machine number.
3. You receive a join-code. Everyone watching the machine sees a card with the same code, your name and your task.
4. The first person to answer decides, within 2 minutes. Yes gives you auth-ok. No gives you join-nak with denied.
5. select the machine and work within your scopes.
6. Send join-leave when you are done.

When a visit ends

• After 60 minutes. The relay's SM_JOIN_MINUTES changes this, from 5 to 1440.
• When an operator ends it.
• When you send join-leave, which answers join-done. The other two end with access-denied.

no, no-host and nobody all arrive after the same 400 ms, so timing can't tell them apart.

During a visit

The relay's one HTTP endpoint answers a machine's status as JSON. It needs no sign-in.

Add ?fresh=1 to skip the cache.

Status codes