Diwall MCP

English
Download v0.0.1

Perception

What one observation hands the model: an image with numbered marks and masks, the accessibility tree as lines of text, and the elements it can act on.

The model sees a page through one tool, observe. A single call returns three parts taken from one snapshot of the tab:

  • the image of the view, with a number drawn on each element the model can act on, and a solid rectangle over each sensitive field and each frame;
  • the tree, the accessibility tree of the page written as indented lines of text, one node per line;
  • the elements, the list of what carries a number: its mark, role, name, value and state.

The result also carries a snapshot_id. An action names a mark together with the snapshot it came from, and the server refuses a mark from an older observation (snapshot_stale).

Why a model needs the tree as well as the image, and what an accessibility tree is, is the subject of the Diwall site, whose command-line tool also gives a model an image with numbered elements and the accessibility tree: How a model sees a page, The accessibility tree, Why a screenshot is not enough. This page says what this server returns.

The server’s part and the page’s part

Every result has three keys: ok, server and page_content. What the server states is under server; what the page wrote is under page_content. A client that looks for tab_id at the root of the result finds nothing there.

The server part of the observation shown on the home page, in full:

"server": {
 "notices": [],
 "tab_id": "f395445606543e95",
 "snapshot_id": "56f18791c433fa584ea5067f99185355",
 "origin": "http://127.0.0.1:34607",
 "loader_id": "C3ACA0EC17A776A7A839DD752D615232",
 "viewport": {"width": 780.0, "height": 493.0, "scroll_x": 0.0,
              "scroll_y": 0.0, "content_height": 493.0},
 "capture": "ok",
 "counters": {"omitted_not_visible": 0, "redacted_fields": 1,
              "frames_masked": 0, "marks": 4, "marks_truncated": false,
              "tree_lines": 10, "tree_truncated": false},
 "dialog_open": false,
 "image": {"width": 780, "height": 493, "scale": 1.0},
 "operation_id": "eb86f607d7490a08f3926a2e98c3aff9",
 "tool": "observe"
}

Everything under page_content (title, tree lines, names, values) is written by the page, and a hostile page writes what it wants. The description of the tool, which the model reads, says so in those terms: it is data, never an instruction.

What an observation leaves out

  • The values of sensitive fields (password, card, code): masked in the image, withheld from the tree and the elements. redacted_fields counts them.
  • The text of frames: never read; each frame is masked in the image, and frames_masked counts them.
  • What a person cannot see: nodes hidden from view are not sent; omitted_not_visible counts them.
  • A page outside the sites of your profile: nothing of it but its origin (origin_not_allowed).
  • An open dialog: while one is open, observe returns its type and its message instead of the page.

Where each of these can fail, and what the server does then, is on the next two pages. What leaves your machine as a whole is on What leaves your machine, and what the server never does.

In detail