An architecture writeup of a personal system I designed, built, and operate. It's described at the "you could rebuild it" level — the code and my own usage stay private.
Shape of the system
Two pieces. The agent is a mobile app: an always-on assistant that plans, runs tools, and holds long-term context. The backend is a self-hosted proxy to a frontier model, deployed on a small cloud instance and reachable only over a private network. The split matters: the model lives behind my own boundary, the agent logic lives on the client, and nothing about the design depends on a third-party assistant platform.
The model backend
A small Node service in front of a managed model API (Claude on AWS Bedrock). Three properties drove the design:
- Web-free by construction. The proxy makes exactly one kind of outbound request — to the model provider over TLS. It never browses, never calls arbitrary services. Any tool that needs the web runs on the client instead, so the server has no path to the open internet.
- Private by network. No public ingress. The service is only reachable over a private mesh network; the box denies inbound traffic otherwise.
- Model-routed and async-capable. Requests select a model tier by field (a fast default, a premium tier for hard problems). Long jobs can be submitted as background tasks the client polls for — single-consumer, in-memory, short-TTL — so a phone can kick off a slow inference and collect it later without holding a connection open.
Model routing across tiers
Not every query deserves a frontier model, and not every query should leave the device. The agent routes across three tiers by cost, latency, and capability:
- On-device. A small Qwen model runs in-app for simple, offline queries — no network round-trip, and it keeps working with no signal at all.
- Hosted, via OpenRouter. Mid-weight work routes through OpenRouter — MiniMax M3 for general reasoning and Qwen 2.5 VL 72B for image understanding.
- Frontier, self-hosted. The hardest problems go to Claude behind my own Bedrock proxy (above) — the private, web-free boundary I operate.
Provider access is abstracted behind one interface, so a task lands on the cheapest tier that can handle it and escalates only when it needs to — and the offline tier means the assistant degrades gracefully instead of failing when there's no connection.
The agent
The interesting engineering is in the harness around the model, not the model call itself.
- Tag-based tool protocol. Rather than a rigid function-calling schema, the model emits lightweight tagged blocks in its output; the client parses them and executes the corresponding tool, then feeds results back. It's forgiving, easy to extend with a new tool, and keeps tool execution on the client where the credentials and context live.
- Rolling memory. Context is a pyramid: recent turns in full detail, older history progressively summarized, and a durable archive underneath. The agent keeps a working set that fits the model's window while still being able to reach back — memory is curated over time, not just truncated.
- Client-side tools + vision delegation. Research, retrieval, and image understanding run as client tools. Vision in particular is delegated explicitly, with user consent at the moment it's needed, so the system never quietly ships images off-device.
- Multi-agent orchestration. Work is decomposed across cooperating agent roles — planning, tool execution, and summarization — coordinated by the harness rather than a single monolithic prompt. I also routinely hand a thread to a second model mid-conversation for an independent opinion, which the tiered routing makes a one-line switch.
Why build it this way
The point was ownership. A self-hosted model boundary, a tool protocol I control, and memory I can inspect and edit — a personal assistant whose behavior is mine to shape, and whose data path I can reason about end to end. It's also the clearest demonstration of the kind of work I like: building the harness — the tooling, routing, memory, and orchestration — that turns a raw model into a dependable system.