ACTUAL CODEX RUNS · INDEPENDENT STATE VERIFICATION
N tasks. Less repeated discovery.
Run: sdk-guided-2026-10-07 · Measured October 7, 2026 at 1:05 AM (America/New_York)
In this controlled pilot, Smart365 had 48.4% lower mean token use and 15.9% lower median confirmation time.
Codex browser only → Codex + Smart365Guide · gpt-5.3-codex · Controlled pilot
Mean token use
48.4% less
Normalized token index: 100 → 51.6Median confirmed time
15.9% less time
Normalized median time index: 100 → 84.1What this means for N tasks
N represents the number of tasks in your workload. T represents your browser-only agent’s average tokens per task. If the pilot’s measured token ratio holds for that workload:
| Token budget | Browser-only agent | Smart365Guide + agent |
|---|---|---|
| N tasks | N × T | Approximately 0.516 × N × T |
On the same normalized scale, median completion time per successful task was 100 → 84.1. Total time for N tasks also depends on task complexity, concurrency and integration overhead.
These ratios are normalized from the controlled Codex pilot. The N-task expression is conditional arithmetic, not a measured large-workload result or a guarantee. Other agents and multi-step journeys need their own validation. Input and output tokens include cached input once; token reduction is not a measured billing discount.
From business actions to multi-step journeys
The measured pilot covers individual business actions. Our roadmap extends the guide to multi-step journeys across screens and connected services; larger performance gains need a separate controlled test.
Smart365Guide’s vision is a shared application guide for people and authorized AI agents:
- Application knowledge: a structured graph of screens, fields, available dropdown choices, transitions and approved actions. Retrieve only the relevant, permitted context and refresh choices that depend on live data or the user’s role.
- For agents: registered clients request scoped access and use published workflows to reach a goal, collect required inputs and verify the result. Registration alone does not authorize an action.
- For people: embedded chat explains the journey, asks for missing information and helps users navigate or take approved actions. Owners choose which workflows and approvals to provide.
Automatic workflow discovery, owner-reviewed publishing and goal-based planning across connected services are roadmap work. Multi-step performance needs a separate controlled benchmark; the measurements above remain the original Codex pilot.
View benchmark methodology and recorded evidence
Previous run compared with this update
Compared with the previous Smart365 run, this update had 59.8% lower mean token use and 33.0% lower median confirmation time.
Earlier run: updated-2026-10-07. Updated run: sdk-guided-2026-10-07. The refreshed browser-only control is included because caching, model variation and network conditions can change across runs.
| Metric | Previous control | Previous Smart365 | Updated control | Updated Smart365 |
|---|---|---|---|---|
| Verified jobs | 9/9 | 9/9 | 9/9 | 9/9 |
| Mean tokens / run | 46,326 | 65,594 | 51,156 | 26,386 |
| Mean uncached input tokens | 8,345 | 10,897 | 9,296 | 4,982 |
| Median confirmation seconds | 12.35 | 18.03 | 14.35 | 12.07 |
| Mean model calls | 5.0 | 5.9 | 5.6 | 2.4 |
Open preserved previous report · First benchmark · Previous CSV
Compare the same task
Bars show measured mean tokens per run. They do not measure billing cost.
| Metric | Codex browser only | Codex + Smart365 |
|---|---|---|
| Verified jobs | 9/9 | 9/9 |
| Verified jobs executed through Smart365 | Not applicable | 9/9 |
| Assisted successes using browser fallback | Not applicable | 0 |
| Mean tokens / run (input + output) | 51,156 | 26,386 |
| Mean input tokens | 50,598 | 25,918 |
| Mean cached input tokens (included above) | 41,301 | 20,935 |
| Mean uncached input tokens | 9,296 | 4,982 |
| Mean output tokens | 558 | 469 |
| Median Codex execution (seconds) | 14.22 | 11.88 |
| Median time to independently confirmed state (seconds) | 14.35 | 12.07 |
| Mean account setup seconds (outside timed task) | 0.33 | 1.90 |
| Mean model tool calls / run | 5.6 | 2.4 |
| Mean browser / Smart365 MCP calls / run | 5.6 | 2.4 |
| Smart365 provider calls (total, separate from model calls) | 0 | 28 |
| SDK contract reads during startup (total, separate from model calls) | 0 | 27 |
| Mean SDK guide and authorization startup seconds (inside task clock) | Unavailable | 1.31 |
| SDK initialization HTTP responses / failed responses | 0 / 0 | 9 / 0 |
| Tool errors | 0 | 0 |
| Runs with verification mismatch and recovery | 0 | 0 |
| Timed-out runs | 0 | 0 |
| Runs with confirmed licensed SDK initialization | 0 | 9 |
| Additional SDK chatbot model requests | 0 | 0 |
Per-task measurements
Every recorded trial
| Task / repeat | Arm | Verified | Tokens | Run + state check | Tool calls |
|---|---|---|---|---|---|
| create-job / 1 | control | Yes | 56,965 | 17.60s | 6 |
| create-job / 1 | assisted | Yes | 31,202 | 14.33s | 3 |
| reschedule-job / 1 | assisted | Yes | 22,572 | 11.20s | 2 |
| reschedule-job / 1 | control | Yes | 45,918 | 14.35s | 5 |
| complete-job / 1 | control | Yes | 53,577 | 13.38s | 6 |
| complete-job / 1 | assisted | Yes | 22,677 | 11.96s | 2 |
| create-job / 2 | assisted | Yes | 23,080 | 13.24s | 2 |
| create-job / 2 | control | Yes | 38,391 | 11.75s | 4 |
| reschedule-job / 2 | control | Yes | 55,126 | 14.39s | 6 |
| reschedule-job / 2 | assisted | Yes | 30,845 | 12.07s | 3 |
| complete-job / 2 | assisted | Yes | 22,794 | 11.22s | 2 |
| complete-job / 2 | control | Yes | 53,625 | 13.14s | 6 |
| create-job / 3 | control | Yes | 57,219 | 16.30s | 6 |
| create-job / 3 | assisted | Yes | 30,839 | 12.94s | 3 |
| reschedule-job / 3 | assisted | Yes | 22,782 | 11.73s | 2 |
| reschedule-job / 3 | control | Yes | 45,978 | 13.57s | 5 |
| complete-job / 3 | control | Yes | 53,604 | 14.69s | 6 |
| complete-job / 3 | assisted | Yes | 30,687 | 12.47s | 3 |
Method, attribution and limits
- 3 repeats × 3 tasks × 2 arms. 18 recorded runs; the planned set is complete.
- Both arms use Codex CLI 0.159.0-alpha.3, gpt-5.3-codex, low reasoning effort, fresh sessions and isolated seeded trial accounts. Arm order alternates by task and repetition.
- Tasks: create a correctly assigned job, reschedule an existing job while preserving other fields, and complete a job with an exact note. Browser fills can batch multiple fields, so the control does not incur artificial one-field-per-tool overhead.
- Success is checked by a separate authenticated database-state read: exact requested fields, all unrelated jobs preserved, expected record count, and exactly one business write. Chat messages claiming success are not evidence.
- 9/9 assisted tasks used a verified Smart365 execution; 0 succeeded through browser fallback. Business outcome success is reported separately from gateway-only success.
- The licensed SDK initialized in 9/9 assisted trials. Initialization failures and all recovery overhead remain included; a correct final business outcome does not erase an integration error.
- Recorded model-visible Smart365 calls requested full diagnostic responses 0 times and compact/default responses 10 times. Agents made 0 inspections and 1 explicit verification calls. Typed SDK actions enforce compact receipts; full diagnostics are kept outside normal model context.
- Both sites share business UI/backend source. The assisted host registers an SDK app manifest and bounded fresh customer/job context. An external adapter reads the licensed SDK guide, retrieves each current authorized server contract and uses the SDK factory to expose typed business tools. The overlay alone does not give Codex this knowledge.
- The timed workflow starts with opening the site in both arms. Confirmation time includes Codex runtime, browser/SDK initialization, model reasoning, tool requests and an independent state check. Account reset, consent/key setup and revocation are excluded and retained as separate setupMs values.
- Browser presentation uses local HTTP to avoid changing this workspace’s Chromium TLS trust store. Both use fixed HTTPS proxies to their own hosted Vercel backends; Smart365 provider calls execute remotely. TLS verification remains enabled. This is not a direct remote-browser latency test.
- Tokens are actual CLI turn.completed usage. Total = input + output; cached input is a subset of input and is not added again. Output includes reasoning tokens when the provider includes them. Missing usage remains null, not zero. No billed dollar-cost measurement is available.
- The CLI emitted a model-metadata fallback warning for this API provider in both arms; the same fixed requested model and CLI configuration were used throughout.
- Model tool-call counts include metadata-resource lookups. Smart365 performs a lookup, write and independent provider read; explicit verify calls may add further reads. These provider calls are recorded separately, so fewer model calls do not imply fewer total API calls.
- The adapter fixes compact receipts and derives idempotency keys from a unique request ID plus capability ID. Exact customer names, New York wall time and verbatim notes are defined in the published typed contracts. No task answer is embedded in the guide.
- Startup contract inspection occurs inside the timed Codex process: 27 gateway reads across assisted runs, separate from model tool calls. No startup knowledge retrieval is hidden in account preparation. Successful receipts update the host through an SDK event and the host rereads its authorized backend.
- An SDK engineering preflight caught a guide-read race after page reload. The adapter now waits for host/SDK registration before reading the guide. The failed preflight log remains archived separately; no measured trial was replaced.
- Permission checks also verified that missing write scope, changed-input replay and revoked agent keys are rejected. Replaying the same operation produces one saved business write.
- The original run exposed an idempotency/connection-change issue. Updated production smoke checks prove that changed-account replay and verification are rejected without another write. This run supplies fresh request identifiers. The original failure and recovery remain in the separately preserved baseline report.
- Provider-call totals count incremental calls per unique execution ID, preventing the reused older execution from being counted twice. They cover the measured task set, not SDK telemetry or all server-internal requests.
- Engineering preflights are archived separately. They exposed a scoped tool-approval problem, MCP proxy inheritance, and ambiguous note punctuation; those were fixed before the main set. All main-set failures remain included. No reruns replace main-set records.
- Small controlled pilot on a synthetic app; not a universal browser/native-app benchmark.
- Assisted arm adds registered structured capabilities and API execution; SDK-alone effect is not isolated.
- No raw-API-only third arm, so gains cannot be attributed uniquely to Smart365 versus direct APIs.
- Model caching, cloud cold starts and network latency can affect timing.
- No billed-cost claim; token totals include cached input once.
- Both arms use the same model/tasks and begin with browser_open. Assisted MCP startup opens the app, reads its licensed SDK guide, obtains authorized contracts and exposes typed actions. All discovery/browser startup is inside the measured Codex process time. The adapter fixes compact output and retry keys; the model supplies only business inputs. The assisted first observation is structured SDK context; the control gets visible UI.
- Updated platform source 290803fcf0696ed0f0541fe981e4e4f8ed272f8b; production deployment dpl_8x2Bhjx5LZJqit8izRyNzTTTSHKx. This is a combined SDK guide, typed contract, context and adapter update; it does not isolate any one change. Historical comparisons do not control every caching/network effect.
- This benchmark does not test the proposed native desktop/iOS adapters, arbitrary external agents or company entitlements. It measures the current web integration, not the entire future platform.
The benchmark is a separate FieldDesk project with fictional records. No benchmark application code was added to the Smart365 repository.