ACTUAL CODEX RUNS · INDEPENDENT STATE VERIFICATION
Does Smart365 improve an agent’s work?
Run: updated-2026-10-07 · Measured October 7, 2026 at 12:32 AM (America/New_York)
In this controlled pilot, Smart365 had 41.6% higher mean token use and 46.0% higher median confirmation time.
18 measured runs · gpt-5.3-codex · 3 business tasks · Complete controlled pilot
Independently verified
9/9 → 9/9
Control → Smart365Mean token use
46,326 → 65,594
Input + output, cached input included onceMedian confirmed time
12.35s → 18.03s
Successful runs onlyPrevious run compared with this update
Compared with the previous Smart365 run, this update had 11.7% lower mean token use and 8.4% lower median confirmation time.
Earlier run: main-2026-10-06. Updated run: updated-2026-10-07. The refreshed browser-only control is included because caching, model variation and network conditions can change across runs.
| Metric | Previous control | Previous Smart365 | Updated control | Updated Smart365 |
|---|---|---|---|---|
| Verified jobs | 9/9 | 9/9 | 9/9 | 9/9 |
| Mean tokens / run | 56,397 | 74,296 | 46,326 | 65,594 |
| Mean uncached input tokens | 7,976 | 12,967 | 8,345 | 10,897 |
| Median confirmation seconds | 13.13 | 19.68 | 12.35 | 18.03 |
| Mean model calls | 6.1 | 6.6 | 5.0 | 5.9 |
Compare the same task
Bars show measured mean tokens per run. They do not measure billing cost.
| Metric | Codex browser only | Codex + Smart365 |
|---|---|---|
| Verified jobs | 9/9 | 9/9 |
| Verified jobs executed through Smart365 | Not applicable | 7/9 |
| Assisted successes using browser fallback | Not applicable | 2 |
| Mean tokens / run (input + output) | 46,326 | 65,594 |
| Mean input tokens | 45,778 | 64,543 |
| Mean cached input tokens (included above) | 37,433 | 53,646 |
| Mean uncached input tokens | 8,345 | 10,897 |
| Mean output tokens | 548 | 1,051 |
| Median Codex execution (seconds) | 12.14 | 17.77 |
| Median time to independently confirmed state (seconds) | 12.35 | 18.03 |
| Mean account setup seconds (outside timed task) | 0.29 | 2.08 |
| Mean model tool calls / run | 5.0 | 5.9 |
| Mean browser / Smart365 MCP calls / run | 5.0 | 5.9 |
| Smart365 provider calls (total, separate from model calls) | 0 | 24 |
| Tool errors | 0 | 1 |
| Runs with verification mismatch and recovery | 0 | 3 |
| Timed-out runs | 0 | 0 |
| Runs with confirmed licensed SDK initialization | 0 | 8 |
| Additional SDK chatbot model requests | 0 | 0 |
Per-task measurements
Every recorded trial
| Task / repeat | Arm | Verified | Tokens | Run + state check | Tool calls |
|---|---|---|---|---|---|
| create-job / 1 | control | Yes | 57,115 | 18.77s | 6 |
| create-job / 1 | assisted | Yes | 92,304 | 30.63s | 8 |
| reschedule-job / 1 | assisted | Yes | 31,469 | 14.83s | 3 |
| reschedule-job / 1 | control | Yes | 37,155 | 12.17s | 4 |
| complete-job / 1 | control | Yes | 53,648 | 15.31s | 6 |
| complete-job / 1 | assisted | Yes | 42,727 | 16.92s | 4 |
| create-job / 2 | assisted | Yes | 117,746 | 32.51s | 10 |
| create-job / 2 | control | Yes | 47,703 | 14.19s | 5 |
| reschedule-job / 2 | control | Yes | 37,190 | 10.39s | 4 |
| reschedule-job / 2 | assisted | Yes | 42,428 | 16.81s | 4 |
| complete-job / 2 | assisted | Yes | 43,007 | 18.03s | 4 |
| complete-job / 2 | control | Yes | 44,871 | 12.26s | 5 |
| create-job / 3 | control | Yes | 57,381 | 18.55s | 6 |
| create-job / 3 | assisted | Yes | 132,474 | 33.09s | 11 |
| reschedule-job / 3 | assisted | Yes | 35,046 | 13.80s | 4 |
| reschedule-job / 3 | control | Yes | 37,051 | 10.40s | 4 |
| complete-job / 3 | control | Yes | 44,822 | 12.35s | 5 |
| complete-job / 3 | assisted | Yes | 53,148 | 19.55s | 5 |
Method, attribution and limits
- 3 repeats × 3 tasks × 2 arms. 18 recorded runs; the planned set is complete.
- Both arms use Codex CLI 0.159.0-alpha.3, gpt-5.3-codex, low reasoning effort, fresh sessions and isolated seeded trial accounts. Arm order alternates by task and repetition.
- Tasks: create a correctly assigned job, reschedule an existing job while preserving other fields, and complete a job with an exact note. Browser fills can batch multiple fields, so the control does not incur artificial one-field-per-tool overhead.
- Success is checked by a separate authenticated database-state read: exact requested fields, all unrelated jobs preserved, expected record count, and exactly one business write. Chat messages claiming success are not evidence.
- 7/9 assisted tasks used a verified Smart365 execution; 2 succeeded through browser fallback. Business outcome success is reported separately from gateway-only success.
- The licensed SDK initialized in 8/9 assisted trials. Initialization failures and all recovery overhead remain included; a correct final business outcome does not erase an integration error.
- Recorded Network calls requested full diagnostic responses 16 times and compact/default responses 9 times. Agents made 0 inspections and 3 explicit verification calls. Compact support does not imply the model chose compact output for every call.
- Both sites share identical business UI and backend source. The assisted arm loads the licensed deployed SDK, then Codex can discover, inspect and execute registered Smart365 capabilities through the real permission-checked gateway.
- The timed workflow starts with opening the site in both arms. Confirmation time includes Codex runtime, browser/SDK initialization, model reasoning, tool requests and an independent state check. Account reset, consent/key setup and revocation are excluded and retained as separate setupMs values.
- Browser presentation uses local HTTP to avoid changing this workspace’s Chromium TLS trust store. Both use fixed HTTPS proxies to their own hosted Vercel backends; Smart365 provider calls execute remotely. TLS verification remains enabled. This is not a direct remote-browser latency test.
- Tokens are actual CLI turn.completed usage. Total = input + output; cached input is a subset of input and is not added again. Output includes reasoning tokens when the provider includes them. Missing usage remains null, not zero. No billed dollar-cost measurement is available.
- The CLI emitted a model-metadata fallback warning for this API provider in both arms; the same fixed requested model and CLI configuration were used throughout.
- Model tool-call counts include metadata-resource lookups. Smart365 performs a lookup, write and independent provider read; explicit verify calls may add further reads. These provider calls are recorded separately, so fewer model calls do not imply fewer total API calls.
- Permission checks also verified that missing write scope, changed-input replay and revoked agent keys are rejected. Replaying the same operation produces one saved business write.
- The original run exposed an idempotency/connection-change issue. Updated production smoke checks prove that changed-account replay and verification are rejected without another write. This run supplies fresh request identifiers. The original failure and recovery remain in the separately preserved baseline report.
- Provider-call totals count incremental calls per unique execution ID, preventing the reused older execution from being counted twice. They cover the measured task set, not SDK telemetry or all server-internal requests.
- Engineering preflights are archived separately. They exposed a scoped tool-approval problem, MCP proxy inheritance, and ambiguous note punctuation; those were fixed before the main set. All main-set failures remain included. No reruns replace main-set records.
- Small controlled pilot on a synthetic app; not a universal browser/native-app benchmark.
- Assisted arm adds registered structured capabilities and API execution; SDK-alone effect is not isolated.
- No raw-API-only third arm, so gains cannot be attributed uniquely to Smart365 versus direct APIs.
- Model caching, cloud cold starts and network latency can affect timing.
- No billed-cost claim; token totals include cached input once.
- Both arms receive a unique request identifier. Assisted search cards include full input/policy contracts, so inspection is optional rather than mandatory. Both still start with browser_open; no guide-first browser-free scenario is claimed.
- Updated gateway source 60b16e5398a63fcf0e25078bdf5a2e4c91124fea; production deployment dpl_FGKDCX1p1P8Ju9DZvLjJanUMFqBU. Search now returns compact action cards and execution returns independently verified receipts. The prompt no longer forces a redundant inspection; this is a combined workflow update, not an isolated serialization-only test. Historical comparisons do not control every caching/network effect.
- This benchmark does not test the proposed native desktop/iOS adapters, arbitrary external agents or company entitlements. It measures the current web integration, not the entire future platform.
The benchmark is a separate FieldDesk project with fictional records. No benchmark application code was added to the Smart365 repository.