ACTUAL CODEX RUNS · INDEPENDENT STATE VERIFICATION
Does Smart365 improve an agent’s work?
In this controlled pilot, Smart365 had 42.0% higher mean token use and 48.1% higher median confirmation time.
2 measured runs · gpt-5.3-codex · 3 business tasks · Complete controlled pilot
Independently verified
1/1 → 1/1
Control → Smart365Mean token use
47,480 → 67,435
Input + output, cached input included onceMedian confirmed time
14.51s → 21.48s
Successful runs onlyCompare the same task
Bars show measured mean tokens per run. They do not measure billing cost.
| Metric | Codex browser only | Codex + Smart365 |
|---|---|---|
| Verified jobs | 1/1 | 1/1 |
| Mean tokens / run (input + output) | 47,480 | 67,435 |
| Mean input tokens | 46,866 | 66,441 |
| Mean cached input tokens (included above) | 39,936 | 49,152 |
| Mean uncached input tokens | 6,930 | 17,289 |
| Mean output tokens | 614 | 994 |
| Median Codex execution (seconds) | 14.21 | 21.22 |
| Median time to independently confirmed state (seconds) | 14.51 | 21.48 |
| Mean model tool calls / run | 5.0 | 6.0 |
| Mean browser / Smart365 MCP calls / run | 5.0 | 6.0 |
| Smart365 provider calls (total, separate from model calls) | 0 | 4 |
| Tool errors | 0 | 0 |
| Timed-out runs | 0 | 0 |
| Confirmed licensed SDK initializations | 0 | 1 |
| Additional SDK chatbot model requests | 0 | 0 |
Every recorded trial
| Task / repeat | Arm | Verified | Tokens | Run + state check | Tool calls |
|---|---|---|---|---|---|
| create-job / 1 | control | Yes | 47,480 | 14.51s | 5 |
| create-job / 1 | assisted | Yes | 67,435 | 21.48s | 6 |
Method, attribution and limits
- 1 repeats × 1 tasks × 2 arms. 2 recorded runs; the planned set is complete.
- Both arms use Codex CLI 0.159.0-alpha.3, gpt-5.3-codex, low reasoning effort, fresh sessions and isolated seeded trial accounts. Arm order alternates by task and repetition.
- Tasks: create a correctly assigned job, reschedule an existing job while preserving other fields, and complete a job with an exact note. Browser fills can batch multiple fields, so the control does not incur artificial one-field-per-tool overhead.
- Success is checked by a separate authenticated database-state read: exact requested fields, all unrelated jobs preserved, expected record count, and exactly one business write. Chat messages claiming success are not evidence.
- Both sites share identical business UI and backend source. The assisted arm loads the licensed deployed SDK, then Codex can discover, inspect and execute registered Smart365 capabilities through the real permission-checked gateway.
- The timed workflow starts with opening the site in both arms. Confirmation time includes Codex runtime, browser/SDK initialization, model reasoning, tool requests and an independent state check. Account reset, consent/key setup and revocation are excluded and retained as separate setupMs values.
- Browser presentation uses local HTTP to avoid changing this workspace’s Chromium TLS trust store. Both use fixed HTTPS proxies to their own hosted Vercel backends; Smart365 provider calls execute remotely. TLS verification remains enabled. This is not a direct remote-browser latency test.
- Tokens are actual CLI turn.completed usage. Total = input + output; cached input is a subset of input and is not added again. Output includes reasoning tokens when the provider includes them. Missing usage remains null, not zero. No billed dollar-cost measurement is available.
- Model tool-call counts include metadata-resource lookups. Smart365’s three provider calls per successful workflow are recorded separately, so fewer model calls do not imply fewer total API calls.
- Permission checks also verified that missing write scope, changed-input replay and revoked agent keys are rejected. Replaying the same operation produces one saved business write.
- Engineering preflights are archived separately. They exposed a scoped tool-approval problem, MCP proxy inheritance, and ambiguous note punctuation; those were fixed before the main set. All main-set failures remain included. No reruns replace main-set records.
- Small controlled pilot on a synthetic app; not a universal browser/native-app benchmark.
- Assisted arm adds registered structured capabilities and API execution; SDK-alone effect is not isolated.
- No raw-API-only third arm, so gains cannot be attributed uniquely to Smart365 versus direct APIs.
- Model caching, cloud cold starts and network latency can affect timing.
- No billed-cost claim; token totals include cached input once.
- This benchmark does not test the proposed native desktop/iOS adapters, arbitrary external agents or company entitlements. It measures the current web integration, not the entire future platform.
The benchmark is a separate FieldDesk project with fictional records. No benchmark application code was added to the Smart365 repository.