A tool call contains braces, field names and constrained values. Generating it as if every next token were unrestricted prose can waste the structure already known to the system.
SchemaWeaver's October 12 paper uses schema-guided parallel decoding for on-device tool calling, without retraining the language model. Its authors report latency reductions across evaluated mobile processors and model families. Read the research paper.
Tool calls are not ordinary sentences
An on-device agent may spend substantial time generating a structured request before any external tool begins its work. Mobile processors make this problem difficult because decoding is often limited by memory bandwidth rather than the amount of arithmetic available. Simply adding compute does not necessarily make token-by-token generation proportionally faster. Yet a tool call contains information that free-form prose does not: its schema already constrains much of what can appear next.
SchemaWeaver treats that application knowledge as a decoding resource. The schema becomes useful during generation, rather than only after generation when a validator checks whether the result is well formed.
Known structure creates a parallel opportunity
The system enforces schema-guided parallel decoding to produce multiple tokens in a decode pass. It does not require retraining or modifying the language model. On a phone, the model may be fixed; adapting the inference engine can be more feasible than creating another checkpoint for every tool interface.
The premise is not that every argument value can be guessed in advance. It is that constrained positions provide exploitable structure. A tool request contains fixed syntax and relationships among fields, so treating each position as unrestricted next-token text can spend bandwidth on uncertainty the application has already removed.
A speed claim needs the whole interaction
Across the mobile systems-on-chip and language-model families tested in the paper, the authors report reductions in tool-calling latency of up to 3.9 times against competitive baselines. That is a result for the evaluated decoding workloads. It does not imply that an entire agent session becomes 3.9 times faster: external tool execution, network access, validation and retries can dominate other parts of the interaction.
The distinction is practical. A local decoding gain may be highly valuable for quick, repeated actions but barely visible when a remote service takes several seconds. Profiling needs to identify how much of the user-visible delay actually lies in generating the tool request.
The tokenizer affects the supposedly fixed skeleton
SchemaWeaver constructs a tool skeleton and decodes argument values in parallel. The full paper notes a subtle implementation issue: a tokenizer may fuse a prefix space with the first character of a value, such as a minus sign. Splitting that fused token between skeleton and value can cause incorrect output, so the skeleton must respect the model’s tokenisation.
The parallel value sequences also end at type-specific boundaries, and padding unfinished sequences can improve mobile batching efficiency. On the reported BFCL workload, mean decode passes fall from 13.1 to 6.8. These details explain why exploiting syntax is an inference-engine intervention rather than a prompt trick: the engine must coordinate token boundaries, multiple argument streams and the write-back of the completed request.
Structural correctness leaves a semantic question
Producing a valid request does not establish that the agent selected the right tool or filled a field with the correct value. A schema can constrain a date format without proving that the date matches the user's intent. Tests should therefore pair latency with execution success, invalid-request rates and the cost of any required repair.
Schema complexity also affects the opportunity. Fixed enumerations and predictable structure offer different leverage from large free-text arguments. The application already knows more about a tool request’s structure than an unrestricted text generator assumes. Using that knowledge can improve serving efficiency, provided the evaluation preserves the distinction between generating a request quickly and choosing a request correctly.
Questions, news tips or corrections? Contact [email protected].
