A robot does not need its next answer merely to arrive quickly. It needs that answer to remain useful when it arrives.
In Beyond LLM Serving, posted October 4, KAIST researchers characterise four vision-language-action models across an edge GPU server and two onboard systems. Their analysis includes 43,200 simulated closed-loop episodes. They find that overlapping inference with action execution creates trade-offs among accuracy, speed and energy; no configuration dominates across all deployment requirements.
The study also finds that different model stages hit different hardware bottlenecks. A single-robot, batch-one control loop does not automatically benefit from the same optimisations that make a busy language-model server efficient.
Motion keeps changing the question
The complication is easiest to see with action chunks. A policy produces several future commands, and the robot begins executing them. Meanwhile, another inference can run. This overlap avoids some idle time, but the next computation begins with an observation taken before all the intervening motion has occurred.
If nothing surprising happens, the arrangement can work well. If an object slips, the robot may continue along a plan based on a grip it no longer has. Short model latency helps, but the scheduling policy determines how that latency translates into stale information.
That suggests a metric engineers should log explicitly: observation age at execution. Timestamp the image and the action, then measure the gap. The distribution matters. A smooth average can conceal occasional old observations at precisely the moments when contact changes.
Energy per useful completion
Power introduces another subtlety. Lowering a processor’s frequency may save energy during some stages and simply prolong others. If the slower setting causes more retries, an attractive per-inference energy figure may fail to translate into lower energy per completed task.
A deployment experiment should therefore include idle time, retries and aborted work alongside compute measurements. These quantities connect the accelerator to the physical job. They also make it possible to compare a policy that acts cautiously with one that issues commands quickly but wastes motion.
The same logic explains why a universal “best” serving configuration is unlikely. A task with long, predictable motions can tolerate a different schedule from a delicate insertion with rapidly changing contact. A fleet sharing compute presents another workload again; the paper’s single-robot assumptions should not be silently extended to it.
For a lab moving a policy onto hardware, the immediate action is modest: profile the stages, then test scheduling choices in the loop. Do not select a configuration from inference time alone. A fast computer can still deliver yesterday’s picture of a world that moved a fraction of a second ago.
References
Questions, news tips or corrections? Contact [email protected].
