Vision-Language and Task and Motion Planning
Turning natural-language instructions and scene observations into executable robot plans — the ViLaIn interpreters and one-shot vision-language guided motion generation.
Classical task and motion planners are reliable but need a formal problem specification that someone has to write. Language models produce plausible plans but no guarantee they are executable. This theme sits between the two: use vision and language to generate the specification, then let a planner do what it is good at.
Generating the problem, not the plan
The Vision-Language Interpreter (ViLaIn) takes a language instruction and a scene observation and emits a machine-readable problem description in PDDL, which a symbolic planner then solves (Shirai et al., 2024). The division of labour is the point: the model handles grounding and ambiguity, the planner handles correctness and guarantees the plan is logically valid.
The feedback loop is what makes it work in practice. When the generated specification is unsolvable, the planner emits an error message, and that message is fed back to the language model as a re-prompt. Failure becomes a correction signal rather than a dead end — ViLaIn generates syntactically correct problems more than 99% of the time and valid plans more than 58% of the time.
A follow-up grounds the interpreter in geometry as well as semantics, extending it from task planning to integrated task and motion planning so that the plan it produces is feasible for the arm that has to execute it (Beltran-Hernandez et al., 2026).
When there is no symbolic model to plan over
Not every task decomposes cleanly into predicates. KeyMPs generates motion one-shot by having a vision-language model select and sequence dynamic movement primitives, which works in occlusion-rich settings where the scene cannot be fully observed up front (Anarossi et al., 2025) — cutting and similar tasks where the relevant geometry is revealed only as the tool moves through the object.
The main testbed for this line of work is food preparation, where instructions are naturally given in language and the environment refuses to stay static — see cooking robotics.
