Cooking Robotics
Cooking as a benchmark for contact-rich, long-horizon manipulation — safe knife skills learned in simulation, and task and motion planning that gets a whole dish from instruction to plate.
Cooking is an unusually demanding benchmark. The tools are dangerous, the materials are deformable and vary from one cucumber to the next, the tasks are long-horizon — slice, then fixture, then plate — and the instructions arrive in natural language. A kitchen exercises nearly every open problem in manipulation at once, which is exactly what makes it useful.
Two threads run through this work: learning the contact-rich skills safely, and planning the sequence that strings them together.
Learning to cut without wasting food or breaking things
Slicing is a force problem before it is a motion problem. The knife has to respond to the reaction forces from both the food and the cutting board, and those forces differ for every product. Reinforcement learning is a natural fit, except that training it on real hardware means a robot swinging a kitchen knife through random exploratory actions, and a great deal of destroyed food.
SliceIt! avoids that with a real2sim2real loop (Beltran-Hernandez et al., 2024). A small amount of real slicing data calibrates a high-fidelity soft-material cutting simulator (DiSECt); that simulator is then paired with a robot simulator (Gazebo) so the two run together — the cutting simulator supplies realistic contact forces, the robot simulator supplies knife motion that an actual arm can produce. The compliant control policy is learned entirely inside that dual environment and only then deployed to the real robot.
The pairing matters because neither simulator is sufficient alone. Cutting simulators model the knife-food interaction in detail but have no notion of whether a robot could physically realize the motion; robot simulators have the opposite blind spot. Running them together closes it. The framework was later featured by IEEE Spectrum.
Planning a whole dish
A single skill is not a meal. Preparing even a simple cucumber salad means sequencing pick, place, tool equipping, fixturing, slicing, and serving, where the stages are interdependent — how one arm fixtures the cucumber determines whether the other arm can reach it with the knife. Off-the-shelf motion planners are largely restricted to single-arm pick-and-place and treat each stage as a black box, which makes those dependencies invisible.
The framework integrates PDDLStream, a sampling-based task and motion planning solver, with the MoveIt Task Constructor, a multi-stage manipulation planner, using MTC as the motion sampler inside PDDLStream (Beltran-Hernandez et al., 2024) (Beltran-Hernandez et al., 2025). The task planner works over a cooking domain with predicates such as isWhole, isSliced and Registered, and actions including fixture, check-extremity, slice and serve-slices.
Two ideas make this workable in a real kitchen rather than only in simulation. Optimistic planning commits to a full plan using an initial belief about object poses, rather than waiting for certainty that a dynamic environment will never provide. Replanning from perception then repairs it: before any state-changing action, the robot registers the object’s actual pose and revises. This also handles effects the planner cannot know ahead of time — slicing changes the number of objects in the scene, and how many slices there will be is not something you can write into a precondition.
The system is a dual-arm setup, and the cooking-specific skills it needed — fixturing an object so it does not roll, force-based detection of where the cucumber actually ends, and the RL slicing policy from SliceIt! — are the parts a generic planner does not supply.
From instruction to plate
These two threads are the second half of a larger goal: a robot that takes a spoken instruction and cooks. The first half — turning language and a scene observation into the formal problem description that the planner above consumes — is the vision-language interpreter work. The compliance controllers underneath the knife come from active compliance control.
This work is supported by JST Moonshot R&D Grant JPMJMS2236 and JSPS KAKENHI Grant 21H04910.
