From LLMs to Robot Motion: How VLMs, VLAs, World Models, RL, Sim2Real and Control Fit Together

Introduction: the short answer

LLMs, VLMs, VLAs, world models, imitation learning, reinforcement learning, simulation, Sim2Real and robot control are often mentioned in the same robotics announcement. They are not interchangeable names for the same thing, and they do not form a mandatory replacement ladder.

A better mental model is layered. Language and vision models represent instructions and observations. A VLA or another policy turns task context into an action representation. Planning chooses longer-horizon steps. A controller tracks targets under dynamics, contact and safety constraints. Actuators turn those commands into force and motion. Imitation learning, reinforcement learning, simulation and Sim2Real cut across training and transfer. This article follows one task—“pick up the red cup and put it in the box”—to show what each layer contributes.

1. Why the vocabulary is so easy to mix up

The same paper may discuss a model architecture, a training dataset, a learning algorithm and a robot evaluation. A company announcement may compress perception, reasoning, action generation and deployment into one sentence. A demo video may then be read as evidence of production operation. These are different claims.

Use five separate axes:

These axes intersect. A VLA may be initialized from demonstrations, refined with reward optimization, trained partly in simulation and executed through a classical feedback controller. Another system may fold perception and policy into one model and have no explicit world-model module.

2. A useful stack: from instruction to motion

01ModelsLLM · VLM · VLA · World Modeldifferent model families · different roles

02Learning MethodsImitation Learning · Reinforcement Learningmethods for learning behavior / policy

03Training / TransferSimulation · Sim2Realtraining context · calibration · transfer

04Robot SystemPerception · Planning · Policy · Control

05Physical ExecutionJoint · Actuator · Motion

Five interacting layers, shown as a spatial system rather than a universal sequence.

1. 01ModelsLLM · VLM · VLA · World Modeldifferent model families · different roles
2. 02Learning MethodsImitation Learning · Reinforcement Learningmethods for learning behavior / policy
3. 03Training / TransferSimulation · Sim2Realtraining context · calibration · transfer
4. 04Robot SystemPerception · Planning · Policy · Control
5. 05Physical ExecutionJoint · Actuator · Motion

This is a functional map, not a claim that every robot contains these modules as separate services. A model can output action chunks, waypoints, end-effector poses, joint targets or another robot-specific interface. A VLA does not necessarily output motor current.

3. LLMs and VLMs: language is not yet a robot action

3.1 What can an LLM contribute?

An LLM models language token sequences. In a robot system it can parse “put the red cup in the box” into a goal, choose a skill, call a tool or propose a sequence such as find → grasp → move → release. SayCan is a useful example of grounding language-level knowledge with what the robot can feasibly do in its current state, rather than treating text generation as motor control.[SRC-LLM-01]

The limitation is just as important. A text model does not automatically see pixels, joint states, contact forces, control deadlines or actuator limits. Its plausible sentence is not a safe command. Perception, action policies, planning, control and safety checks still have to connect the goal to the hardware.

3.2 What does a VLM add?

A vision-language model aligns language with images or video. It can ground “red cup” and “box” in the current scene, answer whether the cup is present, and describe relative positions. CLIP illustrates a visual-language representation in which text can refer to visual concepts in a shared space.[SRC-VLM-01]

This is useful robot perception and semantic grounding. It is not automatically a full manipulation policy. Picking up a cup also requires geometry, reachability, collision avoidance, timing, force and recovery behavior.

4. VLA: the action interface matters

4.1 What a VLA adds to vision-language understanding

A Vision-Language-Action model conditions on visual observations, language or task context and often robot state, then maps that context to a robot action interface. RT-2 studies mapping robot observations to actions while leveraging web-scale vision-language pretraining. RT-1 uses images and instructions to produce discretized action tokens and evaluates real-robot tasks. OpenVLA combines large-scale vision-language data with robot demonstrations for an adaptable policy.[SRC-VLA-01][SRC-VLA-02][SRC-VLA-03]

4.2 “Action” is not one universal output

Depending on the embodiment and controller interface, an action can be:

• a discrete action token;

• an end-effector pose or pose increment;

• an action chunk over a short horizon;

• joint position or velocity targets;

• gripper, base or robot-specific command vectors.

An instruction such as “move left three centimetres” is a different abstraction from a current command for joint one. Always inspect the output space, frequency and downstream interface before calling a system “direct control.”

4.3 Can a VLA directly control a robot?

Sometimes it can be the immediate high-level action generator. It may emit short-horizon actions that a downstream controller converts into velocities, torques, impedance targets or other actuator commands. That is not the same as owning every real-time motor loop.

The precise answer is architectural: a VLA can participate directly in action generation, but it does not automatically replace feedback control, limits, collision checks or a safety state machine.

5. Why a controller usually remains underneath

Policy and control answer related but different questions:

Control is not a legacy layer that disappears when a foundation model is added. A physical robot must handle friction, loads, compliance, latency, noise, joint limits and safety while closing a feedback loop. Modern Robotics describes control as converting task specifications into forces or torques and using feedback to achieve the desired motion.[SRC-CONTROL-01]

Architectures can become more end-to-end. A model may merge perception, policy and some planning, then output an executable sequence. The boundary changes; the physical execution problem remains.

6. World Models: predicting consequences

A world model asks a different question from a VLA: if the system takes this action from this state, what might happen next? The World Models work presents learning a compressed spatiotemporal representation and rolling out imagined futures in latent space.[SRC-WM-01]

For the cup task, such a model could score whether an approach will knock the cup over, whether a path clips the box, or whether releasing now is likely to leave the cup inside. It can support planning, model-predictive control, policy training or candidate-action evaluation. A prediction is not itself a chosen action, and it is not a perfect replica of real dynamics.

VLA and world models are therefore often complementary. A policy proposes; a predictive model looks ahead. Some systems combine them, some do not expose a separate world-model module, and some embed prediction in another component. Compare the function, not the label.

7. Robot learning: imitation and reinforcement

IL vs RL — Imitation learning starts from demonstrations; reinforcement learning optimizes an objective or reward. They can be combined.

7.1 Imitation Learning

Imitation learning learns from expert demonstrations, teleoperation trajectories or state-action pairs. It gives a policy a behaviour prior, which is valuable when real interaction is expensive, risky or difficult to reward. Behaviour cloning can fit demonstrated actions directly; broader Learning-from-Demonstration methods can learn goals and constraints.[SRC-IL-01]

In the cup task, demonstrations show how a person approaches, closes the gripper, lifts, transports and releases. They contain practical details that are hard to write as rules: approach direction, speed, grasp margin and recovery.

Demonstrations are not magic. They may cover few scenes, inherit operator bias and rarely show every failure mode.

7.2 Reinforcement Learning

Reinforcement learning optimizes a policy using rewards, values and transition data. It can improve success rate, collision avoidance, path length, energy or contact quality, and it can explore behaviours not present in the demonstrations.[SRC-RL-01]

RL does not necessarily mean starting on a physical robot. It can use simulation, offline data, demonstrations or limited hardware interaction. The evidence must state where the reward was optimized and where the final behavior was tested.

7.3 Are imitation learning and RL alternatives?

Usually not. A common recipe is demonstrations first, then reward-based or offline refinement. Another is simulation-based exploration followed by real-world calibration. The useful questions are: what data, what reward, what sequence, and what evaluation—not which fashionable acronym wins.

8. Simulation and Sim2Real

Policy vs Controller — A policy chooses what should happen next; a controller makes the target execute under dynamics and feedback.

Simulation makes it cheaper and safer to copy scenes, generate trajectories, test policies and repeat failures. But it approximates reality. Friction, compliance, sensing, lighting, latency, materials and unmodeled contacts can differ.

Sim2Real is the transfer problem and the techniques used to reduce that gap. It is not a standalone model. Domain randomization, for example, varies visual and physical parameters during training so that a policy is less dependent on one idealized simulation.[SRC-SIM-01] System identification, calibration, adaptation and real-world fine-tuning are other tools.

In a simulated cup task, a policy may learn to approach many positions. Transfer still has to face the real cup’s weight, table friction, camera delay and gripper compliance. A successful simulation run is not proof of deployment.

9. Planning, policy and controller

Simulation vs Sim2Real — Simulation provides a controllable training environment; Sim2Real addresses transfer to hardware.

The boundaries are practical rather than universal. Octo and Diffusion Policy illustrate policy families and action-generation interfaces; neither defines every policy nor eliminates the need to inspect the controller below it.[SRC-POLICY-01][SRC-POLICY-02]

An LLM may help plan, a VLM may help estimate state, a VLA may serve as a policy, and a world model may provide look-ahead. One model can cover several roles, or several modules can share one role. Read the actual inputs, outputs, frequency and feedback path.

10. Worked example: pick up the red cup and put it in the box

01Instruction + Scene“Pick up the red cup
and put it in the box.”

02Representation / Reasoninglanguage · vision · scene state

03Policy / VLAstate → action representationVLA ≠ low-level controller

04Optional Planningmay be separate or folded into policy

05Action Representationtoken · pose · waypoint · chunk

06Low-level Controllerfeedback · safety

07Joint / Actuatorcurrent · torque · velocity

08Physical Actiongrasp · move · place

The architecture varies, but the instruction must cross representation, policy and control before the world changes.

1. 01Instruction + Scene“Pick up the red cup and put it in the box.”
2. 02Representation / Reasoninglanguage · vision · scene state
3. 03Policy / VLAstate → action representationVLA ≠ low-level controller
4. 04Optional Planningmay be separate or folded into policy
5. 05Action Representationtoken · pose · waypoint · chunk
6. 06Low-level Controllerfeedback · safety
7. 07Joint / Actuatorcurrent · torque · velocity
8. 08Physical Actiongrasp · move · place

This is a teaching abstraction, not a required architecture:

• Language: parse the instruction into the goal “the red cup ends inside the box.”

• Visual grounding: locate cup, box, obstacles and relevant spatial relations.

• Action policy: condition on instruction, image and robot state to propose approach and grasp actions.

• Prediction: if available, compare short-horizon consequences of approach and placement choices.

• Planning: decompose the goal and check reachability and collision constraints.

• Control: turn poses or trajectories into joint-level commands and correct error continuously.

• Execution: motors, transmissions, gripper and sensors create motion and contact.

• Feedback: observe slip, contact and final placement, then update or recover.

• Training: demonstrations, reward optimization, simulation and Sim2Real may support different stages.

A real system can combine steps 2–3, omit an explicit world model, or call a skill library instead of generating every action. Adding a foundation model does not remove step 6.

11. How to read a Physical AI system without getting trapped by labels

Ask:

• What state and sensors does the robot actually observe?

• How is the language goal grounded in that state?

• Does the model output semantics, subgoals, action tokens, poses, trajectories or joint commands?

• Which component handles planning, constraints and recovery?

• Which loop controls the actuators, at what frequency?

• Was behavior learned from demonstrations, rewards, simulation or a mixture?

• Is the evidence a research demo, internal test, pilot or sustained commercial operation?

This checklist prevents the most common category errors: a VLA is not a complete robot; a world model is not a VLA; RL is not all of robot learning; Sim2Real is not a model; an LLM does not directly drive motors; a foundation model is not automatically a low-level controller.

One-minute recap

• LLM / VLM: language structure and visual-language grounding, not complete physical execution.

• VLA / Policy: an action-generating mapping from task context and observations; the interface is architecture-specific.

• World Model: a predictor of possible state or environment changes that can support planning and training.

• Imitation Learning / RL: learning from demonstrations versus optimizing an objective or reward; they can be combined.

• Simulation / Sim2Real: a controllable training environment versus the transfer problem between simulation and hardware.

• Planning / Control: longer-horizon goals and trajectories versus feedback execution at the joints and actuators.

12. What remains unsolved

The hard problem is not naming the stack. It is making the stack robust under changing scenes, long-tail failures, contact uncertainty and safety constraints. Open questions include data efficiency across embodiments, reliable recovery over long horizons, sim-to-real mismatch, and auditable ways to translate semantic goals into constrained physical behaviour.

When a paper or company says a robot “learned” a task, ask what the task boundary, success definition, hardware, environment, human intervention and run duration were. Demonstration, internal testing, pilot and commercial deployment are separate evidence categories.

13. FAQ

Is a VLA just an LLM for robots?

It is a useful first analogy, but not a precise definition. A VLA connects visual-language context to a robot action interface; its output and feedback position depend on the architecture.

Does a VLA have to contain a separate VLM?

No. Visual and language conditioning can be jointly trained, with perception, representation and action heads inside one architecture.

Can a world model replace a real robot?

No. It can support prediction, planning or training, but hardware validation is still needed for sensing, contact and dynamics mismatch.

Is RL always better than imitation learning?

No. They solve different optimization and data problems and are often combined. Results depend on task, reward, demonstrations and evaluation conditions.

Is Sim2Real a model?

No. It is a transfer problem and a family of mitigation techniques for moving behaviour from simulation to hardware.

Why keep a classical controller?

Because real joints and actuators require high-frequency feedback, dynamics handling, limits, contact management and safety. Semantic reasoning does not provide those guarantees by itself.

Does a research video prove deployment?

No. It demonstrates a capability under stated conditions. Deployment requires stronger evidence about customer, site, duration, intervention and operating scale.

Where should a beginner start?

Start with the perception–planning–policy–control distinction, then study VLA, world models, imitation, RL and Sim2Real as concepts that cross or support those layers.

Continue with Physical AI, VLA, Robot Learning and Robot Manipulation. See the Humanoid Robot Landscape for company-level examples.