What Is a Robot Foundation Model? From AI Models to Physical Intelligence

From Foundation Models to Physical Intelligence

Foundation models are trained on broad data and adapted to many downstream uses. In robotics, the idea becomes more demanding: a model must connect observations, instructions, actions, and feedback in a physical environment rather than only produce a digital response.

The foundation-model idea is attractive in robotics for the same reason it became important elsewhere in AI: instead of training an entirely separate model for every narrow task, researchers want reusable models that can transfer knowledge across tasks and be adapted with less task-specific engineering. But robotics adds a constraint that text and image generation do not have—the model’s outputs eventually have to affect a physical system.

Moving from digital AI to physical intelligence is therefore not a matter of putting a language model inside a robot. Robot systems have to connect perception with an embodied state, map intentions into actions that a particular machine can execute, observe the consequences of those actions, and continue operating when the real world differs from the training data. Our overview of Physical AI provides the wider context for that transition.

Physical intelligence requires model capabilities to be grounded in perception, action, and contact with the world.

Why ChatGPT Alone Is Not a Robot Controller

A general-purpose chatbot can interpret instructions, describe a scene, or contribute to high-level planning, but that does not make it a complete robot controller. Physical control is closed-loop: a system has to estimate what is happening from sensors, account for the robot’s own configuration and physical contact, issue actions at a useful rate, observe their consequences, and correct errors as the environment changes.

Before a robot acts, it must understand the objects, depth, materials, and spatial relationships that define a physical scene.

Research systems illustrate this difference. RT-1 takes robot camera observations and task instructions and produces robot actions with an architecture designed for real-time control. PaLM-E explores another part of the problem by incorporating visual observations and continuous robot state into an embodied multimodal model. These systems differ in architecture, but both show why physical intelligence requires more than text generation alone.

A robot may still use a language model for instruction interpretation, semantic reasoning, or planning while relying on perception models, policies, motion planners, controllers, and safety mechanisms for execution. The important distinction is not that language models are irrelevant to robotics; it is that fluent language output is not, by itself, evidence of reliable closed-loop physical control.

What Is a Robot Foundation Model?

There is not yet one settled architecture—or even one perfectly uniform use of the term—behind “robot foundation model.” Current research systems use different model interfaces, training mixtures, action representations, and target embodiments. In this article, we use the term for a broadly pretrained or jointly trained model, or model system, intended to provide reusable capabilities across multiple robot tasks, environments, or embodiments through adaptation or direct policy execution.

This definition preserves the central idea of a foundation model—reuse across downstream settings—while adding the requirements of embodiment and action. Depending on the system, the reusable capability may be a multimodal representation, an action policy, a planning component, an action-generating expert, or a combination of several components.

A robot foundation model is therefore not the robot hardware itself, not a single low-level controller, and not a product category. Nor does the label imply that one model performs every layer of the robotics stack. Real systems can still combine a broadly trained model with task-specific control, state estimation, motion planning, safety limits, and hardware-specific interfaces.

The Robot Data Challenge

Internet-scale data is abundant, but robot data is expensive, embodied, and tied to sensors, actuators, objects, operators, and environments. Demonstrations can be slow to collect, failures can damage equipment, and the same action may have different consequences on different robots.

Learning continues through contact: the robot adjusts its motion as the world responds.

Robot data also carries a different structure from ordinary web data. A robot trajectory connects what the system observed with what it did next. Depending on the platform, that can include camera observations, robot state, task instructions, commanded actions, and repeated observations as the task unfolds. The data therefore records not only what the world looked like, but how a particular body changed it.

That last point makes cross-robot reuse difficult. The same high-level instruction—such as moving an object to the left—can correspond to different joint motions, gripper commands, coordinate systems, sensing layouts, and physical dynamics on different machines. Scaling robot data is therefore not only a question of collecting more trajectories; researchers also have to decide how observations and actions from different embodiments can be represented and aligned.

The Open X-Embodiment project illustrates why data diversity matters: shared datasets and models can expose transfer opportunities across robot types, but they do not remove the need to align action spaces, embodiments, and evaluation conditions. The Robot Learning article covers the capability-acquisition side of this problem in more detail.

Vision-Language-Action Models

Vision-Language-Action (VLA) models are one important approach within robot foundation model research. They connect visual observations and language instructions to action outputs, often using large pretrained components together with robot trajectories or demonstrations.

Several influential systems show how this direction has evolved. RT-1 demonstrated that a Transformer policy could be trained across a large set of real-world robot tasks while mapping camera observations and language instructions to actions. RT-2 went further by co-fine-tuning vision-language models with robot trajectories and explicitly described this class of systems as Vision-Language-Action models.

Other systems explore different parts of the same broader problem. PaLM-E injects visual and continuous state information into an embodied multimodal language model rather than presenting itself as a complete low-level policy. Open X-Embodiment and RT-X investigate whether shared datasets and models can transfer experience across different robot embodiments. More recent systems such as π0 and NVIDIA GR00T N1 represent additional approaches to generalist robot control and VLA-style models, with different architectures, datasets, target hardware, and evaluation settings.

These systems should not be treated as implementations of one settled architecture. VLA describes an important model interface and family of approaches; “robot foundation model” is the broader role discussed in this article. A robot foundation model may include a VLA policy, multimodal representations, action-generating components, or other reusable modules. The VLA explainer focuses on the narrower VLA family.

How Robot Foundation Models May Influence Robotics

Researchers are exploring whether reusable representations, robot datasets, and policy components can reduce some task-specific engineering across related tasks or embodiments. Natural-language interfaces are another possible benefit when language is grounded in perception and connected to an execution system that can safely act on the resulting instruction.

The practical value of these approaches remains highly dependent on the task, embodiment, data mixture, latency requirements, and evaluation protocol. A model that transfers between laboratory tasks may still require substantial adaptation, monitoring, and task-specific safeguards before it can operate reliably in a production environment.

Current Challenges

Important constraints remain. Generalization is still difficult because a robot can encounter objects, viewpoints, lighting, scene layouts, instructions, and physical conditions that differ from its training distribution. Performance on familiar laboratory tasks does not by itself establish that the same policy will remain reliable under those changes.

Embodiment diversity creates another problem. Different robots can have different numbers of joints, grippers, camera layouts, control frequencies, action spaces, payload limits, and dynamics. Cross-embodiment datasets such as Open X-Embodiment are important precisely because transferring experience between these systems requires more than placing their raw trajectories in one dataset.

Reliability and safety raise a different standard from digital benchmarks. A wrong text answer can be corrected after it is produced; a wrong physical action may drop an object, collide with equipment, or create unsafe motion. Deployment therefore depends on monitoring, intervention mechanisms, hardware limits, recovery behavior, and repeated evidence from the real operating environment—not only on model-scale benchmarks.

These challenges connect robot foundation models to Embodied AI, Sim2Real, and robot manipulation. They are not solved simply by increasing parameter count or adding a language interface.

Conclusion

Robot foundation models extend the foundation-model idea into a setting where data, action, embodiment, and feedback are inseparable. The field does not yet converge on one architecture: VLA policies, embodied multimodal models, cross-embodiment training, and other generalist approaches explore different ways to make robot capabilities more reusable. Their value should ultimately be judged by transfer, robustness, reliability, and deployment evidence rather than by model scale or language fluency alone.