Why robot foundation models need spatial intelligence, not only object recognition

A small sentence in a big announcement

In October, Physical Intelligence opened a beta API for its robot foundation models. Companies can now upload their own robot data, fine-tune a model, and run it on their robots through the cloud. Most people read the post as product news.

I read it differently, because of one sentence. The company listed some problems its partners had faced: a shirt turned inside out, food that would not stay in a scoop, a part that landed in the wrong position. When I read this list, I did not see three robotics problems. I saw three spatial problems: inside versus outside, containment, and relative position. These are the same relations I study every day in vision models. The difference is important. In a vision-language model, a spatial mistake gives a wrong answer on a benchmark. In a robot, it drops the food.

Robots inherit what their vision models know

Many new robot models are not trained from zero. They are built on top of vision-language models (VLMs). For example, Physical Intelligence’s π0 model uses the PaliGemma VLM as its backbone. It then adds an “action expert” that turns the model’s understanding into robot movements. This is a smart design. The robot gets a lot of knowledge about the world for free: names of objects, what they look like, how people describe them. But the robot also inherits what the VLM does not know. And here is the problem. Over the last few years, many studies have shown that VLMs are good at recognizing objects but much weaker at understanding where those objects are. So my main question in this essay is simple:

If vision-language models struggle with spatial relations, what happens when we give them a body?

A simple example: the cup and the box

Imagine you ask a robot: “Put the cup inside the box, to the left of the plate.”

This sounds easy for a person. But for a model, it contains three very different problems:

1. Recognition. Find the cup, the box, and the plate in the image. Modern vision models are very good at this.
2. Spatial understanding. Understand “inside” and “left of”. Know which object is in front, how far things are, and where the free space is. Vision models are much weaker here.
3. Action. Move the arm to the correct place in 3D space, at the correct height and angle. A small spatial error at step 2 becomes a physical failure at step 3.

A model can pass step 1 perfectly and still fail the task. It knows what the cup is. It does not know where the cup should go. This is the core idea of my Spatial Intelligence Notes series: object recognition is not spatial understanding. Robots make this difference very visible.

Why spatial relations are hard for vision models

What research already shows

Several benchmarks have tested VLMs on very basic spatial questions. The What’sUp benchmark (Kamath et al., 2023) used simple photos of everyday objects in different positions: on, under, left of, right of. The authors tested 18 models, and all of them performed poorly. One model that is close to human level on a standard visual question-answering benchmark reached only 56% accuracy here, while humans reached 99%. Other work, such as SpatialVLM (Chen et al., 2024), tried to fix this by training models on large amounts of spatial question-answer data.

The message from this research is clear. These models can name almost everything in an image. But questions like “Is the mug left or right of the laptop?” are still surprisingly difficult.

Why this may happen

We do not fully know the reasons yet. But there are a few likely explanations:

• The training data. Image captions on the internet describe what is in a picture much more often than where it is. People write “a dog in a park”, not “a dog 2 meters behind a bench, slightly to the left”.
• The training objective. Many vision models are trained to match images with text or to classify them. This rewards knowing what is present more than knowing exact positions.
• Where information lives in the model. Spatial information may exist inside the model, in its intermediate layers, but become weaker by the final layer. If the language part of the model only reads the final output, it may never see that information clearly.

What changes when the model has a body

In a VLM, spatial reasoning usually ends in words: “the cup is left of the plate”. A small error gives one wrong answer. In a robot, spatial reasoning must become exact geometry: coordinates, distances, grasp points, and the angle of the gripper. The errors also add up over time. A gripper that is 2 centimeters off on the first move may miss the scoop completely on the third move. This is why robot teams spend so much time on fine-tuning. The π post describes the same loop many teams follow: collect demonstrations, train the model, test it on the robot, and use the failures to decide what data to collect next. This loop works, but it is slow and expensive. And it fixes the task without telling us why the model failed.

Why this matters

When a chatbot makes a spatial mistake, we laugh and try again. When a robot makes one, the cost is real: a broken part, wasted food, a damaged machine, or in some settings a safety risk. Fine-tuning with more demonstrations can fix one task in one place. But it treats the model as a black box. We see that it fails. We collect more data. We hope the next version is better. I think there is a better path. If we understand the spatial weaknesses of the base model before we fine-tune it, we can:

• choose better data, focused on the relations the model finds hard;
• need fewer trial-and-error cycles on the real robot;
• predict which new tasks will be easy and which will be difficult.

To do this, we need tools that measure spatial understanding directly, not only task success.

Where my research fits

This is the question I work on in my PhD: how much do vision models really understand about space, and where in the model is this understanding stored?

Synthetic benchmarks as microscopes

In my project SpaRRTa, I use Unreal Engine 5 to build synthetic 3D scenes for testing spatial perception in vision models. The main advantage of synthetic scenes is control. I can keep everything the same and change only one thing: move an object a little to the left, put it behind another object, or change the camera angle. Then I check if the model notices the change.

Real photos are messy. Many things change at the same time, so it is hard to know what caused a failure. A synthetic scene works more like a microscope. It lets us look at one spatial relation at a time.

What SpaRRTa found

In SpaRRTa, we tested 13 vision models on a simple question: in this scene, is one object to the left, right, front or back of another? We asked it from two viewpoints: the camera’s view (egocentric) and the view of a person standing in the scene (allocentric). I can share the main results here:

• Recognition and spatial understanding are different skills. SpaRRTa scores go together with depth and camera-pose estimation (R² ≈ 0.4), but not with fine-grained classification, such as recognizing aircraft or flower species (R² ≈ 0). Being good at “what” does not make a model good at “where”.
• The image–text model was the weakest. CLIP had the lowest scores in every setting. Even with the best probe, it reached only about 56–68% on the camera-view task, while the strongest models reached about 94–99%. Self-supervised models such as DINOv2, DINOv3 and MAE did much better. This matters because many VLMs use a CLIP-style vision encoder (PaliGemma uses SigLIP, which we did not test directly).
• Spatial information hides in the patches. When we average all patch features into one vector, much of the spatial information disappears. A probe that can attend to individual patches recovers it. For DINOv2 (ViT-L) in the forest scenes, accuracy went from 65% to 94%. Simple evaluation underestimates what these models know.
• Taking another viewpoint is hard for every model. Accuracy was always lower on the allocentric task, where the model must imagine the scene from the person’s point of view. A robot that works next to people needs exactly this skill.
• Clutter makes it harder. Models did best in simple scenes like the desert and the forest, and worse in busy scenes like the city.

For robot models, the lesson is clear: the vision encoder we choose, and the way we read its features, may decide how much spatial information ever reaches the action expert.

Looking inside the model

I am also interested in where spatial information lives inside Vision Transformers. In a related post, I probe the intermediate layers of self-supervised models such as DINOv2, iBOT, and MAE. The idea is simple: instead of only testing the final output, we test each layer and ask how much spatial structure it contains. SpaRRTa supports this idea. When we probed each layer of DINOv2 and VGGT (ViT-L, 24 layers), the best spatial accuracy came from late-intermediate layers, around layers 18 to 20, not from the final layer. For robots, this matters: a model that only reads the final layer may be throwing away useful spatial information. These results lead to one idea I believe strongly:

Before we give a vision model a body, we should know where it stores spatial information, and where it loses it.

What I do not know yet

I want to be careful here, because it is easy to overclaim. First, the π post does not measure spatial ability. It only mentions some problems its partners faced. These problems look spatial to me, but I cannot prove that spatial understanding was the main cause. Second, robot failures have many possible causes: the hardware, the gripper, the control system, timing, or network delays. The π post itself mentions network problems for some cloud-based deployments. Spatial understanding is only one part of the story. Third, results on VLM benchmarks do not automatically transfer to robots. Robot models are fine-tuned on real demonstrations, and this may fix some spatial weaknesses. In the same way, SpaRRTa tests frozen vision encoders, not full robot models. It shows what spatial information the encoder can offer, not what the robot finally does with it.

So instead of conclusions, I have questions:

• Do robot foundation models inherit the spatial weaknesses of their VLM backbones? Or does fine-tuning on robot data remove them?
• Can controlled synthetic benchmarks predict how a robot will perform in the real world?
• Should we probe robot models layer by layer, the same way we probe Vision Transformers?

I think these are good questions for the next few years of research.

From seeing to acting

For a long time, spatial intelligence in vision models was one benchmark among many. A model could score low on spatial questions and still be very useful for captioning, search, or chat. Robots change this. When a model must act in the physical world, the question “does the model know where things are?” is no longer academic. It decides if the food stays in the scoop. As foundation models move from seeing to acting, I think spatial understanding will become one of the most practical questions in AI. And I hope that controlled benchmarks and careful probing can help us answer it before the robot drops the cup.

Recognition tells a robot what to grab. Spatial understanding tells it how.

This is part of Spatial Intelligence Notes, a series about how vision models understand, and fail to understand, space. If you work on robot learning or VLM evaluation, I would be happy to hear your view. You can find more of my work on my website.

<hr><p>From Seeing to Acting was originally published in Data Science Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>