I'm brushing up on my robotics basics i.e. object recognition, trajectory planning and grasp. I am using Claude to help me with it.
Today I was doing a simple task of producing a cube in a random area. The bot is supposed to identify the object through an image using OpenCV then grip it and place it at a designated spot. Simple. Absolutely basic program.
There are a few interesting things that happened not in my simulation but with Claude. 3 times in one day, it imagined problems, confidently suggested wrong solutions and even ignored when I told it the problem and solutions. As the world prepares for physical AI, it’s interesting to see how brutally things are failing in a 5cm cube simulation. And I am curious how we should prepare.
First instance: I had to prove to an LLM what I was seeing
I was testing how a pinhole camera placed above a Franka arm will see a scene. I ran the code and it was working. It gave me correct images which I shared with Claude only to mark the step complete.
Left: red cube dead-center. Right: same camera, cube moved 10cm in +y.
The cube is at exactly the places it should be however Claude said both the images are wrong. See it’s response:
Claude’s response: confident, detailed, and built around three sub-bugs that don’t exist.
Notice how it saw the images, found bugs and not only bugs, it found three sub-bugs. It gave details.
I had to draw a diagonal crosshair on each image and show Claude that the cube was at the center in one and exactly 10cm above center in the other.
The hand-drawn crosses showing the cube exactly where it’s supposed to be — dead-center in one, ~35 pixels above center in the other.
And then it finally acknowledged the issue.
Claude’s eventual concession: “I misread the image badly… the rendering is working perfectly.” Only after seeing what I had drawn.
It acknowledged but I had to make a rule. Draw a crosshair programmatically in every image. Because, I did not trust Claude’s eyes anymore.
Second instance: The fix it inverted
This time, the task was to generate a cube at 7 random places and identify where the cube is and where its centroid is.
Out of the seven times, two times the simulation failed. The other times it worked.
I gave Claude the results and the images and asked its opinion on what might be going wrong. Claude again gave me an imaginary bug — that the two positions were occluded by the arm.
Claude’s proposed fix: move the arm out of the way before detection. A workaround for an imagined bug, not the real one.
Look at the images. The cube under the shadow of the gripper has a darker red, and when it is out of the shadow it is light red — almost pink.
Now look at the centroids, the green crosshairs, drawn on the cubes. This clearly validates that the program is only identifying the dark red as red and ignoring the light red.
However, this time too, Claude was certain that the arm was occluding the view, and even proposed a change instead of identifying the error properly.
Third instance: The drop that wouldn't trigger
This time my program was doing the pickup and placing the cube at designated spots. Out of 10 attempts, it failed in 6 attempts in placing the cube. I was watching the simulation happening and I could clearly see that the trigger condition for bringing the arm down to place the cube and releasing it was not meeting.
Unfortunately, I didn’t record the video of this simulation otherwise I would have shared that too.
In this case again it went ahead and started modifying the whole simulation, the physics variables and all of that. All it needed to do was make the trigger policy for coming down and placing a little relaxed.
What I told Claude I was seeing: the gripper carries the cube to the place but never releases it.
After this message, it gave me a bunch of solves. And I had to…
My correction: the problem isn’t in the XML — the descend-to-place trigger is failing. Loosen it. Claude kept proposing physics tweaks instead.
It was obvious to me but not to Claude!
It kept ignoring my suggestion.
When things worked and it was summarizing, Claude was nice enough to say this:
Claude’s eventual summary: it had “generated three or four wrong hypotheses” and dismissed my correct one as “covering the symptom.”
What does it mean?
While all of this was happening, Claude was being confidently, analytically, eloquently WRONG! If I didn’t know any better, I’d think it was made by a straight, white man in modern USA.
Jokes and vision abilities aside, there are a few things to be observed here and what they mean for these models being plugged into physical robotics systems.
Same input. Different perceived input. No way to tell except human experts.
This is the challenge for AI now. Especially for physical AI. The input in all these cases was the same for both Claude and me. We were looking at the exact same images. Eyeballing an image and trusting your eyes over an LLM's analysis is not a normal first instinct. My mechanical engineering background definitely helped here. If I hadn’t spotted it as a wrong perceived input for Claude, we would have spent hours fixing an imaginary error.
And remember, this was the simplest simulation! The most frontier AI model was failing at that. What’s going to happen for real when we move the responsibility of the models from bits to atoms?
Disregarding human feedback?!
This shows us that not only can AI be wrong, it can be wrong confidently enough that it ignores human feedback. This was a simulation and the context of the conversation was learning — but it becomes increasingly important in settings where minor changes can cause major repercussions.
My thoughts on steps moving forward
There is a lot of interesting work happening for physical AI or bringing AI to robots. We need to remember:
- LLMs are NOT ready yet. They’re far from it. The solutions need to be much better multimodal systems trained on better, real-world data.
- The RLHF phase itself will require a lot of human presence, and ensuring safety and scale is a large responsibility in itself.
3. Edges, again
I am cautious about AI even when it comes to my software interactions. I’ve been building my personal automation system and even in that, I make sure that the edges are not AI but plain software. This tiny simulation strengthens my belief that the edges of the systems still need to be deterministic.
In the physical world, deterministic edges would look like a warehouse controlled completely by robots, or specs being decided by humans and the construction being done by robots, and for personal robots, the actuation system having a controlled decoupling from the AI core of it.
We’re not ready to hand the edges to AI in software yet. We are nowhere near ready to hand them to AI in atoms.
PS: Kindly ignore the fact that I treat my Claude like a human and frustratingly call it Bubba sometimes.
— Ankur Goel
I am brewing a few experiments. I might do some interesting things soon. You can reach me at:
You can check out other blogs I wrote at:
Pi0.7: utilizing bad data to teach good things
Building a personal automation system on the graveyard of past attempts