Evaluating Visual Generalization Boundaries in Low-Parameter Vision-Language-Action Models During Rigid Robotic Manipulation

Abstract

Open-source Vision-Language-Action (VLA) models enable rapid physical deployment in robotics, but their visual generalization limitations remain underexplored. This study evaluates the SmolVLA model when confronted with out-of-distribution visual variables. An SO-101 robotic arm, equipped with a vertical parallel pincer, was trained via imitation learning on over 100 teleoperated demonstrations to grasp and displace a baseline 3D-printed black cube, conditioned on the language instruction "pick the cube." To systematically assess generalization limits, we evaluated four categories: baseline, color, scale, and geometry variation. Performance degrades under out-of-distribution shifts, while most evaluated conditions improve after further fine-tuning. The success rates for the red and yellow cubes gradually improved with fine-tuning but remained lower than those observed under the other evaluated out-of-distribution conditions. Initial fine-tuning favored the smaller cube (90% success), but prolonged training reversed this trend, indicating that object scale may impact long-term policy generalization. The failure modes observed in the colour variation condition may reflect a model limitation in translating learned manipulation trajectories to cubes with differing pixel-level visual appearances. These results provide insight into the limited out-of-distribution generalization of the SmolVLA model, highlighting the need for targeted data augmentation to ensure the reliability of assistive robotic systems in real-world environments.