Robot Scaffolding with Vision Language Models

Utku Ayaz Akkas1, Jonas Kiemel2, Emre Ugur3, Minoru Asada4,5, Erhan Oztop1,4

1Ozyegin University  ·  2Karlsruhe Institute of Technology  ·  3Bogazici University  ·  4University of Osaka  ·  5IPUT Osaka

IEEE International Conference on Development and Learning (ICDL) 2026

Paper Code
VLM Scaffolding Framework

In the proposed framework, instead of selecting actions randomly, the robot receives action suggestions from a Vision–Language Model (VLM) that observes the current environment image. The robot executes the suggested action, altering the environment, and the interaction loop continues.

Abstract

Accounts of developmental learning in psychology posit a central role for scaffolding in infant sensorimotor development, where caregivers structure interactions with the environment to promote informative experiences. Inspired by these principles, developmental robotics has adopted scaffolding mechanisms that enable robots to acquire complex skills. In most robotic implementations, scaffolding is provided by human teachers or designers.

In this work, we investigate whether a Vision–Language Model (VLM) can serve as an effective scaffolding agent for robot learning and exploration. To investigate this, we use an object manipulation setup in which a robotic manipulator with pick-and-place capabilities selects a source object and target location. The VLM is then asked to guide the action selection towards interesting outcomes given the image of the environment for boosting exploration.

The simulation experiments show that VLM-guided learning yields faster discovery of hard-to-reach states. Furthermore, our analysis of the model's reasoning traces shows that the model develops affordance-aware strategies that adapt to the environment. In sum, our study indicates that off-the-shelf VLMs can serve as scaffolding agents for improving robot learning.

Video

Method

We implement a tabletop pick-and-place task in PyBullet simulation where a UR10e robotic arm manipulates cubes and spheres. At each step, the VLM (Gemini 2.5 Flash) receives a 1024×1024 RGB image of the current scene along with a text prompt asking it to suggest an action that creates an interesting outcome. The robot executes the suggested pick-and-place action, the environment transitions to a new state, and the updated image is fed back to the VLM—forming a closed interaction loop.

To maintain temporal consistency, the full conversation history (previous images and action decisions) is preserved in a multi-turn dialogue. This allows the VLM to consider the consequences of its prior actions when generating subsequent suggestions. Each episode consists of 10 sequential action steps, after which the environment resets with a new random object configuration. We evaluate across four environments of progressively increasing complexity, varying the number of object types (cubes only vs. cubes + sphere) and available placement locations (2 vs. 5).

Tabletop environment
Tabletop environment
Example visual input 1
Example visual input
Example visual input 2
Example visual input

BibTeX

@inproceedings{akkas2026vlmscaffolding, title = {Robot Scaffolding with Vision Language Models}, author = {Akkas, Utku Ayaz and Kiemel, Jonas and Ugur, Emre and Asada, Minoru and Oztop, Erhan}, booktitle = {IEEE International Conference on Development and Learning (ICDL)}, year = {2026}, url = {https://akkasayaz.github.io/papers/vlm-scaffolding} }