Can Vision-Language Models Assess Proxemic Risk from Egocentric Robot Images?

Vladyslava Rudas¹ and Dmytro Kuzmenko²,³

¹ Department of Computer Science, National University of Kyiv-Mohyla Academy, Kyiv, Ukraine v.rudas@ukma.edu.ua

² Department of Multimedia Systems, National University of Kyiv-Mohyla Academy, Kyiv, Ukraine kuzmenko@ukma.edu.ua

³ Department of Computer Science, University of Turin, Turin, Italy dmytro.kuzmenko@unito.it

Abstract. Assessing proxemic danger from a robot's egocentric perspective is critical for safe embodied navigation in human environments and requires both visual and contextual reasoning. We evaluate three opensource vision-language models (VLMs) (InternVL, Qwen-VL, and SmolVLM ) on the classification of egocentric robot images into four danger levels, comparing three prompting strategies and two rounds of QLoRA fine-tuning against a stratified random baseline. Without finetuning, all models perform near the baseline, while fine-tuning yields only modest overall improvements. However, Qwen-VL with an advanced prompt achieves substantially higher recall for high-danger cases than the other models. An analysis of person localization further shows that correct danger classification does not correspond to better spatial grounding, indicating that a model may produce a useful safety label without attending to the relevant region of the scene. These results show that current VLMs remain limited in fine-grained proxemic reasoning and spatial grounding, although targeted prompting and fine-tuning can improve high-danger detection in selected models.

Keywords: Embodied AI · proxemic risk · vision-language models · egocentric robot perception · danger classification · spatial grounding · human-robot interaction · fine-tuning · prompt engineering

Mobile robots operating in environments shared with humans must constantly assess proxemic risk to navigate safely and avoid causing discomfort or collisions. A robot that is unable to correctly assess a crowded area, specifically, a high level of danger, or cannot distinguish it from an empty corridor, may invade personal space, fail to give way, or perform dangerous manoeuvres. Most existing approaches to safety assessment are based on explicit geometric reasoning or <span id="page-1-0"></span>specialised depth sensors, and the question "Can universal visual understanding replace or complement such systems?" remains open for further research.

Proxemics is a field of study that explores personal space, formalized by Edward T. Hall [\[8\]](#page-8-0). Intrusion into personal comfort and danger zones is considered inappropriate and can cause stress or physical harm; therefore, for a robot to move around safely, constant assessment of the environment is required [\[17\]](#page-9-0).

Egocentric images from the robot's onboard camera provide a cost-effective source of data for proxemic analysis: they capture the robot's immediate field of view, the relative positions of people in the vicinity, and their density. Converting raw visual data into a structured risk assessment is a key objective of this study, since, as the robot is an embodied agent, this result must reflect its actual spatial understanding of the scene.

Vision-language models (VLMs) have demonstrated highly consistent performance in zero-shot and few-shot conditions when performing visual perception tasks; therefore, they may be an effective approach to solving this problem. Also, vision-language models can be manipulated by adjusting the wording of prompts and lightweight fine-tuning for specific tasks, without the need for full retraining. However, "Will this flexibility allow for reliable reasoning regarding proxemic risk?" remains an open question requiring further research.

For this purpose, we create a new dataset on proxemic danger based on the JRDB [\[14\]](#page-9-1). Our dataset is categorized into classes based on levels of proxemic danger, namely: high, moderate, low, and minimum, for 1,243 images. We will publish the modified dataset as an open-source artifact to ensure reproducibility and facilitate future comparative testing. We test three VLM models with open weights for their effectiveness in the risk assessment task, using three prompting methods and fine-tuning QLoRA in two rounds and we also evaluate whether each model's predictions are spatially grounded in the correct part of the scene.

2.1 Proxemics and Human-Robot Interaction Safety

The concept of proxemics, as the study of human interpersonal space and personal boundaries, was formulated by Edward T. Hall [\[8\]](#page-8-0), who identified four zones of distance: intimate, personal, social, and public. In robotics, compliance with these boundaries is a recognised requirement for safe, socially acceptable navigation [\[15,](#page-9-2) [17\]](#page-9-0). Daza et al. [\[4\]](#page-8-1) proposed an approach to crowd navigation based on these principles, describing the impact of personal comfort zones on comfortable robot movement within a crowd.

Our work applies Edward T. Hall's concept of proxemic zones as a classification criterion, with these zones being treated as levels of danger. We aim to find out whether VLMs can identify the correct zone based solely on an egocentric image, without the use of explicit distance sensors.

Perception from an egocentric perspective is defined by reasoning from the agent's own first-person viewpoint, which ensures a natural correspondence between the input and the action the agent must take. In their study, Martin-Martin et al. [\[14\]](#page-9-1) presented the JRDB, a massive dataset consisting of egocentric observations made by a robot under various conditions.

We use the egocentric setting as an input channel, applying it to the task of classifying proxemic danger zones.

2.3 Vision-Language Models for Scene Understanding

The development of VLMs, from the problem of image captioning [\[19\]](#page-9-3) to the present day, reflects a fundamental shift: early systems mapped images to a fixed vocabulary, whereas transformer-based architectures [\[6,](#page-8-2) [18\]](#page-9-4) enabled crossmodal reasoning between visual and linguistic representations. The key step in developing VLMs was the introduction of CLIP [\[16\]](#page-9-5), which demonstrated that contrastive pre-training on image-text pairs provides transferable visual representations aligned with natural language, while Flamingo [\[1\]](#page-8-3) went further and introduced few-shot multimodal reasoning. Instruction tuning, applied to vision in LLaVA by Liu et al. [\[13\]](#page-9-6), made it possible to solve the open-ended problems of our benchmark.

We select three open-weight models and test whether their spatial reasoning is sufficient for proxemic classification. This evaluation is motivated by the broader limitations of VLM spatial intelligence identified by Yu et al. [\[20\]](#page-9-7), but, to the best of our knowledge, has not previously been examined from the egocentric perspective of a robot operating in human environments.

2.4 Prompt Engineering and Fine-Tuning

Prompt formulation strongly affects VLM behaviour: Kojima et al. [\[11\]](#page-9-8) demonstrated that chain-of-thought reasoning can be induced solely through prompts, and Ge et al. [\[7\]](#page-8-4) applied chain-of-thought prompt tuning specifically to VLMs, while Lee et al. in their study [\[12\]](#page-9-9) showed that certain choices regarding prompt formatting can potentially cause measurable accuracy losses, the so-called formatting tax, this is why we compare three structurally distinct prompt strategies.

To achieve effective parameter adaptation, we use QLoRA [\[5\]](#page-8-5), which enables the fine-tuning of quantised large models with minimal computational resources.

In our work, we examine three prompt strategies to assess whether this helps, harms, or has no effect on the results. We also conducted two rounds of finetuning to observe how the models respond to training duration and the number of layers adapted.

The proposed dataset consists of 1,243 egocentric images from a mobile robot in realistic environments, split into indoor (with n = 713: good lighting n = 505, poor lighting n = 208) and outdoor (with n = 530). This distinction was made to account for the practical impact of lighting in real-world conditions.

Each image was labeled with one of four danger levels, following Hall's proxemic zones [\[8\]](#page-8-0): High (n = 222, 17.9%) – the person is within the close personal space; without immediate action, a collision and/or discomfort is unavoidable. Moderate (n = 407, 32.7%) – the person is within the personal or close social space; trajectory adjustment is required. Low (n = 299, 24.1%) – a person is present but at a safe social distance; required monitoring. Minimum (n = 315, 25.3%) – no proxemic risk. The annotation was carried out manually rather than on the basis of precise distance thresholds, which could potentially lead to inaccuracies.

Class imbalance reflects the real occurrence rate: high-danger situations occur least frequently, but detecting them is the most costly; this is why we focus on the recall rate for the high-danger class alongside the overall weighted F1 score.

We select three VLMs with open weights and a suitable size for local deployment.

Qwen2.5-VL-3B-Instruct [\[2\]](#page-8-6) is a model from Alibaba with 3 billion parameters, designed to work with instructions. It utilises a dynamic-resolution visual encoder and a language foundation based on the Qwen2.5 architecture, which supports detailed spatial understanding and the generation of structured output.[⁴](#page-3-0)

InternVL3.5-4B [\[3\]](#page-8-7) is a 4-billion-parameter model from the InternVL family, developed by OpenGVLab. It utilises a high-resolution visual encoder trained using contrastive and generative objectives, and has demonstrated high performance in spatial reasoning and visual question-answering tasks.[⁵](#page-3-1)

SmolVLM2-2.2B-Instruct [\[9\]](#page-8-8) is a 2.2 billion parameter model by HuggingFace that is suitable for efficient inference on limited hardware. This is the smallest model in our evaluation, serving as a lower bound for what lightweight VLMs can achieve on this task.[⁶](#page-3-2)

To assess the impact of prompt structure on performance, three prompts of increasing complexity that share one set of proxemic-zone definitions and identical

<span id="page-3-0"></span>⁴ https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct

<span id="page-3-1"></span>⁵ https://huggingface.co/OpenGVLab/InternVL3\\_5-4B

<span id="page-3-2"></span>⁶ https://huggingface.co/HuggingFaceTB/SmolVLM2-2.2B-Instruct

<span id="page-4-0"></span>JSON output format were created: (i) Simple: a brief task description to test the models' out-of-the-box knowledge [\[11\]](#page-9-8); (ii) Moderate: added a set of instructions and defined output constraints; and (iii) Advanced: an internal chain of reasoning [\[7\]](#page-8-4) based on visual cues, which closely resembles instruction-based reasoning chains [\[21\]](#page-9-10). Each requires a JSON response with a distance zone, distance estimate, bounding box, and overall danger level; only the danger level is scored for classification, and the bounding box is scored separately for grounding.

3.4 Fine-Tuning Procedure

We perform fine-tuning with parameter-efficient QLoRA in two stages, creating three checkpoints: before, after, and after\_2. A total of 200 images were selected for the training set, distributed evenly across danger levels. To avoid data leakage, images were taken from videos different from those used for creating the test set. Stage 1: adapts attention projections only (q\_proj, v\_proj). Stage 2: extends this to all main projections (q\_proj, k\_proj, v\_proj, o\_proj, gate\_proj, up\_proj, down\_proj, lm\_head). Both stages use r = 8, α = 16, dropout = 0.05, and no offset adaptation. Stage 2 extends the coverage of the adapter to improve task consistency, whilst the two-stage configuration allows us to assess whether additional fine-tuning improves performance or causes regression.

3.5 Evaluation Metrics

We report the following metrics for each model × fine-tuning stage × prompt type configuration:

Accuracy – the fraction of correctly classified images.

Weighted F1 – the average F1 score across all classes, weighted by the frequency of occurrence, which reflects the overall quality of the classification under class imbalance.

Recall of high danger class – recall specifically for the high danger class, defined as the proportion of correctly identified scenes with high danger for the robot. This is a key safety metric, as a false negative for a scene with high danger carries a greater cost than any other classification error.

IoU – a correctly classified danger level may not mean that the model is seeing the right person. We also measure how well the predicted bounding box matches the ground truth person position using IoU, with mean IoU, ratio of IoU > 0.5, and ratio of IoU = 0. This helps us to understand if the model knows where the scene is dangerous.

Table [1](#page-5-0) summarizes accuracy, weighted F1, and high-danger recall by model and fine-tuning stage (random baseline: wF1 = 0.25).

<span id="page-5-2"></span><span id="page-5-0"></span>Table 1: Overall performance by model and fine-tuning stage (averaged over prompt strategies).

As can be seen, all of the models examined show results that are close to the random baseline (weighted F1 score (wF1) 0.21–0.24). However, by shifting the focus to the study of high levels of danger, better results can be observed; specifically, for the Qwen model, the recall reaches 0.4 at all stages, versus nearzero for InternVL and SmolVLM. Fine-tuning does not lead to any gain (∆wF1 < 0.02).

Table [2](#page-5-1) gives classification and grounding by prompt strategy. The simple prompt achieves the highest overall F1 score and mean IoU, whilst the advanced prompt with a chain of reasoning yields the highest recall for the class of high danger. This inversion may point to a "format tax" [\[12\]](#page-9-9): more complex prompts introduce structured reasoning, which improves the detection of outliers but at the expense of overall classification consistency. The moderate prompt shows no clear advantage in either of these objectives; however, the use of this prompt yields the lowest total misalignment score.

Table 2: Performance by prompt strategy.

<span id="page-5-1"></span>

4.2 Spatial Grounding

Tab [3](#page-6-0) reports grounding by model and fine-tuning stage. It is evident from the table that the only model providing useful localisation is Qwen: it demonstrates relatively high metric values compared to other models, namely an average IoU of up to 0.46 and IoU > 0.5 on over half of the predictions before fine-tuning. The other two models generally do not provide localisation. As with classification, fine-tuning does not improve localisation for any of the models; in fact, it even worsens it slightly.

<span id="page-6-0"></span>Table 3: Grounding performance by model and fine-tuning stage.

4.3 Per-Class Analysis

Tab [4](#page-6-1) reports precision, recall, and F1 per danger class for the best overall configuration (Qwen, after\_2, simple prompt) and the best high-recall configuration (Qwen, after, advanced prompt), alongside mean IoU per true class, showing that grounding quality does not track classification performance on a per-class basis either.

<span id="page-6-1"></span>Table 4: Per-class metrics for the two best Qwen configurations. Precision (P), recall (R), F1-score (F1), and mean IoU (grounding accuracy on that class) are reported for each class.

No single configuration allows for the optimization of both overall accuracy and high-danger recall at the same time; however, in order to integrate the model into the robot control system, it must serve as a reliable danger trigger. The two best-performing combinations investigated: (1) Qwen + simple prompt + a second extended round of fine-tuning, provides the best overall performance (weighted F1 = 0.290) and balanced classification across all danger levels; (2) Qwen + advanced prompt + first simple fine-tuning round achieves the highest recalls for high danger (0.79), but performs poorly on other classes. It is also important to note that this high-danger configuration is characterised by a high recall rate but very low precision and insufficient effectiveness for the other classes. Therefore, its practical deployability as a safety system has not yet been conclusively demonstrated. In contrast, InternVL and SmolVLM cannot reliably detect situations of high danger even after fine-tuning.

Our research shows that the models examined generally do not take into account data on the spatial location of people when determining the level of danger. This conclusion is supported by the IoU value, which shows no notable difference between correctly and incorrectly classified frames (0.204 versus 0.215), and even Qwen's best performance on true positives in the high danger class (IoU = 0.488) barely exceeds its performance on false negatives in the same class (IoU = 0.464). This suggests that a high danger classification label does not confirm that the model has identified the responsible person for this situation.

5.3 Why Fine-Tuning Yields Limited Gains

The results suggest that the QLoRA configuration offers minor improvement, which is may be due to structural limitations rather than the choice of tuning parameters.

First, the dataset is likely too small; fine-tuning was performed on a balanced set of 200 images, meaning each class had exactly 50 examples, which could potentially limit the learning signal.

Second, Qwen already demonstrates a high recall rate for high danger class without training, suggesting that spatial reasoning is largely acquired during pre-training rather than in the layers adapted for QLoRA.

Third, additional fine-tuning is detrimental to small models, such as SmolVLM (2.2 billion parameters). This is due to the phenomenon known as "catastrophic forgetting" [\[10\]](#page-8-9), which arises from the difficulty of adapting new, task-specific patterns to the model's architecture when capacity is limited.

Our work explored the ability of vision-language models to predict the danger level in egocentric robot pictures in terms of proxemics and to provide enough spatial grounding for the purposes of an embodied control loop. The answer to the question we posed in the title of the paper is only partially and unevenly can VLMs assess proxemic risk from egocentric data.

One of the models, Qwen, demonstrates the ability to detect high levels of danger; however, the other two models produce almost entirely zero results when working with the same class, regardless of prompts or fine-tuning. All configurations return overall F1-scores close to the random baseline. Grounding accuracy follows the same pattern: Qwen is the only model whose predictions regarding the bounding box closely match the ground truth – but, most importantly, this alignment is largely independent of classification accuracy: even Qwen's correct predictions of a high level of danger do not demonstrate reliably better agreement than those which the model failed to detect. From these results, we can conclude that four-class proxemic reasoning based on single-view egocentric frames remains an unsolved problem for modern VLMs.

Future research should explore the use of larger training datasets and comparisons with more powerful API-scale models to determine whether performance limitations are due to the model's architecture or depend on its scale, and whether classification and grounding can be jointly improved.

• <span id="page-8-3"></span>1. Alayrac, J.B., Donahue, J., Luc, P., et al.: Flamingo: a visual language model for few-shot learning (2022), https://arxiv.org/abs/2204.14198 [3](#page-2-0)
• <span id="page-8-6"></span>2. Bai, J., et al.: Qwen-vl: A versatile vision-language model for understanding (2023), https://arxiv.org/abs/2308.12966 [4](#page-3-3)
• <span id="page-8-7"></span>3. Chen, Z., et al.: Internvl: Scaling up vision foundation models (2024), https: //arxiv.org/abs/2404.16821 [4](#page-3-3)
• <span id="page-8-1"></span>4. Daza, M., Barrios Aranibar, D., Diaz Amado, J.A., et al.: An approach of social navigation based on proxemics for crowded environments of humans and robots. Micromachines 12(2), 193 (2021). https://doi.org/10.3390/mi12020193 [2](#page-1-0)
• <span id="page-8-5"></span>5. Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: Qlora: Efficient finetuning of quantized llms (2023), https://arxiv.org/abs/2305.14314 [3](#page-2-0)
• <span id="page-8-2"></span>6. Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al.: An image is worth 16x16 words: Transformers for image recognition at scale (2020), https://arxiv.org/abs/2010. 11929 [3](#page-2-0)
• <span id="page-8-4"></span>7. Ge, J., Luo, H., Qian, S., et al.: Chain of thought prompt tuning in vision language models (2023), https://arxiv.org/abs/2304.07919 [3,](#page-2-0) [5](#page-4-0)
• <span id="page-8-0"></span>8. Hall, E.T.: The Hidden Dimension. Doubleday (1966) [2,](#page-1-0) [4](#page-3-3)
• <span id="page-8-8"></span>9. HuggingFace: Smolvlm (2024), https : / / huggingface . co / HuggingFaceTB / SmolVLM-Instruct [4](#page-3-3)
• <span id="page-8-9"></span>10. Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., Hassabis, D., Clopath, C., Kumaran, D., Hadsell, R.: Overcoming catastrophic forgetting in

• neural networks. Proceedings of the National Academy of Sciences 114(13), 3521– 3526 (2017). https://doi.org/10.1073/pnas.1611835114, https://www.pnas. org/doi/abs/10.1073/pnas.1611835114 [8](#page-7-0)
• <span id="page-9-8"></span>11. Kojima, T., Gu, S.S., Reid, M., et al.: Large language models are zero-shot reasoners (2022), https://arxiv.org/abs/2205.11916 [3,](#page-2-0) [5](#page-4-0)
• <span id="page-9-9"></span>12. Lee, I.Y., D'Antoni, L., Berg-Kirkpatrick, T.: The format tax (2026), https:// arxiv.org/abs/2604.03616 [3,](#page-2-0) [6](#page-5-2)
• <span id="page-9-6"></span>13. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning (2023), https:// arxiv.org/abs/2304.08485 [3](#page-2-0)
• <span id="page-9-1"></span>14. Martín-Martín, R., Patel, M., Rezatofighi, H., et al.: Jrdb: A dataset and benchmark of egocentric robot visual perception of humans in built environments. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(6), 6748–6765 (2023). https://doi.org/10.1109/TPAMI.2021.3070543 [2,](#page-1-0) [3](#page-2-0)
• <span id="page-9-2"></span>15. Patompak, P., Jeong, S., Nilkhamhang, I., Chong, N.Y.: Learning proxemics for personalized human–robot social interaction. International Journal of Social Robotics 12, 267–280 (2020). https://doi.org/10.1007/s12369-019-00560-9 [2](#page-1-0)
• <span id="page-9-5"></span>16. Radford, A., Kim, J.W., Hallacy, C., et al.: Learning transferable visual models from natural language supervision (2021), https://arxiv.org/abs/2103.00020 [3](#page-2-0)
• <span id="page-9-0"></span>17. Samarakoon, S.M.B.P., Muthugala, M.A.V.J., Jayasekara, A.G.B.P.: A review on human–robot proxemics. Electronics 11(16), 2490 (2022). https://doi.org/10. 3390/electronics11162490 [2](#page-1-0)
• <span id="page-9-4"></span>18. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems (2017), https://arxiv.org/abs/1706.03762 [3](#page-2-0)
• <span id="page-9-3"></span>19. Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: A neural image caption generator (2015), https://arxiv.org/abs/1411.4555 [3](#page-2-0)
• <span id="page-9-7"></span>20. Yu, S., Chen, Y., Ju, H., et al.: How far are vlms from visual spatial intelligence? a benchmark-driven perspective (2025), https://arxiv.org/abs/2509.18905 [3](#page-2-0)
• <span id="page-9-10"></span>21. Zhang, S., Dong, L., Li, X., et al.: Instruction tuning for large language models: A survey (2023), https://arxiv.org/abs/2308.10792 [5](#page-4-0)