Long-Term Human Trajectory Prediction Using 3D Dynamic Scene Graphs

被引:0
|
作者
Gorlo, Nicolas [1 ]
Schmid, Lukas [1 ]
Carlone, Luca [1 ]
机构
[1] MIT, MIT SPARK Lab, Cambridge, MA 02139 USA
来源
IEEE ROBOTICS AND AUTOMATION LETTERS | 2024年 / 9卷 / 12期
基金
瑞士国家科学基金会; 芬兰科学院;
关键词
Trajectory; Probabilistic logic; Three-dimensional displays; Predictive models; Indoor environment; Planning; Cognition; Annotations; Service robots; Legged locomotion; AI-enabled robotics; human-centered robotics; service robotics; datasets for human motion; modeling and simulating humans; NAVIGATION;
D O I
10.1109/LRA.2024.3482169
中图分类号
TP24 [机器人技术];
学科分类号
080202 ; 1405 ;
摘要
We present a novel approach for long-term human trajectory prediction in indoor human-centric environments, which is essential for long-horizon robot planning in these environments. State-of-the-art human trajectory prediction methods are limited by their focus on collision avoidance and short-term planning, and their inability to model complex interactions of humans with the environment. In contrast, our approach overcomes these limitations by predicting sequences of human interactions with the environment and using this information to guide trajectory predictions over a horizon of up to 60s . We leverage Large Language Models (LLMs) to predict interactions with the environment by conditioning the LLM prediction on rich contextual information about the scene. This information is given as a 3D Dynamic Scene Graph that encodes the geometry, semantics, and traversability of the environment into a hierarchical representation. We then ground these interaction sequences into multi-modal spatio-temporal distributions over human positions using a probabilistic approach based on continuous-time Markov Chains. To evaluate our approach, we introduce a new semi-synthetic dataset of long-term human trajectories in complex indoor environments, which also includes annotations of human-object interactions. We show in thorough experimental evaluations that our approach achieves a 54% lower average negative log-likelihood and a 26.5% lower Best-of-20 displacement error compared to the best non-privileged (i.e., evaluated in a zero-shot fashion on the dataset) baselines for a time horizon of 60 s .
引用
收藏
页码:10978 / 10985
页数:8
相关论文
共 50 条
  • [41] RP-SG: Relation Prediction in 3D Scene Graphs for Unobserved Objects Localization
    Ying, Zhongmou
    Yuan, Xianfeng
    Yang, Baojiang
    Song, Yong
    Xu, Qingyang
    Zhou, Fengyu
    Sheng, Weihua
    IEEE ROBOTICS AND AUTOMATION LETTERS, 2024, 9 (02) : 1412 - 1419
  • [42] 3D dynamic scene surveillance and management using a 3D kinetic spatial data structure
    Mostafavi, Mir Abolfazl
    Beni, Leila Hashemi
    Gavrilova, Marina
    INTERNATIONAL CONFERENCE ON ADVANCED GEOGRAPHIC INFORMATION SYSTEMS AND WEB SERVICES: GEOWS 2009, PROCEEDINGS, 2009, : 45 - +
  • [43] Motion Field Estimation for a Dynamic Scene Using a 3D LiDAR
    Li, Qingquan
    Zhang, Liang
    Mao, Qingzhou
    Zou, Qin
    Zhang, Pin
    Feng, Shaojun
    Ochieng, Washington
    SENSORS, 2014, 14 (09) : 16672 - 16691
  • [44] Dynamic 3D Scene Reconstruction and Enhancement
    Jiang, Cansen
    Fougerolle, Yohan
    Fofi, David
    Demonceaux, Cedric
    IMAGE ANALYSIS AND PROCESSING,(ICIAP 2017), PT I, 2017, 10484 : 518 - 529
  • [45] Automatic Generation of 3D Scene Animation Based on Dynamic Knowledge Graphs and Contextual Encoding
    Wenfeng, Song
    Xinyu, Zhang
    Yuting, Guo
    Shuai, Li
    Aimin, Hao
    Hong, Qin
    INTERNATIONAL JOURNAL OF COMPUTER VISION, 2023, 131 (11) : 2816 - 2844
  • [46] Automatic Generation of 3D Scene Animation Based on Dynamic Knowledge Graphs and Contextual Encoding
    Wenfeng Song
    Xinyu Zhang
    Yuting Guo
    Shuai Li
    Aimin Hao
    Hong Qin
    International Journal of Computer Vision, 2023, 131 : 2816 - 2844
  • [47] Lost & Found: Tracking Changes From Egocentric Observations in 3D Dynamic Scene Graphs
    Behrens, Tjark
    Zurbrugg, Rene
    Pollefeys, Marc
    Bauer, Zuria
    Blum, Hermann
    IEEE ROBOTICS AND AUTOMATION LETTERS, 2025, 10 (04): : 3739 - 3746
  • [48] Look Outside the Room Synthesizing A Consistent Long-Term 3D Scene Video from A Single Image
    Ren, Xuanchi
    Wang, Xiaolong
    2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2022), 2022, : 3553 - 3563
  • [49] 3D Culture Supports Long-Term Expansion of Mouse and Human Nephrogenic Progenitors
    Li, Zhongwei
    Araoka, Toshikazu
    Wu, Jun
    Liao, Hsin-Kai
    Li, Mo
    Lazo, Marta
    Zhou, Bing
    Sui, Yinghui
    Wu, Min-Zu
    Tamura, Isao
    Xia, Yun
    Beyret, Ergin
    Matsusaka, Taiji
    Pastan, Ira
    Esteban, Concepcion Rodriguez
    Guillen, Isabel
    Guillen, Pedro
    Campistol, Josep M.
    Belmonte, Juan Carlos Izpisua
    CELL STEM CELL, 2016, 19 (04) : 516 - 529
  • [50] Incorporating long-term observations of human actions for stable 3D people tracking
    Sugimura, Daisuke
    Kobayashi, Yoshinori
    Sato, Yoichi
    Sugimoto, Akihiro
    2008 IEEE WORKSHOP ON MOTION AND VIDEO COMPUTING, 2008, : 30 - +