About me

My name is Haoyuan Fu. I'm a graduate student majoring in Computer Science in Shanghai Jiao Tong University (SJTU). I am expected to graduate in March. 2024. My educational experiences include:

  • Bachelor of Engineering (B.Eng) @ Computer Science and Engineering,
    Shanghai Jiao Tong University, Shanghai, China, Sep. 2017 - Jun. 2021;
  • Master of Engineering (M.Eng) @ Computer Technology,
    Shanghai Jiao Tong University, Shanghai, China, Sep. 2021 - Mar. 2024 (expected).
My research interest lies in Robotics, Simulation and Reinforcement Learning.

My hobbies include billiards, basketball and computer DIY. I won the champion of SJTU 'Freshman-Cup' billiards contest in Oct, 2021 and runner-up of SJTU SEIEE 'Rolling-Egg-Cup' basketball contest in Jun, 2021.



MVIG - Robotflow

Robotics, Simulation, Reinforcement Learning
Jul. 2019 - present
Advised by Prof. Cewu Lu. and Dr. Wenqiang Xu.
  • One of the main developers in project MVIG-Robotflow and contribute the initial version of it.
  • Develop RFUniverse simulation environment which aims to support all kinds of house-hold tasks. Mutiple tasks and baseline Reinforcement Learning algorithms have been tested on it. As a simulation system, this work has been heavily relied on by other works in our group: AKB-48, RCareWorld and RoboTube.
  • Co-author of Towards Real-World Category-level Articulation Pose Estimation (TIP 2022) and OMAD: Object Model with Articulated Deformations for Pose Estimation and Retrieval (BMVC 2021), where main contribution lies in data annotation and paper polishing.
  • Build simulation environment and train policices of all manipulation tasks in paper AKB-48: A Real-World Articulated Object Knowledge Base (CVPR 2022) using RL algorithms.
  • Design and build simulation environment and train policices for ADLs in paper RCareWorld: A Human-centric Simulation World for Caregiving Robots (IROS 2022), and this paper has been selected as a possible finalist of IROS Best Paper Award / Best Student Paper Award / IROS Best RoboCup Paper Award sponsored by RoboCup Federation.
  • Build the digital twin and train baselines in paper RoboTube: Learning Household Manipulation from Human Videos with Simulated Twin Environments (oral presentation of CoRL 2022 and RSS 2022 Workshop L-DOD).
  • Paper RFUniverse: A MultiPhysics Simulation Platform for Embodied AI is accepted by RSS 2023!


Tencent / Robotics X

Application Research
Dec. 2022 - present
I work as an intern in Agent center of Robotics X Lab. My main work includes optimizing Reinforcement Learning tools and developing simulation tasks using Nvidia Isaac Gym. Besides, I participated in smart car project of our group and tried to solve Sim2Real gaps for hardware and software.
  • Developing and optimizing distributed RL tool: As a widely used Distributed Reinforcement Learning tool in Tencent, Tleague originally uses TensorFlow as its structure. I replaced all TensorFlow codes with Pytorch with the help of Tianshou reserving socket communication under distributed setting. The main modifidied modules lies in Policy, Replay Buffer and corresponding Data Structure, Actor and Learner.
  • Sim2Real transfer of multi-agent smartcars collaboration task: A multi-agent smartcars collaboration task is created and trained with PPO algorithm in TLeague. We employed the trained policy on real smartcars and tried to solve the gap during transferring. The problems include: QRCode tag localization, PID control, creation and maintenance of dynamic map, and the continuousity of car actions.

Teaching Assistant

Operating System (D)

Spring 2022 semester
Feb. 2022 - Jul. 2022
Taught by Prof. Qianni Deng
The requests for this TA include understanding basic concepts and theory in Operating System, basic operations in Linux OS and C-programming on Linux. During the work, I was mainly responsible for refining slides, homeworks and answers which will be published to students. I worked on the second project: The HOST Dispatcher Shell which aims to help students understand process priority, process scheduling algorithms, memory allocation algorithms and C-programming on Linux platform. Besides, I made 3 presentations as tutorials during class to help students know better about project codes.

Selected Projects

Household environments are important testbeds for embodied AI research. Many simulation environments have been proposed to develop learning models for solving everyday household tasks. However, though interactions are paid attention to in most environments, the actions operating on the objects are not well supported concerning action types, object types, and interaction physics.
To bridge the gap at the action level, we propose a Unity3D-based action-centric environment, RFUniverse, for robot learning of everyday household tasks. It also provides a client-server communication framework based on gRPC, which can enable full functionality control of Unity with Python language. To demonstrate the usability of the simulation environment, we perform learning algorithms on various types of tasks, including manipulation, locomotion, multi-agent collaboration and task-and-motion-planning.
  • We construct a simulation environment RFUniverse, which can support physics-based interactions for all the extracted action and object types.
  • We benchmark 7 kinds of learning tasks on RFUniverse, which cover different aspects of the household tasks.
  • As a simulation system, this work has been heavily relied on by other works in our group: RoboTube, RCareWorld and AKB-48.

RoboTube video dataset contains 5,000 video demonstrations recorded with multi-view RGB-D cameras of human-performing everyday household tasks including manipulation of rigid objects, articulated objects, granular objects, deformable objects, and bimanual manipulation. RT-sim, as the simulated twin environments, consists of 3D scanned, photo-realistic objects, minimizing the visual domain gap between the physical world and the simulated environment.
  • We identify the issues in existing human videos for robot learning, and create a benchmark, RoboTube, which is designed by jointly considering the human video dataset and the evaluation platform.
  • RoboTube setups environments for 5 task families, namely drawer-closing, mug-pouring, cabinet-opening, bimanual-pot-lifting, and cloth-folding.
  • To benchmark the baseline methods, we construct a simulated twin environment, RT-Sim, for the tasks and objects. It is developed upon RFUniverse with photo-realistic objects to minimize the visual domain gap.
  • We deploy the learned model on real robot experiments.

In this paper, we present RCareWorld, a human-centric simulation world for physical and social robot care-giving with support for realistic human modeling, home environments with multiple levels of accessibility, and robots used for assistance.
  • We developed Care-Homes that accurately model caregiving homes with modifications for varying levels of accessibility and containing various assistive devices that are commonly used in caregiving scenarios.
  • We propose a set of realistic robotic caregiving tasks which can provide meaningful assistance to the modeled care-recipients with activities of daily living.
  • We present baseline control policies for each task in RCareWorld and validate our simulator by successfully applying one of them to real-world physical and social caregiving scenarios.

Social Activities

  • Group leader of “Zhu Fei Jiu Zhou Xing Project”, Aug. 2018.
  • Blood Donation, Nov. 2018 / Nov. 2019.
  • Volunteer of Shanghai International Marathon, Nov. 2018 / Nov. 2019.
  • Volunteer of Shanghai International Half Marathon, Apr. 2019.



  • Zhiyuan Honor Scholarship, 2017 / 2018 / 2019 / 2020; top 5%; CNY ¥5,000.
  • Shanghai Jiao Tong University Class B Excellent Scholarship, 2018; top 15%; CNY ¥1,000.
  • Shanghai Jiao Tong University Class C Excellent Scholarship, 2019 / 2020; top 30%; CNY ¥500.
  • Chen Hao Alumnus, WOSAI Technology Scholarship, 2019; top 15%; CNY ¥5,000.


  • National Undergraduate Mathetical Contest In Modeling, 2nd prize in Shanghai, 2018.
  • Mathetical Contest In Modeling, Honorable Mention Award, 2020.


  • Haoyuan Fu*, Wenqiang Xu*, Ruolin Ye*, Han Xue, Zhenjun Yu, Tutian Tang, Yutong Li, Wenxin Du, Jieyi Zhang and Cewu Lu. RFUniverse: A Multiphysics Simulation Platform for Embodied AI, Robotics: Science and Systems (RSS) 2023.
  • Haoyu Xiong*, Haoyuan Fu*, Jieyi Zhang, Chen Bao, Qiang Zhang, Yongxi Huang, Wenqiang Xu, AnimeshGarg and Cewu Lu. RoboTube: Learning Household Manipulation from Human Videos with Simulated TwinEnvironments, Conference on Robot Learning (CoRL), 2022.
  • Ruolin Ye*, WenqiangXu*, Haoyuan Fu, Rajat Kumar Jenamani, Vy Nguyen, Cewu Lu, Katherine Dim-itropoulou and Tapomayukh Bhattacharjee. RCareWorld: A Human-centric Simulation World for Caregiving Robots, IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022
  • Liu Liu, Wenqiang Xu, Haoyuan Fu, Sucheng Qian, Yang Han and Cewu Lu. AKB-48: A Real-World Articulated Object Knowledge Base, IEEE Computer Vision and Pattern Recognition (CVPR), 2022.
  • Liu Liu*, Han Xue*, Wenqiang Xu, Haoyuan Fu, and Cewu Lu. Towards Real-World Category-level Articulation Pose Estimation, in IEEE Transactions on Image Processing, vol. 31, pp. 1072-1083, 2022, doi: 10.1109/TIP.2021.3138644.
  • Han Xue*, Liu Liu*, Wenqiang Xu, Haoyuan Fu and Cewu Lu. OMAD: Object Model with Articulated Deformations for Pose Estimation and Retrieval, British Machine Vision Conference (BMVC), 2021.