Software

Open-source projects

DanceMVP: Self-Supervised Learning for Multi-Task Primitive-Based Dance Performance Assessment via Transformer Text Prompting

A novel method for human dance motion evaluation. Published at AAAI 2024. Download Code.

Forecasting Bimanual Object Manipulation Sequences From Unimanual Observations

A novel method for forecasting bimanual object manipulation sequences from unimanual observations. Published at AAAI 2024. Download Code.

Robot-Assisted-Dressing-Pipeline

Code to accompany our Science Robotics paper on "Learning garment manipulation policies toward robot-assisted dressing". Download code.

Action-Conditioned Generation of Bimanual Object Manipulation Sequences

A novel method that receives an action label to generate a motion sequence of a person performing bimanual object manipulation. Published at AAAI 2023. Download Code.

Holo-Spot Unity App and ROS packages (IROS 2022)

This project includes the Unity application and ROS packages needed for controlling Boston Dynamics' Spot using the Microsoft HoloLens 2. Download Code

Using Eye Gaze to Forecast Human Pose in Everyday Pick and Place Actions

A novel method that uses gaze information to predict the object being fixated, as well as the human pose the instant the fixated object is picked or placed. Published at ICRA 2022. Download Code.

Message-Passing Framework for Vision Prediction Stability in Human-Robot Interaction

To stabilise vision prediction results used in HRI, we present a message-passing framework (MPF) that uses the memory of individual modules to correct each other's outputs. The framework includes pose estimation, object detection, hand-object detection, and gaze estimation. Download Code

What is the Patient Looking At? Robust Gaze-Scene Intersection Under Free-Viewing Conditions

For gaze tracking under free-viewing conditions where there is large parallax between the user's head pose and the camera, we propose a fast and robust method to intersect the participant's gaze with the scene.  Download Code

Using a Single Input to Forecast Human Pose in Everyday Pick and Place Actions

A method that uses an input from a single timestep to directly forecast the human pose the instant a pick or place action is performed, as well as the time taken to complete the action. Published at ICASSP 2022. Download Code

Multitask Variational Autoencoding of Human-to-Human Handover

A method that directly forecasts the human pose and object orientation in human-to-human handover sequences. Published at IROS 2021. Download Code

Event-based vision model estimation

Entropy minimisation framework for event-based vision model estimation (see Nunes and Demiris ECCV2020). Download EventEMin.

RT-BENE blink estimation

A robust blink estimator based on the RT-GENE dataset (see Cortacero et al. ICCV2019W). Download RT-BENE (includes ROS code and link to the dataset).

RT-GENE gaze estimation

A new robust gaze estimation framework applicable in natural environments with large camera-subject distances (see Fischer et al. ECCV2018). Download RT-GENE (includes ROS code and link to the dataset).

Shared Control

An obstacle avoidance algorithm in C++ that has been adapted for shared control on differential-drive robots (see Zolotas and Demiris, IROS 2019 for description of its use on a robotic wheelchair). Download Shared Control Source (ROS package).

TRACA visual object tracking

A new context-aware correlation filter based tracking framework to achieve both high computational speed and state-of-the-art performance among real-time trackers (see Choi et al. CVPR2018). Download TRACA (includes ROS code).

iCub-HRI

A set of convenience wrappers related to perception, object manipulation and social interaction for the iCub humanoid robot (see Fischer et al. Frontiers in Robotics and AI 2018). Download iCub-HRI

Attentional Visual Tracking

Visual tracking with an attentional mechanism that chooses a subset of the associated correlation filters for increased robustness and computational efficiency (see Choi et al. CVPR2017). Download Attentional Visual Tracking Source. Please see also our TRACA CVPR2018 paper below.

Markerless Perspective Taking

C++ / Python framework which implements markerless perspective Taking (see Fischer and Demiris ICRA2016). Download Perspective Taking Source

Lifelong Augmentation of Multi-Modal Streaming Autobiographical Memories

C++ framework which implements an Autobiographical Memory geared towards the developmental robotics community (see Petit, Fischer and Demiris TCDS2016). Download Autobiographical Memory Source

Kinematic Structure Correspondences

Matlab package which implements learning correspondences between kinematic structures (see Chang et al. TPAMI). The download includes the dataset. Download Kinematic Structure Correspondences Source

Kinematic Structure Learning

Matlab package which implements unsupervised learning of kinematic structures (see Chang and Demiris CVPR2015). Download Kinematic Structure Learning Source

C++ package which implements offline and online unsupervised learning of kinematic structures based on 3D data (see Nunes and Demiris ICCV2020). Download 3D Kinematic Structure Learning Source

HAMMER

HAMMER is a C++ library to easily instantiate cognitive architectures based on the simulation theory of mind. Download HAMMER

Online Temporal Learning Library

The Online Temporal Learning C++ library implements several memory structures and includes two “high-level” online learning methods. Download OTL

Datasets

ImperialDance Dataset

This dataset contains human dance motion and music for different genres, choreographies and expertise levels (see Zhong et al. AAAI 2024). Download ImperialDance Dataset.

RT-BENE Blink Estimation Dataset

This dataset contains blink labels for the RT-GENE dataset (see Cortacero et al. ICCV2019W). Download RT-BENE dataset (includes source code).

RT-GENE Gaze Estimation Dataset

This dataset contains face and eye images along with the ground truth head pose and eye gaze angles (see Fischer et al. ECCV2018). Download RT-GENE dataset (includes source code).

Kinematic Structure Correspondences Dataset

This dataset contains kinematic structures and their correspondences, see Chang et al. CVPR2016. The download includes the CVPR2016 source code. Download Kinematic Structure Correspondences Dataset

Kinematic Structure Learning Dataset

This dataset contains various kinematic structures which can be used for unsupervised complex kinematic structure learning (see Chang and Demiris CVPR2015). Download Kinematic Structure Dataset

iCub Grasp Dataset

This dataset contains grasping data for nine-everyday objects collected using the iCub humanoid platform. Download Grasp Dataset

Other contributions

In addition, we have contributed code to the following open-source robotics projects: