Unitree Go1 Software Package

The Go1 Software Package is a modular, high-level control framework developed for Unitree robots. Designed at the DiCE Lab at San Diego State University, this framework bridges robotics, Building Information Models (BIM), and Unity-based simulations, simplifying mission planning and enabling seamless integration with sensors, edge devices, and APIs. The system includes a construction-specific implementation for Human Activity Recognition (HAR) using machine learning, enhancing trust in human-robot collaboration through real-time inference and explainability. The code, research paper, and image gallery for this project can be found at the links below.

GitHub Repository Research Paper Image Gallery

My Role

As a research assistant at the DiCE Lab, I contributed to the development and implementation of this project, and I was the primary author for the Explainable AI (XAI) feature for HAR. My responsibilities included implementing core components of the Python backend, integrating client-server communication, and enhancing transparency through natural language explanations generated by OpenAI's GPT model. In addition to my contributions as a programmer, I designed and manufactured custom components for the robotic vehicle that allowed for the installation of the electronics and sensors that facilitate human activity recognition.

Go1 and post-processing equipment — Go1 robot and 3D printing post-processing equipment during the construction of the mounting solution for the electronics used for human activity recognition.

Technical Details

Explainable AI Feature:
- Utilizes the explain_processor.py module to generate detailed explanations of activity inferences.
- Implements a pipeline that gathers inference data, including video frames and model outputs, and uses OpenAI's API to provide natural language explanations.
- Maximizes utility of the system by saving generated explanations alongside auto-labeled data.
Client-Server Architecture:
- Enables TCP-based communication through a modular Python backend.
- Facilitates real-time data exchange between Unity, the Python backend, and the robotic vehicle.
BIM and Unity Integration:
- Leverages BIM data for pathfinding and mission planning in Unity.
- Enables high-fidelity simulation of the robot's environment to facilitate a wide variety of applications.

Raspberry Pi serving video streams — A view of the Raspberry Pi that serves the video stream to the local server, enabling human activity recognition.

Challenges and Solutions

Dataset Scarcity: Construction-specific HAR datasets are limited. To address this, I helped develop a self-labeling mechanism that dynamically generates labeled data during HAR, demonstrating the potential for semi-supervised learning to streamline dataset creation.
Explainability in AI: Increasing transparency is crucial to promoting trust in human-robot collaboration. By integrating the OpenAI API, I ensured that the system could generate clear, context-aware explanations for all inferences made during HAR.

Initial testing of the explainable AI (XAI) feature during development. A researcher (Robert Ashe) swings a hammer to simulate a construction task, and the system responds with an explanation of the inference.

Outcome and Impact

Enhanced accessibility of robotics for non-experts by simplifying sensor integration and customization.
Improved human-robot collaboration through real-time HAR and explainable AI.
Open-source availability of the package supports adoption and customization by researchers across various industries.

Why Include This Project?

This project exemplifies my ability to collaborate using machine learning, robotics, and software engineering to address practical problems. It highlights my skills in Python development, explainable AI, and experience with modular systems that facilitate real-world applications.

Back to Homepage