Sensor Fusion and Multi Object Tracking

This project was completed as part of Udacity Nanodegree Program. LIDAR and Camera detections from the Waymo Open Dataset (highway sequence) are fused for Tracking Objects in the Driving scene using an Extended Kalman filter. A Track Management and Data Association module has been implemented for handling multiple objects in the scene. It takes care of Track Initialisation and maintains a Track State (confirmed, tentitive, initialised)and Track score. False positive objects with low score or those that exit the driving scene are deleted.

The following diagram contains an outline of the data flow and of the individual steps that make up the algorithm.

Installation Instructions.

Package Requirements

All dependencies required for the project have been listed in the file requirements.txt.

Waymo Open Dataset Reader

The Waymo Open Dataset Reader is a very convenient toolbox that allows you to access sequences from the Waymo Open Dataset.The installation instructions can be found in tools/waymo_reader/README.md.

Waymo Open Dataset Files

This project makes use of different sequences for Testing Algorithm: These are:

• Sequence 1 : training_segment-1005081002024129653_5313_150_5333_150_with_camera_labels.tfrecord
• Sequence 2 : training_segment-10072231702153043603_5725_000_5745_000_with_camera_labels.tfrecord
• Sequence 3 : training_segment-10963653239323173269_1924_000_1944_000_with_camera_labels.tfrecord

To download these files, you will have to register with Waymo Open Dataset first: Open Dataset – Waymo

Once you have done so, please click here to access the Google Cloud Container that holds all the sequences. Once you have been cleared for access by Waymo (which might take up to 48 hours), you can download the individual sequences. The sequences listed above can be found in the folder "training". Please download them and put the tfrecord-files into the dataset folder of this project.

Project File Structure

📦project
┣ 📂dataset --> should contain the downloaded Waymo Open Dataset sequences
┃
┣ 📂misc
┃ ┣ evaluation.py --> plot functions for tracking visualization and RMSE calculation
┃ ┣ helpers.py --> misc. helper functions, e.g. for loading / saving binary files
┗ params.py --> parameter file for the tracking
┣ 📂results --> binary files with pre-computed Lidar Detections
┃
┣ 📂student
┃ ┣ association.py --> data association logic for assigning measurements to tracks
┃ ┣ filter.py --> extended Kalman filter implementation
┃ ┣ measurements.py --> sensor model and measurement processing for camera and lidar
┃ ┗ trackmanagement.py --> track and track management classes
┃
┣ 📂tools --> external tools
┃ ┗ 📂waymo_reader --> functions for loading Waymo data sequences
┃
┣ loop_over_dataset.py --> Main script for Object Tracking

Using Sensor Data and Precomputed measurements

This project uses pre-computed lidar detections of Waymo Sequences so everyone can use same input data for implementing Tracking Algorithm download the pre-computed lidar detections (~1 GB), unzip them and put them in the folder results. For Camera measurements, noise is added the Waymo Image Labels and resulting noisy data is used as input for Sensor Fusion and Tracking.

Running the Project

Please execute loop_over_dataset.py to run the project

Results and Performance:

1.Ghost Object/ Clutter Rejection :

The Algorithm sucessfully deletes false positive Lidar or Camera detections which have a low Track Score and High estimation Covariance.

2.Initialisation of New Tracks:

New Tracks are sucessfully initialised for handling new Cars that enter the driving scene

3.Root Mean Squared Error:

The state estimations were compared to Ground truth data and accumulated trackwise RMSE values for the entire sequence have been plotted below.

External Dependencies

Parts of this project are based on the following repositories:

• Simple Waymo Open Dataset Reader