Skip to main content
Michigan State Universitysite name

Autonomous Vehicle Sensing

We are aiming for a world where vehicles are smart and truly safe, and lives are not tragically lost on our roadways. A big part of achieving this will be to add sensing capability to vehicles to give them super-human perception. Now sensors on modern vehicles generate gigabits of data per second, and making sense of these data and turning them into useful knowledge can be hard. Our work creates new algorithms for sensor processing (especially video, lidar and radar) to enable automated scene modeling and understanding. This page is a selection of the recent work in the 3D Vision Lab and collaborations related to autonomous vehicles.


We are excited to be joining the team from Politecnico di Milano and University of Alabama to develop sensing and control for an Indy autonomous racecar. Upcoming races will be in Monza (Italy), Goodwood (UK) and the Indianapolis speedway.


Automotive radars provide complementary data to cameras, and so have potential to be leveraged for improved object detection. We propose a Transformer-based fusion for radar and camera-based detection. A pre-print is available and will be presented at IROS 2023.

RADIANT: Radar-Image Association Network for 3D Object Detection

Here we propose a technique that associates radar returns with image-based object centers and then uses them to improve the performance of image-based object detection. This work is published in AAAI 2023.

Radar-Camera Association

Fast CLOCs Fusion of Lidar and Camera

Su Pang's latest Lidar-camera fusion technique achieves a big speed-up and reduced memory requirements over our earlier CLOCs fusion paper, and can now run on a single-GPU computer. It provides a convenient and effective way to combine a lidar-based detector and image detector to achieve state-of-the-art 3D detection performance. Our 2022 WACV paper is now available.


Full-Velocity Radar Returns by Radar-Camera Fusion

Yunfei Long's new work calculates full-velocity radar returns using radar-camera fusion. This overcomes the limitation of Doppler radar, which only provides radial velocity measurements. Our paper was an oral presentation at ICCV 202. Enjoy the video demo below.

Depth Completion with Twin Surface Extrapolation at Occlusion Boundaries

Saif Imran's new, improved depth completion addressed the tricky problem of super-resolving depth at occlusion boundaries. We do this by extrapolating both foregound and background surfaces across the boundary, and then fuse them. This work was presented at CVPR 2021, see: paper.

Depth completion obtaining fine details and boundaries

Radar-Camera Pixel Depth Association for Depth Completion

Yunfei Long's new radar-camera depth completion method was presented at CVPR 2021, see paper. We address the problem of probabilistically associating radar returns to image pixels, and then use radar plus images to estimate a dense depth map.

Radar-Camera Depth Completion

3D Multi-Object Tracking using Random Finite Set-based Multiple Measurement Models Filtering (RFS-M^3) for Autonomous Vehicles

Su Pang's new tracking method using Random Finite Sets for more rigorous probabilistic scene modeling obtains state-of-the-art tracking performance and was presented at ICRA 2021. The paper is now available.

Random Finite Set Tracking

Automated Vehicles Sharing the Road: Surveying Detection and Localization of Pedalcyclists

Automated vehicles create dangers and opportunities for improved safety of pedalcyclists as they share the road. Our paper appearing in Dec 2020 IEEE Trans. on Intelligent Vehicles, surveys detection and localization methods for pedalcyclists.

Sharing road with pedalcyclists

Lidar Essential Beam Model

Yunfei Long showed that using a traditional ray model for lidar beams often results in shape erosion or dilation. This can have a deleterious effect on width estimation of thin objects common in driving environments (see figure below). Our work instead uses a linearly diverging model for lidar beams with a finite beam width at measurement. We show how this beam width can be calibrated, and how it can be used to more accurately estimate the dimensions of scene objects. Here is our IROS 2020 paper.

Examples of erosion and dilation in a lidar scan

CLOCs: Camera-Lidar Object Candidates Fusion for 3D Object Detection

Su Pang's new Camera-Lidar fusion technique, which we call CLOCs, is able to combine 3D and 2D detection methods to achieve superior 3D performance. With very little overhead, our fusion network can be trained to combine lidar and camera techniques to achieve state-of-the-art fused performance. Our paper was presented at IROS in October 2020. And our code is now available.

CLOCs Example Detections CLOCs Example Detections

Depth Completion for Super-resolving Lidar

Saif Imran, Xiaoming Liu and I developed a new technique for color-image-guided upsampling of lidar. In contrast to many other approaches, our method avoids smearing depth across object boundaries and depth discontinuities. Our CVPR 2019 paper Depth Coefficients for Depth Completion describes this, and code is here. The video below shows a color image with sparse lidar points plotted on top. The lower half shows the dense depth estimated by our method.

Depth Completion Example

Localization with HD and FLAME Maps

High-definition 3D maps enable precise localization and pose estimation for autonomous vehicles without the need for GPS. We compared the suitability of ICP and NDT for localization in our 2018 VTC paper. Below we are localizing to 10cm accuracy via registration to a HD map:

Vehicle pose estimation from HD map

A downside of HD maps is the large data requirement. To address this we developed FLAME (Feature Likelihood Aquisition Map Emulation) which appeared in IROS 2019. Below we achieve similar localization accuracy with a map 1/1000 of the size.

Localization from FLAME map

Turning behavior modeling

We have developed accurate vehicle pose tracking and turning models to enable rapid vehicle turning prediction, see: Turning Behavior Classification in IEEE Transactions on Vehicular Technology, 2019. The goal is to reduce the chance of collisions, especially at intersections.

Oncoming vehicle turning at an intersection

Autodrive Challenge

MSU Autodrive

The MSU Autodrive Challenge team integrated sensing and control hardware and software into a GM Bolt.

Lidar-based Object detection and tracking

Kinematic models for vehicle trackingAutonomous vehicles of the future will need precise sensing of the world around them.� LIDAR is a promising sensor and provides 3D point clouds of the world.� Within these points clouds we seek for objects such as people and vehicles, track them and provide trajectory predictions.� Components of this problem include clustering 3D points to objects, rejecting clutter objects, developing appropriate shape and motion models, and accounting for self-occlusions and scene-occlusions.
Visual Classification of Coarse Vehicle Orientation using Histogram of Oriented Gradients Features
Paul Rybski, Daniel Huber, Daniel D. Morris, and Regis Hoffman 2010 IEEE Intelligent Vehicles Symposium, June, 2010.

A View-Depdenent Adaptive Matched Filter for Ladar-Based Vehicle Tracking, Daniel D. Morris, Regis Hoffman, and Paul Haley Proc. of 14th IASTED Int. Conf. on Robotics and Applications, November, 2009.

Object classification using LIDAR

LIDAR pointsWhile range measurements from LIDAR are precise, they are sparse at long range.� As a result determining object shape and category can be difficult.� We develop 3D shape-based object categorization methods to classify object types

Rough Terrain and Ground Segmentation

Rough terrain and ground segmentation in cluttered environments

An important initial step in local scene understanding is to estimate the ground surface. In flat open areas this is straight forward, but in cluttered environments and in rough terrain in can be challenging to separate ground surfaces from other objects. We have recently developed a new robust ground measurement cost function that accounts for occlusions and clutter. When modeled with a Markov Random Field and optimized with Loopy Belief Propagation, it produces high-quality ground segmentations of LIDAR data, see:
Ground Segmentation based on Loopy Belief Propagation for Sparse 3D Point Clouds, Mingfang Zhang, Daniel D. Morris, Rui Fu, Proceedings 3DV 2015.