Vision for Autonomous Vehicles
We are aiming for a world where vehicles are smart and truly safe, and lives are not tragically lost on our roadways. A big part of achieving this will be to add sensing capability to vehicles to give them super-human perception. Now sensors on modern vehicles generate gigabits of data per second, and making sense of these data and turning them into useful knowledge can be hard. Our work creates new algorithms for sensor processing (especially video, lidar and radar) to enable automated scene modeling and understanding. This page is a selection of the recent work in the 3D Vision Lab and collaborations related to autonomous vehicles.
TransCAR
Automotive radars provide complementary data to cameras, and so have potential to be leveraged for improved object detection. We propose a Transformer-based fusion for radar and camera-based detection. A pre-print is available and will be presented at IROS 2023.
RADIANT: Radar-Image Association Network for 3D Object Detection
Here we propose a technique that associates radar returns with image-based object centers and then uses them to improve the performance of image-based object detection. This work is published in AAAI 2023.
Fast CLOCs Fusion of Lidar and Camera
Su Pang's latest Lidar-camera fusion technique achieves a big speed-up and reduced memory requirements over our earlier CLOCs fusion paper, and can now run on a single-GPU computer. It provides a convenient and effective way to combine a lidar-based detector and image detector to achieve state-of-the-art 3D detection performance. Our 2022 WACV paper is now available.
Full-Velocity Radar Returns by Radar-Camera Fusion
Yunfei Long's new work calculates full-velocity radar returns using radar-camera fusion. This overcomes the limitation of Doppler radar, which only provides radial velocity measurements. Our paper was an oral presentation at ICCV 202. Enjoy the video demo below.
Depth Completion with Twin Surface Extrapolation at Occlusion Boundaries
Saif Imran's new, improved depth completion addressed the tricky problem of super-resolving depth at occlusion boundaries. We do this by extrapolating both foregound and background surfaces across the boundary, and then fuse them. This work was presented at CVPR 2021, see: paper.
Radar-Camera Pixel Depth Association for Depth Completion
Yunfei Long's new radar-camera depth completion method was presented at CVPR 2021, see paper. We address the problem of probabilistically associating radar returns to image pixels, and then use radar plus images to estimate a dense depth map.
3D Multi-Object Tracking using Random Finite Set-based Multiple Measurement Models Filtering (RFS-M^3) for Autonomous Vehicles
Su Pang's new tracking method using Random Finite Sets for more rigorous probabilistic scene modeling obtains state-of-the-art tracking performance and was presented at ICRA 2021. The paper is now available.
Automated Vehicles Sharing the Road: Surveying Detection and Localization of Pedalcyclists
Automated vehicles create dangers and opportunities for improved safety of pedalcyclists as they share the road. Our paper appearing in Dec 2020 IEEE Trans. on Intelligent Vehicles, surveys detection and localization methods for pedalcyclists.
Lidar Essential Beam Model
Yunfei Long showed that using a traditional ray model for lidar beams often results in shape erosion or dilation. This can have a deleterious effect on width estimation of thin objects common in driving environments (see figure below). Our work instead uses a linearly diverging model for lidar beams with a finite beam width at measurement. We show how this beam width can be calibrated, and how it can be used to more accurately estimate the dimensions of scene objects. Here is our IROS 2020 paper.
CLOCs: Camera-Lidar Object Candidates Fusion for 3D Object Detection
Su Pang's new Camera-Lidar fusion technique, which we call CLOCs, is able to combine 3D and 2D detection methods to achieve superior 3D performance. With very little overhead, our fusion network can be trained to combine lidar and camera techniques to achieve state-of-the-art fused performance. Our paper was presented at IROS in October 2020. And our code is now available.
 |
 |
Depth Completion for Super-resolving Lidar
Saif Imran, Xiaoming Liu and I developed a new technique for color-image-guided upsampling of lidar. In contrast to many other approaches, our method avoids smearing depth across object boundaries and depth discontinuities. Our CVPR 2019 paper Depth Coefficients for Depth Completion describes this, and code is here. The video below shows a color image with sparse lidar points plotted on top. The lower half shows the dense depth estimated by our method.
Localization with HD and FLAME Maps
High-definition 3D maps enable precise localization and pose estimation for autonomous vehicles without the need for GPS. We compared the suitability of ICP and NDT for localization in our 2018 VTC paper. Below we are localizing to 10cm accuracy via registration to a HD map:
A downside of HD maps is the large data requirement. To address this we developed FLAME (Feature Likelihood Aquisition Map Emulation) which appeared in IROS 2019. Below we achieve similar localization accuracy with a map 1/1000 of the size.
Turning behavior modeling
We have developed accurate vehicle pose tracking and turning models to enable rapid vehicle turning prediction, see: Turning Behavior Classification in IEEE Transactions on Vehicular Technology, 2019. The goal is to reduce the chance of collisions, especially at intersections.
Autodrive Challenge
 |
The MSU Autodrive Challenge team integrated sensing and control hardware and software into a GM Bolt.
Lidar-based Object detection and tracking
Autonomous vehicles of the future will need precise sensing of the world around them.� LIDAR is a promising sensor and provides 3D point clouds of the world.� Within these points clouds we seek for objects such as people and vehicles, track them and provide trajectory predictions.� Components of this problem include clustering 3D points to objects, rejecting clutter objects, developing appropriate shape and motion models, and accounting for self-occlusions and scene-occlusions.
Visual Classification of Coarse Vehicle Orientation using Histogram of Oriented Gradients Features Paul Rybski, Daniel Huber, Daniel D. Morris, and Regis Hoffman 2010 IEEE Intelligent Vehicles Symposium, June, 2010.
A View-Depdenent Adaptive Matched Filter for Ladar-Based Vehicle Tracking, Daniel D. Morris, Regis Hoffman, and Paul Haley Proc. of 14th IASTED Int. Conf. on Robotics and Applications, November, 2009.
Object classification using LIDAR
While range measurements from LIDAR are precise, they are sparse at long range.� As a result determining object shape and category can be difficult.� We develop 3D shape-based object categorization methods to classify object types
Rough Terrain and Ground Segmentation
An important initial step in local scene understanding is to estimate the ground surface. In flat open areas this is straight forward, but in cluttered environments and in rough terrain in can be challenging to separate ground surfaces from other objects. We have recently developed a new robust ground measurement cost function that accounts for occlusions and clutter. When modeled with a Markov Random Field and optimized with Loopy Belief Propagation, it produces high-quality ground segmentations of LIDAR data, see:
Ground Segmentation based on Loopy Belief Propagation for Sparse 3D Point Clouds, Mingfang Zhang, Daniel D. Morris, Rui Fu, Proceedings 3DV 2015.
