of Objects
3D Localization of Objects Using Pointcloud and Image (Automotive)
The Challenge
3D Object detection and localization play an important role in environmental understanding, as it will not only identify surrounding objects but also locate them with respect to them. A self-driving car continuously collects data from multiple sensors.
In this case study, we used data collected from the Lidar sensor and 6 Camera Sensors to detect and localize surrounding cars using Lyft self-driving data published in 2019.
Approach
Data Preparation: We Voxelized the Lidar Point Cloud and used the Bird’s eye view and cylindrically projected images for model input.
Base Model: We used the Aggregate View Object Detector (AVOD), a two-staged detection network with slight modifications. The first stage of the model finds all the regions that are likely to have the object, and the second stage of the model refines those findings for more accurate predictions.
Model Training: Performed feature extraction with convolution on processed images and lidar data and fused the feature map crops based on anchor boxes for both the data. Performed objectness classification for each crop and used regression to find offsets and orientation of anchor box crops.
Experimentation: We experimented with different ways for cylindrical projections and mapping of 3D points to images and with a different set of anchor boxes for model training.
Infrastructure and Hardware: We preformed data engineering and model training on the local workstation with 20 CPU Cores, 128gb RAM, and TITAN RTX GPU.
Results
%
for detecting cars in 50m radius (0.5 3D IOU)
%
for detecting cars in 25m radius (0.5 3D IOU)
Conclusion
Accurate detection of thousands of classes of objects on the streets such as road signs, road edges, lanes, traffic signals, cars, trucks, pedestrians, and their respective travel paths to predict and decide the next action in real-time is the fundamental task for autonomous operations.
In this initial effort, we combined lidar data with the cylindrically stitched visual frames from the 6 cameras to identify vehicles around the car in 360 degrees. This process can be extended to use only cameras for 3-D object detection, tracking, and monitoring with other robotic systems.
Making autonomous vehicles possible – and safer – is just one way Artificial Intelligence is changing the world around us. Check out our article where we dove into how AI is revolutionising business and the 8 key functions seeing massive impacts today.