Perception

The Perception team is responsible for classifying objects detected by our LIDAR system.

Computer Vision

Our boat currently uses a front-facing camera to collect image data, and we run both classical and machine learning algorithms to detect and classify objects in our field of view. To better utilize LIDAR data, we will add auxiliary cameras to our boat.

Synthesize data from multiple camera inputs to provide 360-degree computer vision
Train/evaluate YOLO models and interface with camera and sensor fusion
Technologies: Python, machine learning, ROS2, Pytorch, YOLO

Next steps: We would like to work more closely with LIDAR, creating multimodal models that integrate both LIDAR and camera data to produce more accurate predictions.

Lidar

Our boat uses a LIDAR sensor that returns point cloud data that we can use for object detection. We currently use geometric fitting algorithms to classify objects. We are exploring further strategies to improve our classification.

Train neural networks on simulated point cloud data to classify objects
Implement temporal object mapping by overlaying all same-cluster scans and building a map of all individual clusters with accumulated data
Technologies: Python, Pytorch, C++, Point Cloud Library

Next steps: We would like to explore semantic segmentation models to classify and detect objects in our point cloud, as well as explore early and late fusion models with camera data.

Sensor Fusion

Our boat fuses LIDAR and CV inputs into a cohesive global game state using spatial-temporal bookkeeping and layout-aware coordinate transforms. We deal with uncertainty, strict performance requirements, and synchronization.

Implement aggregation over all characteristics of objects (color, size, position, type, etc.) over time
Implement frontier/production-style object tracking utilizing probabilistic methods, and reducing reliance on permanent object memory
Explore algorithmic performance improvements for object equivalence checking
Technologies: Python, ROS2, Linear Algebra