The Michigan State University Four Seasons (MSU-4S) Dataset

We are at CVPR! Come see us on Friday, June 21st at 10:30 AM in the Arch Building, Hall 4E, Poster 302.

The Michigan State University Four Seasons (MSU-4S) Dataset is a unique dataset taken in and around MSU’s campus throughout all four seasons in various weather and environmental conditions.

Graphic showing front camera imagery in a grid, labels on top indicating what segment they came from. Top two rows show scenery from the same physical location.

MSU-4S was introduced at CVPR 2024 and is available to download under a non-commercial Creative Commons license.

Download Manuscript (Citation)

Download the Dataset

Current dataset version: v1.0. Release Notes.

Data labels will be available in a forthcoming release (est. Summer 2024).

To download the datset, we recommend using our downloader script. This script can be easily customized to only download the files you need.

All public dataset files are available to browse here.

Leaderboards

We plan to publish leaderboards for 3D and 2D object detection and domain adaptation tasks. We will announce more specific plans at a later date.

Selected Samples and Graphics

Fused frame output with camera, lidar, radar, and 3d object labels

Example of reflections and dead zones in rainy-weather lidar data

Technical Information

Vehicle and Sensor Hardware

Vehicle

The vehicle used for data collection is a modified 2017 Chevrolet Bolt EV. The vehicle was originally built as a part of the SAE AutoDrive competition, and as of publication remains the only one of these vehicles still operating in a research capacity.

Sensors

  • Compute system: Crystal Rugged compact rugged server, 44-core Intel Xeon
  • Network hardware: Netgear XXXX 10Gb link aggregation switch
  • Cameras: FLIR Blackfly BFLY-PGE-31S4C-C, 2048x1536 resolution.
  • Primary Lidar: 1x Ouster OS-1 (rev. 1) 64-line lidar
  • Additional Lidar: 1x Velodyne VLP-32 32-line lidar* (see errata)
  • Radar: 6x Continental ARS-430 radar
  • IMU: Yost 3-Space Nano 9DOF; onboard 6DOF IMU
  • Other hardware: custom-built CAN isolation system designed to allow one-way flow of vehicle CAN data to the vehicle compute system when system does not have autonomous driving capabilities enabled.

Data Format

There are currently 9 data collection segments in the dataset, separated by date of capture. Within each segment, there are subdirectories containing each data source, as follows:

  • top_left, top_mid, top_right : Three front-facing cameras. Imagery is 2048x1536, in JPEG format. Images are not dedistorted; calibration parameters can be found in the misc files. Optionally, pre-dedistorted imagery is available to download in both JPEG and PNG format.
  • oust : Ouster OS1-64 lidar, PCD format (main lidar data).
  • velo : Velodyne VLP-32 lidar, PCD format (additional lidar data; some data degradation present, see errata).
  • misc : Other sensor data including GPS, IMU, radar, transforms, camera calibration, and limited CAN data. YAML format.
  • label3d : 3D object data labels, YAML format.
  • label2d : 2D object data labels (center camera), YOLO format.

Each frame of the dataset is named with the time of collection. For example, one frame appears as such:

msu4s/
└── 2023_late_summer/
    ├── top_left/
    │   └── 1692390643884248064_top_left.jpg
    ├── top_mid/
    │   └── 1692390643884248064_top_mid.jpg
    ├── top_right/
    │   └── 1692390643884248064_top_right.jpg
    ├── oust/
    │   └── 1692390643884248064_oust.pcd
    ├── velo/
    │   └── 1692390643884248064_velo.pcd
    ├── misc/
    │   └── 1692390643884248064_misc.yaml
    ├── label2d/
    │   └── 1692390643884248064_label2d.txt
    └── label3d/
        └── 1692390643884248064_label3d.yaml

All frames are time synchronized in postprocessing, using the timestamp from each frame of the Velodyne lidar as the synchronization target.

Not all frames have published 3D and 2D labels; these labels are being reserved for future applications.

Format of misc.yaml

The misc.yaml file contains sensor readings from all non-camera and non-lidar sensors on the vehicle. Here is a selection of relevant keys:

gnss: GNSS location data from the Piksi Multi RTK GNSS. acc subkey is an estimated variance based on current fix type.

seasons: Current season. Current labeled seasons are spring, summer, fall, and winter.

weather: Labeled weather. Current labeled weather conditions are clear, rain, and snow.

zone: Sub-area of campus where frame was collected.

steering_angle: Approximate angle of steering. Based on publicly available specifications, the 2017 Chevy Bolt has a steering ratio of 16.8:1, a wheelbase of 260.096 cm (102.4 in) and a track width of 150.1 cm (59 in).

/can/chassis/imu - Data from the vehicle’s built-in IMU.

/sensors/imu/ouster_1/imu - Data from the IMU on board the Ouster OS-1 lidar.

/sensors/imu/piksi/imu - Data from the Piksi Multi GNSS’s onboard IMU. Orientation is not provided by the onboard IMU and has a reported covariance matrix of all -1.

/sensors/imu/yost/imu - Data from a Yost 9DOF IMU. Orientation is not absolutely calibrated to true north.

/sensors/camera/{top_left,top_mid,top_right}/image_color/compressed - Contains supplemental data for each of the cameras. Includes K and d matrices for dedistorting imagery, plus K_corr for a precomputed projection matrix for dedistorted imagery.

transforms: The static transform tree for all sensors on the vehicle. See the Transforms subsection for more details.

Sensor Transforms / Frames of Reference

We strongly recommend code using this dataset uses per-frame transform data instead of static transform data.

Diagram showing approximate sensor placements for vehicle.

Errata

Velodyne Lidar

We have noticed that in some segments, the Velodyne lidar data has artifacts in the output data. Based on manual review of data, we believe the sensor began to malfunction mid-collection, and thus should not be used when accuracy is desired. We are providing the Velodyne lidar data as-is in the hopes it can be useful for certain applications.

Radar

2023_late_summer and 2023_early_fall only have radar coverage from radars 2 and 5 (front and rear facing, respectively); all other radar channels in these segments have no data.