Making and maintaining maps accurate to the centimeter; automation demands it

High definition (HD) map making is part of the HERE brand pedigree. We use AI, LIDAR and a host of other data to continue that legacy.

Today, as the company launches worldwide users into the next century of artificial intelligence, it continues to build on its unrivaled expertise and global map data assets to revolutionize the way complexities of map making are improved. This work focuses on automatically identifying road features such as curbs, paint stripes, and crosswalks for developing HD maps in complex urban environments.

Previous versions of digital maps represent each road with an approximate centerline, along with a collection of basic attributes, such as speed limit and travel direction. Today’s applications require vastly more detailed information that is spatially accurate down to the smallest detail. Contemporary HD map applications require detailed lane information, including the precise location of lane boundaries, width, and how they interconnect. Scaling globally with this level of detail requires focusing on automated feature detectors, such as those presented.

HERE uses these features to create logical structures of HD maps that capture lane extent, connectivity, travel direction, and access characteristics. This requires further automation with sophisticated, structured models to capture interdependencies of all detected features, lane geometry, and map attributes. Since data is used to position and navigate a vehicle within a lane, it must be accurate to the centimeter. This additional effort means automation is even more critical.

Robust Data

HERE invests in car-based observations and thousands of other data sources that are deployed to gather, and continually update, millions of kilometers of road-feature information in more than 200 countries. HERE True Cars travel roads worldwide, capturing surroundings with the highest fidelity. Each vehicle is outfitted with state-of-the-art, data-capture sensors, including high-resolution cameras and laser range finders that provide 360-degree views and a high-accuracy positioning system.

Images captured by sensors provide a detailed surface model, allowing HERE developers to measure features ranging from centimeter-scale bumps and holes, to road slope, bank, crown and incline. In addition to building HD maps from high-fidelity data, HERE uses this as the standard to fine-tune, validate, and refine the process for combining original equipment manufacturer (OEM) features.

HERE True LiDAR 3D Point Clouds

LiDAR is at the core of HERE mapping, as sensors deliver quality, accurate information for modeling complex urban environments. A 360-degree point cloud is generated 10 times per second, producing a collection of 3D measurements. Fusing these distance measurements with the car position and orientation, HERE accesses a fully georeferenced 3D point cloud that captures different views and rich details of the world, while eliminating sensitivity to lighting and the visual ambiguity of 2D cameras.

Working directly with these points in 3D is difficult, requiring optimized data structures and complex recognition models. Instead, HERE projects point clouds onto a 2D image, leveraging successes in image-based “deep learning,” a machine-learning method for inferring relationships between input and desired output, such as identifying lane boundary locations from an image.

Deep learning avoids the need for task-specific engineering, where developers must specify features such as the color of grass, or where lane edge boundaries appear. LiDAR is prepared for deep learning by isolating the ground surface and projecting these points onto a top-down 2D image. For each pixel, reflectivity and ground surface altitude at that point is recorded, providing detailed 3D surface information.

A fully convolutional, deep network takes these images as input, then predicts a value for each pixel. In this application, HERE predicts the presence of road features such as lane and road boundaries, crosswalks and stop lines. These models are efficient and easy to optimize, serving as the backbone for many prediction tasks. In addition to per-pixel classifications for semantic segmentation, they feed more sophisticated models that predict bounding boxes for object detection, or at the far extreme, generate entirely new photorealistic images. This ensures that work is future-proofed for new map features.

The video below shows the HERE system running in a complex urban environment. The processed LiDAR surface is on the left. Road clutter has been removed, greatly simplifying the detection task. Detected features are overlaid on imagery, with lane boundaries appearing in yellow, road boundaries in blue. Since HERE retains all 3D information from the original point cloud, features are localized in global, 3D coordinates and rendered directly onto the 3D point cloud on the right.

Operational Efficiencies

Automating with deep learning increases operational efficiencies and is effective in resolving other challenges. Building an HD map is an intricate and laborious task when created by hand, and impossible to produce on a global scale without automation. Rather than manually making the map, HERE shifts the focus to annotating training data for machine-learning algorithms.

Automating feature detection and map creation processes allows HERE to react to changes as they are observed by any sensor. Automation is developed and applied quickly, and these feature-detection techniques transfer to many data sources and diverse mapping processes throughout the company. As one example, very similar approaches have also been used for real-time video processing for semantic segmentation.