The evolution of the HERE HD Live Map at Daimler

In late 2010, a group of researchers gathered in a small conference room in Stuttgart. They were halfway through a two-day workshop to determine which new projects they would tackle when, while summarizing the first day, their boss voiced frustration that most of the proposals were merely increments of things they had done before.

His people looked at him a bit puzzled.  In their field of work, driver assistance systems, this was exactly what you did: you identified potential accident situations, and applied technology to solve them, one by one. When you managed to avoid rear collisions at a driving speed of 30 km/h, your next natural goal was to do it for 50, then perhaps 70.

“Can we turn things around? Rather than developing systems that just kick in when an accident is imminent, can we design a car that does the driving itself?”

“But we did this before. It did not work well. And, by the way, no one we showed it to wanted it,” replied one of the team.

Daimler was one of the first manufacturers to investigate self-driving vehicles in the mid-Nineties. Although their project produced vehicles that managed to travel a thousand kilometers on the Autobahn by themselves, the conclusion was that the technology was too complex, too unreliable, and too costly.

One of the biggest problems in research is that you can miss the right window of opportunity. Daimler was too early with its autonomous vehicles, and was at risk of overlooking that advances in sensor technology and vehicle intelligence would become game changers.

“Can we be absolutely sure that the problem was too hard to solve?  Or, was the approach to solving it just not good enough?”

I happened to be the new boss at the time and I owe it to the brilliance of my people that it only took them one night to reconsider. In the morning, our workshop room was buzzing with energy and in the early afternoon we had laid out how our self-driving car would work. At the end, only two things were missing: how long would we give ourselves to be ready and where would we test our car?

One of the benefits of working in a company that is more than a hundred years old is heritage. With both inventors of the motor vehicle as the founding fathers of the company, you cannot just draw on one set of old stories, but two. One of the most famous ones in Daimler's history goes back to the year 1888, when Karl Benz’s wife Bertha literally stole a vehicle from her husband to visit her relatives some one hundred kilometers away. Her journey was to become the world’s first long-distance drive in a motor vehicle.

I still very vividly remember the energy of that moment on that second workshop day when we joined the vision of the car that we wanted to build with the notion of testing it on the original route that Bertha Benz took. What’s more, it was clear that we needed to be done by the 125th anniversary of her original drive, coming in the Summer of 2013.  And as a brand-new Mercedes Benz S-Class was due to make its debut that summer, it was also clear which base vehicle we would use.

We got a bit drunk from celebrating that night. Our hangover got severely amplified when we found out what we had gotten ourselves into. When we formed our bold plan, none of us had actually seen the Bertha Benz Route. When we then traveled it from Mannheim via Heidelberg and Karlsruhe to Pforzheim for the first time, the car grew quieter and quieter as reality confronted us.

We had not anticipated the complex intersections, countless roundabouts, and narrow passages that make up the route today. I still remember that at one point one of us said “I am pretty sure Bertha would have taken the Autobahn nowadays. Can we not simply do that?” But we concluded that this would have been cheating.

Any autonomous vehicle reconstructs its environment via its own onboard sensors. Ours would do this as well. But to help our vehicle to make better sense of what it saw, we wanted to give it some advance information about the road ahead.

A map of some sort

An autonomous vehicle would need to be more than an autopilot system that keeps the car in the lane when it travels down the highway. It would need a map -  one that can almost be used like an additional sensor, to reinforce and validate what the vehicle saw itself. The content elements and the level of detail in such a map needed to be higher than what traditional navigation maps typically show. And to make this distinction clear, we started to call it a “high-definition map”.

As we were not familiar with building maps, we reached out to the company our navigation department typically worked with: HERE. It did not take us long to explain to them what we needed, though I remember we kept the full project scope hidden from them for quite a few weeks. Once we saw that they could deliver what we asked for, we disclosed what we had in mind and you could see something clicking. It became a very good collaboration and a very professional one as well because we managed to keep it a secret until our self-driving S-Class actually mastered the Bertha Benz Route all by itself and we could proudly announce it – to the surprise of quite a few in the industry – at the Frankfurt Motor Show 2013.

Once the idea of an HD map was created, our colleagues at the Karlsruhe Institute of Technology made us aware of another use of such a map. If we stored camera images of the route in the map, the localization of our vehicle on the road could be much improved over poor GPS positioning. Comparing geo-referenced image elements in the map to what our vehicle saw brought its localization accuracy down to a few decimeters.

So, our idea of an HD map consisted of two elements that we added to the road model known from conventional navigation maps: first, a lane model that showed exactly where lanes were (or were not or, for crying out loud, needed to be shared with oncoming traffic), and second a localization layer that provided the vehicle with visual cues to help it position itself on the road.

As part of the lane model, the map also included the position of stopping lines, crosswalks or traffic lights. We also used it to inform the vehicle on whether it needed to be more careful (i.e. drive slower) because it navigated an area with an unusually high likelihood of pedestrians or cyclists appearing on the road out of the blue. After all, our route went straight through downtown Heidelberg…

These two layers on top of the regular map of the road network still make up HERE’s high-definition map today. But, what was once a prototype of a mere 100 kilometers of road between Mannheim and Pforzheim, is soon going to be available for 1,000,000 kilometers throughout the world. It is no longer just a data file that gets stored in the vehicle, but it is a live stream of data that becomes available to the vehicle when and where it needs it, in its most up-to-date form.

This is why it is now called the HERE HD Live Map

And on top of this all, each element in the map, by a concept called Quality Index, carries an attribute that identifies how accurate and current it is. To keep it current, HERE developed a scheme where regular vehicles on the road contribute to keeping the map fresh. In 2018, it is all much more mature than the map used in the autonomous Bertha Benz Drive 5 years ago, though the concepts remain the same. What was research then, has turned into a real product now.

As for me, something else has changed. About a year ago, I made a career switch. Now, as HERE’s Head of Automotive I am, among other things, in charge of driving the product development of the HERE HD Live Map forward. I made the move from the vehicle side to the cloud side of autonomous driving because I think that we ain’t seen nothing yet in what a map database can do for self-driving vehicles. Going forward, I see it as the collective brain and ultimate driving school for all autonomous cars out there. But that’s for the future.

For now, seeing my old colleagues at Daimler firmly adopting the HERE HD Live Map for their future self-driving vehicles not only feels like raising our partnership to a higher level, it also means that it was worthwhile to start this collaboration in the first place. Thanks to everyone – on both sides – who took part in this journey.

For now, seeing the HERE HD Live Map powering self-driving vehicles feels like an evolution of what we started all those years ago, and reminds me how problems are rarely too hard solve, they only need better solutions.