How to Access and Download HERE Probe Data to Analyze Traffic Patterns

HERE Probe Data is a location enrichment service for understanding traffic patterns and mobility trends. It consists of anonymous GPS traces collected from vehicles and mobile devices that use the HERE services. Last week, we discussed the significance of HERE Probe Data, how that data is anonymized, and what parameters are available in the dataset. In this blog post, we will explore how to access and analyze the HERE Probe Data. We will cover steps to download probe data and convert it to CSV for visualization and analysis in any GIS application.

Accessing Probe Data in HERE platform

Assuming that you have a HERE platform account and have requested your custom HERE Probe Data, the first step is to access the probe data catalog in your organization’s HERE platform account. We will access the Data section from Launcher and search for the name of our Probe data catalog. Once we find the catalog, we will copy the Catalog HRN. Within the catalog, you will see the Probe data layer, formatted in protobuf or protocol buffers format. We’ll also copy the Layer ID that we will need later to access the data using Command Line Interface (CLI). From the Schema tab of the probe data layer, we’ll download the schema as a ZIP archive. The schema is a formal definition that specifies the structure of the probe data and is useful for serializing and deserializing the data. The file is available with the .proto extension.

Configuring Access and Obtaining OAuth Credentials

The next step is to create an app and get credentials for accessing the probe data catalog using OLP CLI. From the Launcher, we’ll select Access Manager. We’ll create a new group or use an existing one. In the group, we’ll grant access to the probe data catalog by searching for it. This can be done without Group access too, but it's better to give data access to a group instead of individuals, as it makes it easier to manage access rights. We’ll create a new app or use an existing app, and add it to this Group too.

In the Apps tab, we’ll now go to our app. You will see an option to create credentials for your app. We’ll create OAuth 2.0 Credentials. Download the credentials – this is a credentials.properties file that contains your app ID, secret key, token endpoint URL and other information.

Downloading Probe Data using OLP CLI

To download the probe data, we’ll use the OLP Command Line Interface. OLP CLI allows you to access platform services from the command line, such as creating, retrieving, managing catalogs and layers, publishing data, managing permissions, and more. If you don’t already have OLP CLI installed, follow through these steps.

To download the probe data with OLP CLI, follow these steps:

Move the credentials.properties file that you downloaded in the previous step to the folder that you will use for data download.

Open a command line terminal and navigate to the folder where you have the credentials.properties file.
Import the credentials and create a new profile called probedataprofile that will have your app’s credentials, using the following command:

Copied

        olp --import-credentials --credentials-file credentials.properties --profile probedataprofile

Verify if you are able to get a token for your app:

Copied

        olp --profile probedataprofile token

If the command returns a token, it means you are successfully authenticated and authorized to access the platform resources. If not, check your credentials and try again.

Download the probe data using the Catalog HRN and Layer ID that you copied to your clipboard in the first step. To do this, run the following command:

Copied

        olp catalog layer get partition get <Catalog HRN> <Layer ID> [optional parameters]

Replace <Catalog HRN> and <Layer ID> with the actual values that you copied earlier. The optional paramteres can be using your credentials file directly, as --credentials credentials.properties or using the profile we just created --profile probedataprofile. We will also use --filter "timeWindow >= 0". The --output <path to dir> option specifies the directory where the downloaded data will be stored. More about catalog layer partition commands can be found in user guide.

This will take a few minutes to download all files, depending on the data. You will have a output folder with the name you specified, that contains protobuf files with probe data.

Converting Protobuf to CSV

To make the data readable with various tools and applications, we’ll convert the protobuf files to CSV format. We have done so using a simple Python script, as an example:

Copied

        import os, configparser
from pprint import pprint

import google.protobuf.json_format as json_format
import google.protobuf
from google.protobuf.json_format import MessageToDict
import glob
from com.here.hpd.schema.v1.probe_historic_data_pb2 import ProbeData

import pandas as pd

datadir = 'C:\Data\ParisProbe' ## update protobuf files location
fileno = 0
outputDir = 'outcsv'  ## update destination path
os.mkdir(outputDir)  ##destination path
...

for data in os.scandir(datadir):

    file = open(data.path, "rb")
    partition = ProbeData()
    partition.ParseFromString(file.read())

    probes = json_format.MessageToDict(partition) 

    df = pd.DataFrame(probes['probepoints'])
    ...

    fileno += 1
    ...
    df.to_csv(outputDir + probes['timewindow'] + "-probes{}.csv".format(str(fileno)))

...

The output CSV file has attributes such as latitude, longitude, speed, heading, timestamp, and traceID.

Conclusion

HERE Probe Data is a valuable source of information and can be used to accurately map and analyze congestion, historic traffic speeds and volumes, optimize traffic flow and city planning. Connect with us on our Slack channel, we're eager to know how you plan to use the HERE Probe Data.

In the next blog post, we will show you some of the many ways to visualize HERE Probe Data in GIS applications. Stay tuned for more tips!