HERE Probe Data: Geospatial Analysis and Visualization

Visualizing data is an essential part of conveying complex information in an easily understandable way. In this blog post, we'll show you how to use ArcGIS Pro to analyze and visualize HERE Probe Data. We've previously introduced HERE Probe Data and explained how to access and download it on the HERE platform using OLP Command Line Interface. Now, getting into the fun part - visualizing this data! We can analyze and visualize the probe data CSV files using various GIS applications. For this blog post, we've opted to use ArcGIS Pro. We'll provide a step-by-step guide on how to convert CSV files to feature classes, select areas of interest, create time-enabled layers, and use heatmaps and symbol rotation to visualize the data. We'll also share some best practices and examples to inspire you. So, let's get started!

CSV to feature class 

In the previous blog post, we have shown how to convert probe data protobuf into csv files, which has attributes such as latitude, longitude, speed, heading, timestamp, and traceID. In ArcGIS Pro, we added the csv to Contents pane. We can either right click the csv and select Display XY Data or use the Geoprocessing tool XY Table to Point. With the csv as input table, it auto-populates X Field as longitude and Y Field as latitude values from the probe data. In this example, we have a dataset of traffic probes spanning two days in Paris, organized into multiple CSV tables. To streamline the process, rather than manually handling each file individually, we developed a Python script to batch process all the CSV data, converting it into feature classes within a dedicated feature dataset.

        import arcpy 
import os 

# Set your workspace to the folder containing your CSV files 
arcpy.env.workspace = r"c:\data" 

# Path to the geodatabase where you want to store the feature classes 
output_gdb = r"C:\data\ParisProbeData\ParisProbeData.gdb" 

# List all CSV files in the workspace 
csv_files = arcpy.ListFiles("*-probes*.csv") 

# Loop through each CSV file and process 
for csv_file in csv_files: 
    input_csv = csv_file  # Input CSV file 
    x_field = "Longitude"  # Replace with your actual X (longitude) field name 
    y_field = "Latitude"   # Replace with your actual Y (latitude) field name 
    output_layer = f"TempLayer_{csv_file[:-4]}"  # Temporary layer name 
    coordinate_system = arcpy.SpatialReference(4326)  # WGS 1984 

    # Create the temporary point layer 
    arcpy.MakeXYEventLayer_management(input_csv, x_field, y_field, output_layer, coordinate_system) 

    # Extract the suffix from the input CSV file name (e.g., probe1 to probe55) 
    suffix = csv_file.split("-")[-1].replace(".csv", "") 

    # Create the output feature class name using the extracted suffix 
    output_feature_class = os.path.join(output_gdb, f"probe_{suffix}")  # Feature class name in the geodatabase 

    # Copy the temporary layer to the geodatabase as a feature class 
    arcpy.CopyFeatures_management(output_layer, output_feature_class) 

print("Batch processing complete.") 
  

Select Area of Interest (optional)

Time-enabled Layer

As each probe point captures a vehicle's location at a particular point in time, we used this information for configuring the feature class as a time-enabled layer. To achieve this, we converted the datatime string field from the csv/feature class into a new field of data type 'Date' and utilizing a Python 3 expression in the Field Calculator:

        import datetime 
def convert_to_date(date_string): 
    return datetime.datetime.strptime(date_string, "%Y-%m-%dT%H:%M:%SZ")  
  

Once the data was formatted correctly, the next step was enabling time for the feature class. To achieve this, we navigated Layer Properties, and selected the Time tab. On this tab, we selected 'Filter layer content based on attribute values', and linked it to the newly created 'Date' field. 

Enabling time for the layer added a new contextual tab called Time in the ribbon. Here, we defined the temporal properties for the map, including the current time extent, step interval, snapping and playback speed. 

Heat Map

Heat maps are a powerful tool for visualizing the density of data points in specific areas. Selecting the heatmap symbology in ArcGIS Pro through the Symbology pane is a straightforward process. Depending on your data's scale and specific aspects that you want to emphasize, you have the flexibility to choose a suitable color scheme and fine-tune the radius accordingly. 

By adding points of interest (POIs), we contextualized the cluster - why and where they occur - and gained a deeper understanding of our findings. For example, landmarks and major train stations emerged as hotspots, aligned with the high visitation rates. This also confirms that during peak hours, certain hotspots emerge, highlighting when commuters traverse the 'Périphérique' to travel between suburbs and work districts.

Symbol rotation by direction

HERE Probe Data provides not only location data but also heading (direction of travel) and speed data. To visualize this information, we used a different approach - a map that shows the direction of travel and associated speed as an arrow. To achieve this, we selected graduated colors in the Symbology pane to choose colors based on the speed at which the vehicle was moving. We rotated the arrow symbol by 90 degrees to point up since the default arrow points to the right. On the Vary symbology by attribute tab, we expanded the Rotation section and selected the heading field, which has the direction of travel from 0 to 360 degrees. We chose Geographic as the rotation style since the probe data rotation starts at zero in the north direction and rotates clockwise. To avoid cluttering the display with points with speed below 7, we added Transparency within the 'Vary symbology by attribute' settings. 

This visualization provides a good sense of how fast the vehicles were moving and in which direction. 

Conclusion

In this blog post, we provided a step-by-step guide on how to use ArcGIS Pro to visualize HERE Probe Data. We hope you found this helpful and are excited to try it out for your data and use case. You can also explore different ways of visualizing data, such as using charts, 3D visualization, or story maps. If QGIS is your tool of choice, here's a tutorial for enabling time. We encourage you to share your visualizations with us and the community, and we look forward to seeing what you build!