Write Index Data
- Last UpdatedDec 19, 2024
- 4 minute read
Write Index Layer Data
The Data Client Library provides the class LayerDataFrameWriter
, a custom Spark DataFrameWriter for writing DataFrames to index layers.
For index layers, data will be grouped by the values of the index attributes defined for the layer. If writing to a Protobuf-Encoded layer, there needs to be only one Row
for each set of index attributes.
Project Dependencies
If you want to create an application that uses the HERE platform Spark Connector to write data to index layer, add the required dependencies to your project as described in chapter Dependencies for Spark Connector.
Write process
For index layers, DataFrame
rows are grouped by indexes attributes to create the index data. The data will be uploaded into the index layer using the write engine and afterwards published using the Publish API. Also, DataFrame
can include additional metadata.
Metadata fields
All provided metadata columns for index layer:
Write Data as Avro, Parquet, or Protobuf-Encoded Files
The following snippet demonstrates how to write a DataFrame
as an Avro, Parquet, or Protobuf-Encoded data file to an index layer of a catalog:
Write Data in Csv Format
The following snippet demonstrates how to write a DataFrame
as a file with an arbitrary format. In this example, the input DataFrame
contains column field1
as integer, field2
as string.
Write Data in Text Format
The following snippet demonstrates how to write a DataFrame
as a file with an arbitrary format. In this example, the input DataFrame
contains a column data
with message as string.
Write Data in JSON Format
The following snippet demonstrates how to write a DataFrame
as a file with an arbitrary format. In this example, the input DataFrame
contains column intVal
as integer, strVal
as string.
Write Data in Other Formats
The following snippet demonstrates how to write a DataFrame
as a file with an arbitrary format to an index layer of a catalog. In this example, the input DataFrame
contains a column data
with messages as strings and the data of multiple rows is simply concatenated together:
Note
For information on RSQL, see RSQL.