Workspace & Marketplace 2.20 release notes

Highlights

New Location Services

We are happy to announce the availability of new services on platform.here.com. These are ready to use, but do note that charges will apply. All new HERE Location services are billed at a rate of $/€0.0005 per transaction except for Japan API transactions which are priced at $/€0.00065. See the transaction definition for each service in the chart below.

Please see the entire services suite at https://platform.here.com/services/

Exceptions:

• Fleet Telematics: 1 Transaction is counted for each individual Service feature included in a Request.
• Fleet Telematics Custom Routes: 1 Transaction is counted for each routing Request.
• Fleet Telematics ETA Tracking: 2 Transactions are counted for each ETA update Request.
• Fleet Telematics Custom Locations: One Transaction is counted for each location, points of interest, polygon or finding a POI along a route in a Request.

 

New SDKs

We are happy to announce the availability of new SDKs on platform.here.com. These are ready to use, but do note that charges will apply. These HERE SDKs services are billed based on transactions at a rate of $/€0.0005 per transaction except for Japan API transactions which are priced at $/€0.00065.

Please see the entire services suite at https://platform.here.com/sdk

 

Improved Platform Portal login

Org ID is no longer required when logging into the platform portal, except in cases where a users belongs to more than one organization.

It's also now possible for a pending user to request a new invitation in the login flow, eliminating the need for and org admin to re-invite users.

For users in multiple organizations, we've added "forgot your Org ID" functionality which emails users a list of the organizations to which they belong.

 

Change layer configurations after a layer is created to correct mistakes without needing to delete and recreate the layer

It is now possible to edit certain layer configurations after a layer has been created and even after the parent catalog as been marked as "Marketplace Ready". This change was made to ease certain changes, mostly to the advantage of developers when making configuration mistakes during layer creation or who want to change the configuration during CI/CD testing. It is important to note that making certain layer configuration changes after data is stored in the layer and/or after the parent catalog is available in Marketplace is very risky and can lead to irrecoverable impacts on data. Such changes should be made with great caution and understanding of the ramifications. Learn more here. The following configurations are now mutable via API, Data Client Library and CLI after a layer has been created:

• Stream layer configurations: Throughput, retention (TTL), content type, content encoding (compression) and schema association.
• Index layer configurations: Retention (TTL) and schema association
• Versioned layer configurations: Schema association

Volatile layer configuration changes as well as support for this functionality via the Portal will be delivered in a future release.

 

Decode and encode protobuf partitions with the CLI for more convenience during debugging and testing

You can now use the get and put commands for partitioned layers and for stream layers with a flag to have your JSON data encoded to protobuf prior to uploading it or decoded to JSON after downloading it. 

 

Popular testing frameworks integrated in a new data validation module

The Data Validation Library (DVL) has the purpose to enable scalable and efficient testing and validation of versioned catalogs on HERE Platform. In particular, testing of optimized maps and other generated versioned content should be implemented using DVL. Based on the Data Processing Library (DPL), DVL is supposed to abstract the underlying distributed processing on SPARK and Data API in order to facilitate quick creation of large-scale validation pipelines. However, to write tests, knowledge of DPL is required.

To lower the entrance barrier for new platform developers, we now follow a more lightweight approach by means of a new data validation module which integrates popular testing frameworks such as Cucumber, Junit or ScalaTest, allowing test engineers to write the tests without any previous knowledge of the DPL, concepts of partitioning or map compilation. This is achieved by separation of the test-data extraction phase from the test scenarios which are executed in parallel on each self-contained test-data partition. Only the test-extraction phase requires knowledge of DPL features and underlying catalog structure. Once created, it can be used by test engineers without platform experience for implementation of numerous tests. The new module pragmatically computes metrics inside the test scenarios and does the assessment based on a predicate on the aggregated metrics satisfying the vast majority of all testing use cases.

Please note that the above means that we are deprecating the DVL from the SDK for Java & Scala. The new data validation module in the DPL covers the functionality of the DVL.

 

 

 

Changes, additions and known issues

 

SDKs and tools

To see details of all changes to the CLI, Data SDKs for Python, TypeScript, C++, Java and Scala as well as to the Data Inspector Library, visit the HERE platform changelog

 

Web and portal

Known issue: Pipeline templates can't be deleted from the platform portal UI. 
Workaround: Use the CLI or API to delete pipeline templates.

Known issue: In the platform portal, new jobs and operations are not automatically added to the list of jobs and operations for a pipeline version when the list is open for viewing.
Workaround: Refresh the "Jobs" and "Operations" pages to see the latest job or operation in the list.

 

Projects and access management

Known issue: A set number of access tokens (~250) is available for each app or user. Depending on the number of resources included, this number may be smaller.
Workaround: Create a new app or user if you reach the limitation.

Known issue: A set number of permissions is allowed for each app or user in the system across all services. This will be reduced depending on the inclusion of resources and types of permissions.

Known issue: All users and apps in a group are granted permissions to perform all actions on any pipeline associated with that group. There's no support for users or apps with limited permissions. For example, you can't have a role that is limited to viewing pipeline statuses, but not starting and stopping a pipeline.
Workaround: Limit the uses in a pipeline group only to those who should have full control over the pipeline.

Known issue: When updating permissions, it can take up to an hour for the changes to take effect.

Known issue: Projects and all resources in a project are designed for use only in HERE Workspace, not the Marketplace. For example, a catalog created in a platform project can only be used in that project. It can't be marked as "Marketplace-ready" nor be listed in the Marketplace.
Workaround: Don't create catalogs in a project that are intended for use in both Workspace and Marketplace.

 

Data

Known issue: The "Upload data" button in your Layer UI under "More" is hidden when the "Content encoding" field in the layer is set to "gzip".
Workaround: Files (including .zip files) can still be uploaded and downloaded as long as the "Content encoding" field is set to "Uncompressed".

Known issue: The changes released with 2.9 (RoW) and with 2.10 (China) - for adding OrgIDs to catalog HRNs - and with 2.10 (Global) - for adding OrgIDs to schema HRNs - could impact any use case (CI/CD or other) where comparisons are made between HRNs used by various workflow dependencies.  For example, requests to compare HRNs that a pipeline is using with those to which a group, user or app has permissions will result in errors if the comparison is expecting results to match the old HRN construct. With this change, data APIs will return only the new HRN construct, which includes the OrgID, e.g. olp-here…, so a comparison between the old HRN and the new HRN will fail.   

• Reading from and writing to catalogs using old HRNs will continue to work until this functionality is deprecated (see deprecation notice summary).
• Referencing old schema HRNs will continue to work indefinitely.

Workaround: Update any workflows comparing HRNs to perform the comparison against the new HRN construct, including the OrgID.

Known issue: Searching for a schema in the platform portal using the old HRN construct returns only the latest version of the schema. The portal won't show older versions associated with the old HRN.
Workaround: Search for schemas using the new HRN construct, or look up older versions of schemas using the old HRN construct in the CLI.

Known issue: Visualization of index-layer data isn't supported yet.

 

Pipelines

Deprecation reminder: The batch-2.0.0 environment will soon be removed as its deprecation period has ended. It would be best to migrate your batch pipelines to the batch-2.1.0 run-time environment to benefit from the latest functionality and improvements.

Known issue: A pipeline failure or exception can sometimes take several minutes to respond.

Known issue: Pipelines can still be activated after a catalog is deleted.
Workaround: The pipeline will fail when it starts running and show an error message about the missing catalog. Find the missing catalog or use a different one.

Known issue: If several pipelines are consuming data from the same stream layer and belong to the same group (pipeline permissions are managed through a group), then each pipeline will only receive a subset of the messages from the stream. This is because, by default, the pipelines share the same application ID.
Workaround: Use the Data Client Library to configure your pipelines so they consume from a single stream. If your pipelines/apps use the Direct Kafka connector, you can specify a Kafka Consumer group ID per pipeline/application.  If the Kafka consumer group IDs are unique, the pipelines/apps can consume all the messages from the stream.
If your pipelines use the HTTP connector, create a new group for each pipeline/app, each with its own app ID.

 

Marketplace (Not available in China)

Known issue: There is no throttling for the beta version of the External Service Gateway. When the system is overloaded, the service slows down for everyone reading from the External Service Gateway.
Workaround: Contact HERE support for help.

Known issue: When the Technical Accounting component is busy, the server can lose usage metrics.
Workaround: If you suspect you're losing usage metrics, contact HERE support to get help with rerunning queries and validating data.

Known issue: Projects and all resources in a project are designed for use only in HERE Workspace and not available for use in HERE Marketplace. For example, a catalog created in a platform project can only be used in that project. It can't be marked as "Marketplace-ready" nor be listed in the Marketplace.
Workaround: Don't create catalogs in a project if intended only for use in the Marketplace.

 

Summary of active deprecation notices across all components