HERE Workspace & Marketplace 2.16 release

Highlights

Reduced Stream Pipeline downtime during Upgrade and Restart

The downtime experienced during a Stream pipeline's Upgrade or Restart has been reduced to under 30 seconds, delivering a 50% improvement that is important for critical stream processing.

 

Use the improved GeoJSON plugin interface to combine and visualize data from multiple layers

The GeoJSON plugin interface now enables you to retrieve, combine and visualize data from multiple layers in different catalogs. For example, the Places MetaData layer contains a lookup table to translate category IDs used in the Places layer to category names. With this feature we can now show the category names in the properties rather than just the ID.

 

Use SBT to fetch schemas and the Data Client Library

So far, you could fetch platform dependencies only with the Maven build system. Java and Scala developers that prefer the Scala Build System (SBT) may now use our new SBT resolver plugin to fetch their project dependencies from the platform. Like the previously released Maven Wagon plugin, this SBT resolver manages the access to the platform via your platform credentials. The plugin is open sourced on GitHub and also published to Maven Central so it can be fetched as part of your build process. Any contribution to both these GitHub projects is welcome. Please note, this SBT resolver is provided 'as is' and not officially part of Workspace or Marketplace. That means the official support that you may have licensed does not apply to this SBT resolver.

 

Access additional HERE Map Content: ADAS Attributes via the Optimized Map for Location Library

The following ADAS attributes from the HERE Map Content catalog are now available: Slope, BuildUpArea, LinkAccuracy and CurvatureHeading. Note: "Elevation" is targeted to be added to OMLL in the next platform release (v2.17). We will continue to iteratively compile HERE Map Content attributes into the Optimized Map for Location Library enabling fast and simplified direct access to these attributes via the Location Library.  For more detail on the added attributes, please see: HERE Map Content- ADAS Attributes layer

 

Changes, Additions and Known Issues

 

SDKs and tools

Go to the HERE platform changelog to see all detailed changes to our CLI, the SDKs for Python, TypeScript, C++, Java and Scala as well as to the Data Inspector Library.

 

Web & Portal

Issue: The custom run-time configuration for a Pipeline Version has a limit of 64 characters for the property name and 255 characters for the value.
Workaround: For the property name, you can define a shorter name in the configuration and map that to the actual, longer name within the pipeline code. For the property value, you must stay within the limitation.

Issue: Pipeline Templates can't be deleted from the Portal UI.
Workaround: Use the CLI or API to delete Pipeline Templates.

Issue: In the Portal, new jobs and operations are not automatically added to the list of jobs and operations for a pipeline version while the list is open for viewing.
Workaround: Refresh the Jobs and Operations pages to see the latest job or operation in the list.

 

Projects & Access Management

Issue: A finite number of access tokens (~ 250) are available for each app or user. Depending on the number of resources included, this number may be smaller.
Workaround: Create a new app or user if you reach the limitation.

Issue: Only a finite number of permissions are allowed for each app or user in the system across all services. It will be reduced depending on the inclusion of resources and types of permissions.

Issue: All users and apps in a group are granted permissions to perform all actions on any pipeline associated with that group. There is no support for users or apps with limited permissions. For example, you cannot have a reduced role that can only view pipeline status, but not start and stop a pipeline.
Workaround: Limit the users in a pipeline's group to only those users who should have full control over the pipeline.

Issue: When updating permissions, it can take up to an hour for changes to take effect.

Issue: Projects and all resources in a Project are designed for use only in Workspace and are unavailable for use in Marketplace. For example, a catalog created in a Platform Project can only be used in that Project. It cannot be marked as "Marketplace ready" and cannot be listed in the Marketplace.
Workaround: Do not create catalogs in a Project when they are intended for use in both Workspace and Marketplace.

 

Data

Issue: The changes released with 2.9 (RoW) and with 2.10 (China) to add OrgID to Catalog HRNs and with 2.10 (Global) to add OrgID to Schema HRNs could impact any use case (CI/CD or other) where comparisons are performed between HRNs used by various workflow dependencies.  For example, requests to compare HRNs that a pipeline is using vs what a Group, User or App has permissions to will  result in errors if the comparison is expecting results to match the old HRN construct.  With this change, Data APIs will return only the new HRN construct which includes the OrgID (e.g. olp-here…) so a comparison between the old HRN and the new HRN will be unsuccessful.   

• Reading from and writing to Catalogs using old HRNs is not broken and will continue to work until July 31, 2020.
• Referencing old Schema HRNs is not broken and will work into perpetuity.

Workaround: Update any workflows comparing HRNs to perform the comparison against the new HRN construct, including OrgID.

Issue: Versions of the Data Client Library prior to 2.9 did not compress or decompress data correctly per configurations set in Stream layers. We changed this behavior in 2.9 to strictly adhere to the compression setting in the Stream layer configuration but when doing so, we broke backward compatibility wherein data ingested and consumed via different Data Client Library versions will likely fail. The Data Client LIbrary will throw an exception and, depending upon how your application handles this exception, could lead to an application crash or downstream processing failure. This adverse behavior is due to inconsistent compression and decompression of the data driven by the different Data Client Library versions. 2.10 introduces more tolerant behavior which correctly detects if stream data is compressed and handles it correctly.

Workaround: In the case where you are using compressed Stream layers and streaming messages smaller than 2MB, use the 2.8 SDK until you have confirmed that all of your customers are using at least the 2.10 SDK where this Data Client Library issue is resolved, then upgrade to the 2.10 version for the writing aspects of your workflow.

Issue: Searching for a schema in the Portal using the old HRN construct will return only the latest version of the schema.  The Portal will not show older versions tied to the old HRN.

Workaround: Search for schemas using the new HRN construct OR lookup older versions of schemas by old HRN construct using the OLP CLI.

Issue: Visualization of Index layer data is not yet supported

 

Pipelines

Changed: 

Issue: A pipeline failure or exception can sometimes take several minutes to respond.

Issue: Pipelines can still be activated after a catalog is deleted.
Workaround: The pipeline will fail when it starts running and will show an error message about the missing catalog. Re-check the missing catalog or use a different catalog.

Issue: If several pipelines are consuming data from the same Stream layer and belong to the same Group (pipeline permissions are managed via a Group), then each of those pipelines will only receive a subset of the messages from the stream. This is because, by default, the pipelines share the same Application ID.
Workaround: Use the Data Client Library to configure your pipelines to consume from a single stream: If your pipelines/applications use the Direct Kafka connector, you can specify a Kafka Consumer Group ID per pipeline/application.  If the Kafka consumer group IDs are unique, the pipelines/applications will be able to consume all the messages from the stream.
If your pipelines use the HTTP connector, we recommend you to create a new Group for each pipeline/application, each with its own Application ID.

Issue: The Pipeline Status Dashboard in Grafana can be edited by users. Any changes made by the user will be lost when updates are published in future releases because users will not be able to edit the dashboard in a future release.
Workaround: Duplicate the dashboard or create a new dashboard.

Issue: For Stream pipeline versions running with the high-availability mode, in a rare scenario, the selection of the primary Job Manager fails.
Workaround: Restart the stream pipeline.

 

HERE Content

Added: additions to existing layers in the HERE Map Content (Japan) catalog:

• Added Postal Code Boundaries content to Postal Area Boundaries layer
• Added Postal Code Points content to Postal Code Points layer
• Added Carto Roads content to the Cartography layer
• Added Truck Service Locations content to the Truck Service Locations layer
• Added Fueling Stations content to the Fueling Stations layer

 

Location Services

Issue: Lack of usage reporting for Location Services released in version 2.10 (Routing, Search, Transit, and Vector Tiles Service)
Workaround: Usage is being tracked at the service level.  Following the 2.12 release wherein usage reporting is expected to be in place, customers may request usage summaries for usage incurred between the 2.10 release and 2.12.

 

Marketplace (Not available in China)

 

Issue: There is no throttling for the beta version of the External Service Gateway. When the system is overloaded, service will slow down across the board for all consumers who are reading from the External Service Gateway.

Workaround: Contact technical support for help.

Issue: Users do not receive stream data usage metrics when reading or writing data from Kafka Direct.
Workaround: When writing data into a Stream layer, you must use the Ingest API to receive usage metrics. When reading data, you must use the Data Client Library, configured to use the HTTP connector type, to receive usage metrics and read data from a Stream layer.

Issue: When the Technical Accounting component is busy, the server can lose usage metrics.
Workaround: If you suspect you are losing usage metrics, contact HERE technical support for assistance rerunning queries and validating data.

Issue: Projects and all resources in a Project are designed for use only in Workspace and are unavailable for use in Marketplace. For example, a catalog created in a Platform Project can only be used in that Project. It cannot be marked as "Marketplace ready" and cannot be listed in the Marketplace.
Workaround: Do not create catalogs in a Project when they are intended for use in the Marketplace.

 

Summary of active deprecation notices across all components

No.	Feature Summary	Deprecation Period Announced (Platform Release)	Deprecation Period Announced (Month)			Deprecation Period End
1	OrgID added to Catalog HRN (RoW)	2.9 (ROW) 2.10 (China)	November 2019			July 31, 2020
	Deprecation Summary:	Catalog HRNs without OrgID will no longer be supported in any way after July 31, 2020. ·       Referencing catalogs and all other interactions with REST APIs using the old HRN format without OrgID OR by CatalogID will stop working after July 31, 2020, ·       Please ensure all HRN references in your code are updated to use Catalog HRNs with OrgID before July 31, 2020 so your workflows continue to work. ·       HRN duplication to ensure backward compatibility of Catalog version dependencies resolution will no longer be supported after July 31, 2020. ·       Examples of old and new Catalog HRN formats: ·       Old (without OrgID/realm): hrn:here:data:::my-catalogNew (with OrgID/realm): hrn:here:data::OrgID:my-catalog
2	OrgID added to Schema HRN (Global)	2.10	December 2019			July 31, 2020
	Deprecation Summary:	References to pre-existing schemas created before this feature release will work with either new or old HRNs so those references are not impacted. New schemas created as of this feature release will support both HRN constructs for a period of (6) months or until June 30, 2020. After this time, only the new HRN construct will be supported.
3	Data Client Library data compression behavior	2.10	December 2019			June 20, 2020
	Deprecation Summary:	As of June 30, 2020, all data streaming related workflows must be updated to use at least SDK version 2.9.  The Data Client Library behavior as it relates to compressing and decompressing streaming data will strictly adhere to stream layer configuration settings.
4	Spark-ds-connector replaced by SDK for Java and Scala Spark Connector	2.12	February 2020			August 19, 2020
	Deprecation Summary:	The spark-ds-connector will be deprecated (6) months from this release on August 19, 2020. Please upgrade to the latest SDK for Python version before then to get the latest SDK for Java and Scala Spark Connector.
5	Batch-2.0.0 run-time environment for Pipelines	2.12	February 2020			August 19, 2020
	Deprecation Summary:	Batch-2.0.0 run-time environment for Batch pipelines is now deprecated. Existing Batch pipelines that use the Batch-2.0.0 run-time environment will continue to operate normally until August 19, 2020. During this period, Batch-2.0.0 run-time environment will receive security patches only. For this period, to continue developing pipelines with the Batch-2.0.0 environment, please use OLP SDK 2.11 or older. After August 19, 2020 we will remove the Batch-2.0.0 run-time environment and the pipelines still using it will be canceled. We recommend that you migrate your Batch Pipelines to the Batch-2.1.0 run-time environment to utilize the latest functionality and improvements.
6	Schema validation to be added	2.13	March 2020			September 30, 2020
	Deprecation Summary:	For security reasons, the platform will start validating schema reference changes in layer configurations as of September 30, 2020. Schema validation will check if the user or application trying to make a layer configuration change indeed has at least read access to the existing schema associated with that layer (i.e. a user or application cannot reference or use a schema they do not have access to). If a non-existing or non-accessible schema is associated with any layer after this date, any attempt to update any configurations of that layer will fail. Please ensure all layers refer only to real, existing schemas, or contain no schema reference at all before September 30, 2020. It is possible to use the Config API to remove or altogether change schemas associated with layers to resolve these invalid schema/layer associations. Also, any CI/CD jobs referencing non-existing or non-accessible schemas will need to be updated by this date or they will fail.
7	Customizable Volatile layer storage capacity and redundancy configurations	2.14		April 2020	October 30, 2020
	Deprecation Summary:	The Volatile layer configuration option to set storage capacity as a "Package Type" will be deprecated within (6) months or by October 30, 2020. All customers should deprecate their existing volatile layers and create new volatile layers with these new configurations within (6) months of this feature release or by October 30, 2020.