Usability of data products - tips and best practices from everyday project work

In everyday life, however, inadequate structuring, preparation and quality of the data as well as unclear responsibilities have turned many a data lake into a data swamp. In this article, we have compiled some good practices to avoid exactly that.

What is a data product?

A data product is more than just a data set. It is an organization of responsibilities, processes, knowledge and technologies around a data set. This enables the Productive evolution of a data set over time. It makes the data set valuable in the long term.

For each data product there is a Data Product Owner (PO)who is responsible for ensuring that his product "works". To ensure this, he has several tasks: He is in contact with the users of the data product, as well as with the "producers" (development team, data engineers, data scientists) and the "suppliers" (source systems or specialist departments). This data PO is driven by the following questions, among others:

The data PO maintains its data product like a real productHe sells and explains it, collects suggestions for improvement from his customers, develops new product ideas, checks whether his product still generates "profit" - i.e. is used - and supports it throughout its product life cycle. Each data product can have its own team that is responsible for the development and operation of the product.

The number of data products on a platform can be reduced in this way. Scale more easilythan, for example, with a central data lake, where the operating team is in principle responsible for all data products created from the data lake.

It can Simple data products that are very close to the source data. But there are also Complex data products. They are composed of source data from several sources or even from other data products.

What makes a good data product?

A good data product is characterized by the fact that

• it quickly and easy to understand is.
• it Easy to use or difficult to use incorrectly.
• the database has a High data integrity i.e. the data is correct, complete and consistent.

The basic characteristics of a good data product are described by the author for software architecture Martin Fowler further with:

• Findability (~entry in the central data catalog),
• Addressability (~unique, unique ID),
• trustworthy and truthful content (~ Correct and consistent),
• (if possible) self-describing structure,
• Inter-operability (with other data products of the organization, e.g. through referencing by means of IDs; the data is at best available in a common standard and can be accessed in a standardized way).

What are good practices for good usability of data products?

We have identified the following good practices when working on various data-related problems as part of projects in the connected car, e-mobility, energy and enterprise resource planning (ERP) domains:

• Definition of a User group and their Intended use for a data product. This enables the derivation of...
  • Professional requirements:
    • Relevant information
    • Level of detail
    • Partitioning
    • Historicization
    • Actuality/update interval.
  • Technical requirements:
    • Storage solution
    • Data format
    • Access technology

 

• A complete and up-to-date Entry in the data catalog is mandatory. This serves as a "manual" or "operating instructions" for the use of the data product. The entry
  • includes FAQs to the data product.
  • clearly defines the meaning of NULL, "", 0, -1 and constants.
  • clearly defines the "unit" in which the data product is available (e.g. euro vs. cent, KM/H vs. meter/second).
  • shows from which Sources the composition of the data product and how it is prepared (data lineage).
  • provides information on which Aggregation level the data is available (e.g. per year, per month, etc.)
  • describes how the data changes (regular update, event-based update, etc.)
  • contains a reference to the data PO and the Data steward for further inquiries.
  • can be a Information classification according to ISO-27001 include (public, internal, confidential, strictly confidential)
  • contains a documentation of a Quantity scaffolding (number of data records and size per data record) as well as an outlook on the change in quantities.
  • may also include a "usage example", or Examples of use.

 

• Clear, unambiguous Data structures help to "understand" the data. They allow you to quickly gain an overview of the entire database and make it easier to find specific data records during a search. Structuring can be improved by the following aspects:
  • Use of meaningful entity and attribute namespreferably in English.
  • Use of flat data structuresIf necessary, denormalize ("specialist users do not think in normalized schemes").
  • Data partitioning (e.g. by time, alphabetical sorting, etc.) in order to limit the amount of data to be retrieved at a high level.
  • Data sortingThe data already has a predefined, unique sorting.
  • Delta update capabilityEach data record has information about the time of its creation and the last modification.
  • Clear Versioningcompatibility and migration paths of the schema (possibly in transition periods / in case of breaking changes parallel operation of several versions).

 

• The enrichment of a data product with Metadata (number of data records over time, number of accesses to the data, etc.) allows, for example, the tracking of usage, but also the growth of the data stock. This allows the effectiveness of change measures on the data product to be measured.

 

• Technical criteriawhich facilitate the use of the data are:
  • Documentation of the Access channels to the data product (REST API, ODBC database connection, etc.) with examples of use.
  • Provision of Test data setsReduced but as complete as possible test data sets should be provided. If possible, these should contain all "borderline cases of the data" (spikes, max, min, zero, etc.) as well as common patterns.
  • If necessary/useful: SLAs for the Availability of the data product or the infrastructure on which the data product is operated.

 

To ensure a good data user experience, a Central committee which can be used across all data products Standards defined and their implementation.

Example data product "Rated Customer Topics"

To better illustrate the good practices collected above, we have developed a fictitious example of a data product that contains and implements some of these criteria. All technologies and process models have been selected for this specific example and are therefore not transferable to other use cases.

This is the data product "Rated Customer Complaint Topics" of the fictitious Umbrella Corporation.

User group and intended use

In the first step, the technical requirements described:
Users of the "Rated Customer Complaint Topics" data product are Product Manager of Umbrella Corp.

• They use the data product - in this case the complaints list - to identify potential improvements to the product they are responsible for.
• These should serve as focus topics for research and development. Corresponding activities are intended to exploit the identified potential for improvement and thus solve the customers' problems.
• The information compression of the data product "Rated Customer Complaint Topics" should be very high, as only limited capacities are available in the area of research and development. For this reason, only a one to two-digit number of topics can be processed in parallel.
• To smooth outliers, the data product "Rated Customer Complaint Topics" is to be updated at one-month intervals.
• The data product "Rated Customer Complaint Topics" is to be historicized for ten years in order to identify and react to trends in the "resolution of customer problems" as well as recurring trends and developments.

In the next step, the technical requirements formulated:
Product managers are primarily professional level on the road.

• Therefore, one planned use of the data product is the easily accessible preparation in the form of a web dashboard.
• This dashboard should contain the top 10 customer complaint topics per month.
• In addition, it should offer the possibility of viewing detailed customer complaints by means of a drill-down per topic.
• A back and forth mechanism should make it possible to switch between consecutive and past months.

Entry in the data catalog

As already described in the article above, a complete and up-to-date Entry in the data catalog as a "manual" or "operating instructions" for the use of the data product.

• The data product "Rated Customer Complaint Topics" contains weighted topic areas for each product (=> In a real data catalog, there would be a link to the description of the individual subject areas here) for which customer complaints were received in the respective month. It is also possible to drill down to specific customer complaints via the subject areas. The weighting of the subject areas is based on the number of customer complaints recorded for this area.
• The data product is aggregated from individual specific customer complaints, which are submitted via the company website and telephone support (via the CRM system). Invalid entries are filtered out. A detailed derivation can be found in the section Finding data staging.
• The data product is created monthly and contains a monthly history of the last 10 years. It is updated on the 1st day of the following month.
• The data product does not contain any personal data, but information on product details and is therefore classified as "INTERNAL".
• As at 31.12.2020, the data stock of the data product amounted to 6,000 data records with a size of approx. 4KB per data record. In future, the data stock is expected to grow linearly by approx. 1200 data records per year. Due to the limitation of historization to 10 years, a maximum data stock of approx. 12,000 data records can be expected.
• The technical contact person for the data product (data PO) is Max Mustermann (=> Here would be a link to the contact details in a real data catalog). The technical contact for the data product (data steward) is Klaus Müller (=> Here would be a link to the contact details in a real data catalog).

Do you have questions about the data product? Please ask them directly using the contact form (=> in the real data catalog here would be a link to the contact form) to Max Mustermann. A list of questions that have already been asked and answered can be found in the forum of the data product (=> in the real data catalog, here would be a link to the data product).

Technical criteria

The following are examples of important technical criteria for the data product:

The data product can be accessed via ODBC. An access example is documented in the FAQs (=> in the real data catalog, here would be the link to the FAQs). Access can be requested electronically via the application portal at Daten-PO (=> in the real data catalog, here would be the link to the application portal). Access is subject to the general SLAs of the BigData infrastructure of Umbrella Corp (=> In the real data catalog, the link to the SLAs would be stored here).

The data product specifically contains the following entities and attributes:

Entity "Rated Customer Complaint Topics"

 

Attribute	Description	Identifier	Type	Unit	Format
topicName	Name of the subject area (=> Link to the description of the individual subject areas).	Yes	Text	Subject area	—
consideredPeriod	Period of observation; period in which the customer complaints from which the "numberOfComplaintsInTopic" attribute is calculated were received.	Yes	Text	month	yyyy-MM
productReference	Reference to the product to which the subject area refers (=> Link to the product directory)	Yes	ID	—	—
numberOfComplaintsInTopic	Number of customer complaints in the "consideredPeriod" period.	No	Integer	Quantity	x.xxx.xxx
detailedCustomerComplaints	Drill-down link to the detailed customer complaints that have been included in the subject area and the period under review. The link references a list of Detailed Customer Complaint IDs.	No	List	—	—
creationTimestamp	Time at which this data record was created in the Trusted Zone in the UTC time zone.	No	Timestamp	Date, time	ISO_8601 DataTime
lastUpdateTimestamp	Time of the last update of this data record in the Trusted Zone in the UTC time zone.	No	Timestamp	Date, time	ISO_8601 DataTime
schemaVersion	Version of the schema for "Rated Customer Complaint Topics".	No	Integer	—	—

 

The instances of the entity are partitioned according to "consideredPeriod". The partitions are sorted in descending order according to the latest period.
Within the partition, the instances of the entity are sorted by default according to "numberOfComplaintsInTopic" in descending order, "topicName" alphabetically".

Entity "Detailed Customer Complaint"

 

Attribute	Description	Identifier	Type	Unit	Format
detailedCustomerComplaintId	Sequential number to uniquely identify the customer complaint. The number is assigned during import into the Trusted Zone.	Yes	UUID	—	Timestamp-based_UUIDs
productReference	Reference to the product to which the subject area refers (=> Link to the product directory)	Yes	ID	—	—
complaintReceivedTimestamp	Timestamp of receipt of the customer complaint in the source system in the UTC time zone.	No	Timestamp	Date, time	ISO_8601 DateTime
complaintText	Text of the customer complaint in the language used by the customer, adjusted for control characters.	No	Text	—	Text in UTF8 encoding.
complaintCountryCode	Country code of the country in which the customer complaint was recorded.	No	Text	Country code	ISO_3166-1_alpha-2
creationTimestamp	Time at which this data record was created in the Trusted Zone in the UTC time zone.	No	Timestamp	Date, time	ISO_8601 DateTime
lastUpdateTimestamp	Time of the last update of this data record in the Trusted Zone in the UTC time zone.	No	Timestamp	Date, time	ISO_8601 DateTime
schemaVersion	Version of the schema for "Detailed Customer Complaint".	No	Integer	—	—

Data staging

Data staging shows how the data pipeline for the data product "Rated Customer Complaint Topics" is structured in detail. It therefore enables the user of the data product to understand which data sources the data product is made up of and how it is created in detail.

[A detailed description of the individual elements of the data pipeline for the data product could now be provided here.]

 

Using the example of "Rated Customer Complaint Topics", it was shown how an exemplary data product can be designed in such a way that a potential user can, if possible Pleasure in its use has. Only if this is the case will the data product and the data on which it is based be used. This use not only justifies the storage of the data and the associated costs, but also contributes to the fact that it is used by the profit. generate "value" from new findings. By following the good practices described in this article, the use of such data products can be simplified. This is because simpler use usually leads to increased and more intensive use. This in turn means that potentially greater value can be created. The good usability of data products should therefore be a priority for all those who use the Maximize the use of your data would like to.