Softability Services

High-Quality Test Data Combining Data Profiling and Generation

Kari Hakulinen | 2016/11/24

This article presents a novel idea of combining a data profiling tool with a data generator to generate large amounts of quality test data and to provide completely anonymized test data protecting sensitive customer data.

Introduction

Testing is an activity that produces and consumes large amounts of data. Test data itself is a wide concept. It ranges from specific input data used by developers testing a piece of code or realistic looking data with volume and variety matching production data needed by system testers. We shall look at how to cost-efficiently inflate a test database with meaningful data by using a set of tools.

This article presents a novel idea of combining a data profiling tool with a data generator to generate large amounts of quality test data and to provide completely anonymized test data protecting sensitive customer data.

Interested in reaping the benefits of this new method and pilot it with us? Then please contact Softability Sales.

Test Methodology

Data profiling has become a standard practice in various data management processes such as data warehousing. Nowadays, there are a plethora of commercial and open-source tools from various tool makers or tool and database vendors. Profiling tools have helped data managers to become familiar with a data source and identify problems that have to be fixed.

The most basic use and benefit of data profiling is column profiling. It provides statistics and analysis on the column values in form of computed metrics or graphs. Data profiling tool can easily show the patterns and distributions of values in a particular column. Modern data profiling tools are able to do more. The can analyze the inter-column and inter-table dependencies to discover e.g. violations of functional rules set by the tool user.

Data generator tools have also moved to mainstream and now you can pick up your favorite on-line data generator like mockaroo or generatedata.com. Or use a data mockup library available in many development environments. Modern tools can fill in the data fields with completely random values, values from a list or values according to a pattern or regular expression.

By working with both data profiling and generator tools we can quickly generate large amounts of test data based on our own specifications as follows:

  • Data profiling tool analyses the production data and provides metrics and column patterns
  • Data generator takes them as its specification and produces valid test data with variety

The idea appears to be simple yet powerful. Data profiling can provide knowledge of the real production data but does not reveal individual customer records. Data generator is able to understand patterns and rules and generate a data set closely mimicking the original production data. With the current data profiling and generators tools and with additional integration efforts their coordinated action is possible.

Benefits

Your production data does not need to leave the production environment or be anonymized. The anonymizing process or the code can have shortcomings and bugs and despite best efforts reveal sensitive customer data. It is best to avoid the whole process altogether and now it is possible. This idea should resonate well with your compliance department and management.

Profiling provides an added bonus of test engineers becoming more familiar with the source data. Instead of looking at a handful or records they can build up their understanding of the whole data. If you apply your custom rules you can identify records that violate certain rules or cross certain thresholds and use that knowledge to generate good test data.

If you are interested in reaping the benefits of this new method and pilot it with us please contact Softability Sales.

Keywords:

You might be interested:

Virtual reality is effective for providing induction to new employees and updating experienced employees’ knowledge

Nora Nirhamo | 2022/05/31

Softability created a virtual reality (VR) application for Lumon Oy to support employee induction and update experienced employees’ knowledge. The application is used to make induction cost-effective and to ensure consistent installation methods. Read more!

Read post

A HoloLens 2 smartglasses app supports learning and accelerates induction for new employees

Nora Nirhamo | 2022/05/11

Softability and the Savonia University of Applied Sciences implemented holographic work instructions for HoloLens 2 smartglasses using Microsoft Dynamics 365 Guides to support students’ transition to internships in working life and employee induction for new tasks. Read more!

Read post

Quality assurance – Testing is a key link in software development

Nora Nirhamo | 2022/04/07

Softability engages in high-quality software development in meaningful industries. Top-notch output always requires a team that comprises skilful coders and experienced software testers. The crucial task of software testers is to ensure the software works as intended – by all users and throughout its life cycle. Quality assurance is an integral part of the software development process and the final step before handing over the software to the end user.

Read post

Softability supports the victims of the conflict in Ukraine

Nora Nirhamo | 2022/03/15

Softability donates 2 % of last year’s profits to the Finnish Red Cross Disaster Relief Fund, which supports emergency activities in Ukraine.

Read post

Software developers tell about their employee experience at Softability

Nora Nirhamo | 2022/02/21

Employees have found their way to Softability through diverse career paths, but they all share the same enthusiasm to engage in challenging, meaningful software development. This article gives the floor to two software developers who carry out valuable work for our customers. Read more!

Read post

Softability is a Great Place to Work certified workplace

Nora Nirhamo | 2022/02/16

94% of our employees consider Softability a great place to work! As a token of this, Softability was, yet again, awarded the Great Place to Work certificate. We are proud of everything we achieve together – thank you, Softability employees!

Read post

XReach – developed by an expert team

Nora Nirhamo | 2022/02/01

The story of XReach started in 2017 with a customer’s request. A special team has been developing our remote support solution ever since. Today, the XReach remote support solution works with modern smart glasses, among other tools. Read the whole story!

Read post

AR and VR applications delivered to CTN Group for presenting the CTN eCabin cryo chamber

Nora Nirhamo | 2022/01/19

Softability provided CTN Group with a mobile augmented reality application and a virtual reality application to be used with Oculus Quest 2 VR glasses. The applications were made to facilitate presentation of the new CTN eCabin cryo chamber. Read more!

Read post

Software development that makes a difference in demanding projects – sound like your kind of thing?

Nora Nirhamo | 2022/01/13

Softability attracts developers who know what they want from their career; to be involved in coding meaningful software. As a small software agency, we care about the people who work for us. Read more!

Read post