Skip to the main content.

Curiosity Modeller

Design Complex Systems, Create Visual Models, Collaborate on Requirements, Eradicate Bugs and Deliver Quality! 

Product Overview Solutions
Success Stories Integrations
Book a Demo Release Notes
Free Trial Brochure
Pricing  

Enterprise Test Data

Stream Complete and Compliant Test Data On-Demand, Removing Bottlenecks and Boosting Coverage!

Explore Curiosity's Solutions

Our innovative solutions help you deliver quality software earlier, and at less cost!

robot-excited copy-1              AI Accelerated Quality              Scalable AI accelerated test creation for improved quality and faster software delivery.

palette copy-1                      Test Case Design                Generate the smallest set of test cases needed to test complex systems.

database-arrow-right copy-3          Data Subsetting & Cloning      Extract the smallest data sets needed for referential integrity and coverage.

cloud-cog copy                  API Test Automation              Make complex API testing simple, using a visual approach to generate rigorous API tests.

plus-box-multiple copy-1         Synthetic Data Generation             Generate complete and compliant synthetic data on-demand for every scenario.

file-find copy-1                                     Data Allocation                  Automatically find and make data for every possible test, testing continuously and in parallel.

sitemap copy-1                Requirements Modelling          Model complex systems and requirements as complete flowcharts in-sprint.

lock copy-1                                 Data Masking                            Identify and mask sensitive information across databases and files.

database-sync copy-2                   Legacy TDM Replacement        Move to a modern test data solution with cutting-edge capabilities.

Explore Curiosity's Resources

See how we empower customer success, watch our latest webinars, read our newest eBooks and more.

video-vintage copy                                      Webinars                                Register for upcoming events, and watch our latest on-demand webinars.

radio copy                                   Podcasts                                  Listen to the latest episode of the Why Didn't You Test That? Podcast and more.

notebook copy                                           eBooks                                Download our latest research papers and solutions briefs.

calendar copy                                       Events                                          Join the Curiosity team in person or virtually at our upcoming events and conferences.

book-open-page-variant copy                                          Blog                                        Discover software quality trends and thought leadership brought to you by the Curiosity team.

face-agent copy                               Help & Support                            Find a solution, request expert support and contact Curiosity. 

bookmark-check copy                            Success Stories                            Learn how our customers found success with Curiosity's Modeller and Enterprise Test Data.

file-document-multiple (1) copy                                 Documentation                            Get started with the Curiosity Platform, discover our learning portal and find solutions. 

connection copy                                  Integrations                              Explore Modeller's wide range of connections and integrations.

Better Software, Faster Delivery!

Curiosity are your partners for designing and building complex systems in short sprints!

account-supervisor copy                            Meet Our Team                          Meet our team of world leading experts in software quality and test data.

calendar-month copy                                         Our History                                Explore Curiosity's long history of creating market-defining solutions and success.

check-decagram copy                                       Our Mission                                Discover how we aim to revolutionize the quality and speed of software delivery.

handshake copy                            Our Partners                            Learn about our partners and how we can help you solve your software delivery challenges.

account-tie-woman copy                                        Careers                                    Join our growing team of industry veterans, experts, innovators and specialists. 

typewriter copy                             Press Releases                          Read the latest Curiosity news and company updates.

bookmark-check copy                            Success Stories                          Learn how our customers found success with Curiosity's Modeller and Enterprise Test Data.

book-open-page-variant copy                                                  Blog                                                Discover software quality trends and thought leadership brought to you by the Curiosity team.

phone-classic copy                                      Contact Us                                           Get in touch with a Curiosity expert or leave us a message.

5 min read

Moving from GDPR compliance to full-blown Test Data Automation

Moving from GDPR compliance to full-blown Test Data Automation

I’ve written fairly frequently on the impact of the GDPR on testing, often responding to the news and research that continues to flow in. The below infographic summarises my thinking. It draws on news and research from 2019-2020 to show the need to address test data privacy issues today.*

Masking is not a complete test data solution: Moving from GDPR compliance to full-blown “Test Data Automation”

*As always, just my two cents on this. Hopefully an interesting opinion, but I’m no qualified legal expert and this is not legal advice.GDPR Compliance in testing Infographic

Figure 1 – The cost of non-compliance: test data privacy matters today.

The stats tell a pretty consistent story: the risk of a data breach continues to rise, as do the associated fines and brand damage. Meanwhile, the most effective way to mitigate this risk in testing is to limit the sharing of sensitive information to test environments.

I’ve listed the sources for the infographic below this article and I recommend that you take a look around to see just how high the stakes are for GDPR compliance. But first, let’s consider some ways in which GDPR compliance might impact testing and development, before considering what a good and bad response to this challenge looks like.

If the proposed approach sounds interesting, be sure to watch Curiosity's – Test Data Automation: Delivering Quality Data at Speed.

Watch now

The significance of the GDPR for testing and development

The impact of the GDPR on testing and development can be far-reaching. The particular issues that might need addressing in QA include:

  1. The logistics of fulfilling the “Right to Erasure” and “Right to Portability”: Organisations’ IT estates are often not set up to find and either copy or erase every instance of a person’s data “without delay”. Test environments are often sprawling and poorly understood, including legacy components and uncontrolled spreadsheets. Organisations today therefore rarely know reliably where sensitive information resides, and even fewer have the techniques to identify/copy/erase every instance of one person’s data “without delay”.

  2. Demonstrating legitimate grounds of processing data in QA: Structures are often lacking for demonstrating that each use of personal data in testing and development fulfils one of the grounds for its legitimate processing. These grounds include active and informed consent, legitimate business interest, and national security.

  3. Compliance with data minimization and limitation: Organisations must furthermore only share enough data with as many people as are required to fulfil the legitimate grounds for data processing, and those people must keep that data for only as long as required to fulfil that purpose. Organisations should be able to demonstrate this, but this is again a challenge when many organisations simply do not know what data resides where, nor how it is being used.

The challenge of test data compliance

It’s worth emphasising also that the maximum fines for non-compliance have increased substantially, and that national agencies have already shown willingness to impose them for Data Breaches.

Ultimately, the best defence against sensitive information leakage is to limit the number of people and places to which sensitive information is shared. QA environments are an avoidable place to which PII is routinely copied, and testing is therefore a good place to begin in the battle to ensure GDPR compliance.

Test data masking is a common technique deployed to reduce the risk of exposing sensitive information to less secure test environments. Anonymizing production data can soften the impact of compliance requirements in testing; however, it often also introduces significant bottlenecks in testing and development.

Masking data consistently from numerous sources is slow and complex as it must retain the complex relationships which exist within and across those data sources. Often, a central Ops team must work overtime to fulfil constant requests for data refreshes, but they can never deliver data fast enough to parallel test teams:Logistical approaches to test data undermine agility

Figure 2 – Logistical and linear approaches to test data provisioning undercut parallelism, speed, and agility.

Precious QA time is then wasted waiting idly for out-of-date copies of production, while tests fail due to misaligned or outdated data. Testers and automated tests further compete for the same data, leading to further delays as data is either constrained or has been used up.

Copying production data furthermore does nothing to improve the quality of production data.

The historical data sets by definition lack the data needed for testing unreleased functionality, while production activity focuses almost exclusively on “happy path” journeys through the system. Production data therefore lacks the outliers and negative scenarios needed for sufficient test coverage, leaving systems exposed to damaging defects in production:test data automation

Figure 3 – Production Data lacks the combinations needed for rigorous testing.

Moving to a complete solution

A complete test data solution must look beyond the removal of PII from test environments – it must also ensure that the data testers need is readily available when and where they need it. That means data combinations for each and every test, available in particular test environments at the speed demanded by automated test environments and iterative delivery.

The good news is that this approach, which Curiosity call “Test Data Automation”, builds on many of the skills and technologies used currently in Test Data Management. The following techniques can be added readily to existing masking and subsetting processes.

1. Synthetic test data generation

Generating combinations of data not found in production data plugs the gaps in test coverage stemming from low variety test data. Today, data generation can furthermore be readily integrated into data masking, adding in any missing combinations automatically as anonymized data moves to test environments.

This is a quick win for increasing testing quality while retaining test data compliance. It produces test data sets that do not contain sensitive production information, but do contain the outliers, edge cases and negative scenarios needed to test new or updated functionality.

2. On-the-fly test data allocation

Test data “allocation” can today replace the provisioning of full-size data copies, allocating exact data combinations based on test case definitions. This allocation is furthermore possible “just in time”, finding and making data combinations as tests are generated or executed.

In this approach, each test comes equipped with an up-to-date and unique data combination, cloned and allocated as it runs. This eliminates the delays caused by manual data provisioning and invalid test data, as well as the frustration of competing for shared data sources. Each test and each tester instead has their own isolated data set, available on demand and in parallel.

3. Standardized, re-usable TDM processes

To enable true QA agility, the dependency of testers on a data provisioning team must be removed. The silo between test data provisioning and test execution must be eliminated, providing on tap access to rich test data.

One way to achieve this is to make the processes used in data provisioning re-usable by test teams themselves.

Fortunately, these processes can be readily automated. They are by definition rule based, as they must reflect the relationships that exist in complex data. They can therefore be standardized and exposed to test teams on demand.

In this approach, test teams can parameterise and embed re-usable test data processes within their automated testing and CI/CD pipelines. Data is then found, made and prepared as the tests run, removing the dependency on an overworked Ops team.

In principle, any technique currently deployed by that Ops team can be exposed to test teams on demand. One approach creates self-service web portals to enable testers to re-use these data processes in their automated testing:test data automation

Figure 4 – Self-service web portals remove the upstream dependency on a central team.

See Test Data Automation in practice

Test Data Automation accordingly builds on the techniques used today to ensure test data compliance. It incorporates these processes in a holistic test data solution that also facilitates testing rigour and agility.

Sounds too good to be true?  Come and see Test Data Automation in action. Watch Curiosity's – Test Data Automation: Delivering Quality Data at Speed.

Watch now

References

[1] Danny Palmer (ZDNet: 20/01/2020), GDPR: 160,000 data breaches reported already, so expect the big fines to follow, Retrieved on 15/04/2020

[2] Luke Irwin (IT Governance: 09/03/2020), Infographic: Cyber Attacks and Data Breaches of 2019, retrieved on 15/04/2020/

[3] BBC (08/07/2019), British Airways faces record £183m fine for data breach, retrieved on 14/04/2020.

[4] British Airways faces record £183m fine for data breach

[5] British Airways faces record £183m fine for data breach

[6] Greg Sterling (Marketing Land: 04/12/2019), Nearly all consumers are concerned about personal data privacy, survey finds, retrieved on 15/04/2020.

[7] Thomas C. Redman and Robert M. Waitman (Harvard Business Review: 28/01/2020), Do You Care About Privacy as Much as Your Customers Do?, retrieved on 15/04/2020.

[8] Infographic: Cyber Attacks and Data reaches of 2019

[9] Infographic: Cyber Attacks and Data Breaches of 2019

[10 Rob Davies and Dominic Rushe (The Guardian: 14/07/2019), Facebook to pay $5bn fine as regulator settles Cambridge Analytica complaint, retrieved on 15/04/2020.

12 Differentiators of Test Data Automation

12 Differentiators of Test Data Automation

Are you aiming to modernise your test data strategy in 2024? Is test data holding back your software quality, privacy and delivery speed?

Read More
Quality Testing Requires Quality Data

Quality Testing Requires Quality Data

My two most recent blogs have made the case for a new TDM paradigm called “Test Data Automation”. The first article considered how a logistical...

Read More
Automated Test Data is Key to CI/CD and DevOps

Automated Test Data is Key to CI/CD and DevOps

Software delivery teams across the industry have embraced new(ish) approaches to development, from the different flavours of agile, to DevOps,...

Read More
4 Reasons for Data Migration Failures

4 Reasons for Data Migration Failures

Part two in this series identified some key risk factors associated with a data migration, which underpin the shocking migration failure rates...

Read More
The Test Data You Need, When and Where You Need it.

The Test Data You Need, When and Where You Need it.

Low level, repetitious tasks underpin all business activity. However, these labor-intensive activities are still frequently performed manually, to...

Read More
Is test data the engineering problem to solve in 2024?

Is test data the engineering problem to solve in 2024?

It’s 2024 and the risks associated with poor test data practices show no signs of abating.

Read More
If testing was a race, data would win every time

If testing was a race, data would win every time

Okay, so that title doesn’t make complete sense. However, if you read to the end of this article, all will become clear. I’m first going to discuss...

Read More
Removing Barriers to In-Sprint Testing and Development - Infographic

Removing Barriers to In-Sprint Testing and Development - Infographic

Test automation must be lightweight, re-usable and easy to apply, in order to help organisations, ease its implementation enterprise wide. Curiosity...

Read More
The Democratisation of (Test) Data

The Democratisation of (Test) Data

A glance at industry research from recent years shows that test data remains one of the major bottlenecks to fix in DevOps and CI/CD:

Read More