SAP data scrambling and masking: methods compared

Anonymisation, pseudonymisation, masking, scrambling

No legal advice. This article describes technical methods. Whether a measure is sufficient in your case, and whether the result is anonymous or pseudonymous, is for your data protection officer and legal advisers to assess.

Why production data in test systems is a data protection issue is explained in the article Personal data in SAP test systems. This article is about implementation. First, the terms, because they are often confused:

Anonymisation: According to Recital 26 GDPR, anonymous information does not, or no longer, relate to an identifiable person. The yardstick is the means reasonably likely to be used, judged by factors such as cost, time and available technology. Data protection principles do not apply to anonymous information.
Pseudonymisation: Under Article 4(5) GDPR, the data can no longer be attributed to a specific person without additional information, such as a mapping table, which is kept separately and protected. Pseudonymised data remains personal data.
Masking: An umbrella term, not a legal one, for methods that change or hide values. Whether the result is anonymous or pseudonymous depends on the method and on the data as a whole.
Scrambling: The usual term in the SAP world, for example in SAP TDMS. Data is altered before testers and developers see it in a non-production system. It means the same as masking.

The Article 29 Working Party, a former EU advisory body, tests anonymisation techniques against three questions: can a person still be singled out, can records relating to them be linked, and can information about them be inferred? On this view, pseudonymisation is not anonymisation. It only makes it harder to link a dataset to a person’s real identity.

SAP data masking: six methods compared

Data scrambling tools combine a few basic techniques. SAP TDMS, for example, offers scrambling types such as delete value, fixed value, manual 1:1 mapping, number conversion and random values from lists or time ranges. Six methods can be distinguished:

Method	How it works	Strength	Weakness
Substitution	The real value is replaced with a fictitious one, from a list of values or by a rule.	Realistic data, formats and lengths are retained.	Value lists need maintaining. A stored mapping from old to new is reversible.
Shuffling	Values are swapped between records within a column.	Distribution and value range stay exactly the same.	Every value remains genuine. Fields shuffled on their own no longer match related fields.
Character masking	Parts of a value are replaced with placeholders, for example all but the last four characters.	Easy to implement, values remain distinguishable.	The remaining characters can identify someone. Check digits and format checks fail.
Date shifting	Dates are moved by a fixed or random period.	With the same offset per person or transaction, intervals and sequences are retained.	A fixed offset can be calculated back once one real date is known. Random individual values break sequences.
Generation	Values are created artificially, such as names, addresses or IBANs with a valid check digit.	No link to the original value.	Time-consuming: formats, country rules and dependencies have to be right.
Deleting or blanking	The field is cleared or set to a fixed value.	Simple and thorough for fields that no test needs.	Unsuitable for key and mandatory fields.

A simplified classification. Which method fits depends on the field, the purpose of the test and the requirements of your data protection team.

Two points determine the quality. First: random or deterministic. Deterministic means that the same original value always gets the same replacement, even in the next refresh and in other systems, which interface tests often need. If a mapping table is kept for this, it matches the description of pseudonymisation. SAP’s data protection notes on TDMS explicitly classify manual 1:1 mapping as pseudonymisation.

Second: no method is enough on its own. The Article 29 Working Party notes that shuffling, for instance, does not anonymise by itself and should be combined with removing obvious attributes. Related attributes should be shuffled together, otherwise the shuffling may be reversed.

What makes anonymising SAP data difficult

The methods are quickly explained. In SAP, the work lies in the details:

Consistency: A business partner appears in master data, addresses, documents and payment data, and every occurrence needs the same replacement. SAP’s TDMS guide shows this with the address table ADRC: name, street and city are replaced consistently using the address number as the key, and via a global mapping across several rules. The same applies across systems: if S/4HANA and the connected CRM or BW system replace a customer differently, interface tests no longer match.
Plausible replacement values: A replacement address must match the country, region and postcode. SAP’s TDMS guide illustrates this with country and region: if the country changes, the region has to change with it.
Search helps and matchcodes: SAP stores some values twice so that searching works. In the customer master, for example, name and city are also held in upper-case substitute fields: MCOD1 for NAME1 and MCOD3 for ORT01. Replace only the name field and the real name stays in the search field. Freely maintained search terms often contain names too.
Free text and attachments: Field rules do not capture long texts, notes or attached PDFs. Usually the only option is to blank them or not copy them at all. See Limits of masking.
Check digits: The IBAN contains two check digits. A randomly altered value is almost always invalid, so checks in master data maintenance, payments or interfaces may fail. Replacement IBANs need a recalculated check digit.
Testability: SAP itself describes the conflict in its TDMS data protection notes: depending on the type of anonymisation, for example of an address, follow-on processes may stop working because logical relationships are lost. Conversely, anonymisation fails if the link to a person can still be identified.

The rule of thumb: retain formats, keys and relationships, and change the content that identifies a person. For employee data from SAP HCM, first ask whether it belongs in the test system at all.

When to mask: during the copy or afterwards?

Besides the how, timing matters. There are basically two options:

After the copy, in the target system: The system or client copy runs as usual, then a program changes the data. That fits existing procedures, but until the run finishes the test system holds plain text, possibly in logs and backups too. With large tables, the run also eats into the downtime window.
During the copy: The data is changed during export in the source or during import, before it is written to the target database, so the plain text never reaches the test system. SAP’s data protection notes on TDMS recommend scrambling the data before it is transferred to the non-production system. The system copy and the standard client copy offer no function for this.

If you mask afterwards, keep the target system closed until masking is complete: users locked, interfaces and jobs stopped. Masking then belongs on the checklist of follow-up tasks after a client copy as a separate item, before the system is released to testers.

Mask after the copy

ProductionPersonal data in clear text
System or client copyRuns as usual.
Test system in clear textUntil the masking run has finished, possibly also in logs and backups.Clear text in the test system
Masking runTakes time in the downtime window for large tables.
Test system masked

Mask during the copy

ProductionPersonal data in clear text
Masking during export or importBefore the data is written to the target database.
Test system maskedNo clear text in the test system

If you mask after the copy, the test system holds clear text until the run is complete. Masked during the copy, clear text never reaches it. System copies and the standard client copy offer no function for masking during the copy.

Not the same: masking in the user interface

Not every kind of masking changes data. UI Data Protection Masking for SAP S/4HANA protects sensitive field values in the user interface: users without authorisation see the value masked, empty or not at all, while authorised users see the original. According to SAP’s documentation, this can be set up for SAP GUI, Web Dynpro ABAP and SAP Fiori, among others, controlled by roles or attributes.

UI masking therefore controls who sees a value. The original value remains stored, in the database and so also in backups and copies. Scrambling, by contrast, changes what is stored. For a test system filled from production, one does not replace the other.

Anonymising SAP test data in five steps

Find the data. Record which tables and fields contain personal data, including enhancements, customer-specific tables, search fields, long texts and attachments. The business departments know the special cases. The result is a field catalogue.
Define the rules. Choose a method per field: delete what no test needs, replace what identifies a person, and keep formats, check digits and keys. Decide which values must be replaced identically across tables and systems, and agree the result with your data protection team.
Test. Try the rules on a small dataset first, for example in a separate test client. Do the key processes still work, including interfaces and forms? Does plain text still appear anywhere, for example in search fields, change documents or IDocs?
Make it repeatable. Masking belongs in every refresh. Automate the run, version the rules and add new fields from enhancements and upgrades. The Article 29 Working Party also stresses that anonymisation is not a one-off exercise and that risks should be reassessed regularly.
Document. Record the field catalogue, rules, approvals and every run. Anyone who knows the rules also knows how the data was changed, which is why SAP’s data protection notes on TDMS name an authorisation concept and non-disclosure agreements for everyone involved.

Often the most effective masking is not to copy the data at all. The alternatives, from a time-slice partial copy to synthetic data, are compared in the article Test data for S/4HANA.

Masking in the refresh with Data Protect

The Data Refresh Operator (DRO) addresses the question of timing. It copies a time slice instead of the complete history, for example the last 90 days plus all dependent objects, and integrates masking with Data Protect into the run:

Data Protect helps you find data that needs masking.
You maintain the rules in one place, in the central control system.
They are applied during export or import. When masking during export, the data leaves production already masked.
Masking is a step in the run and can be scheduled in playbooks. A simulation in advance and the audit trail make it traceable what was copied.

Which fields are masked and how, and whether the result is sufficient for your purpose, is something you define with your data protection team. DRO provides the tool, the assessment remains with you. DRO runs on SAP S/4HANA from release 2023 and replaces the client copy, not the system copy. More about the function on the page Anonymising and masking SAP test data with Data Protect. In a demo, we show you Data Protect live and discuss your requirements for test data.

SAP data scrambling and masking: anonymising test data