Skip to content

Issue #8791 : Harden the Mask fields database mapping store #8793

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mattcasters merged 3 commits into from
Oct 9, 2026
Merged

Issue #8791 : Harden the Mask fields database mapping store #8793

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
72caa08
Issue #8791 : Harden the Mask fields database mapping store
bamaer Oct 7, 2026
076baa2
Address review: resolve manual URLs in the mapping store key, reject …
bamaer Oct 9, 2026
a484438
Initialize the Hop environment in MaskingRuntimeTest (#8791)
bamaer Oct 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion docs/hop-user-manual/modules/ROOT/pages/metadata-types/masking-pattern.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,12 +61,21 @@ image::metadata-types/masking-pattern-first-name.jpg[Masking pattern first-name,
|Remember values
|*NONE* assigns a new value every row. *MEMORY* keeps the mapping for one run. *DATABASE* keeps it in a table across runs.

|Ignore surrounding spaces
|Values that only differ in leading or trailing spaces get the same replacement. A value of only spaces is left unchanged. Used by *MEMORY* and *DATABASE*.

|Ignore case
|Values that only differ in upper and lower case get the same replacement. Used by *MEMORY* and *DATABASE*.

|Connection
|Database connection for *DATABASE*.

|Schema
|Schema of the mapping table. Empty uses the connection default.

|Table
|Mapping table. A second table with the suffix `_seq` stores the next sequence number. Both are created when they are missing. The map table stores the original value and is itself personal data.
|Mapping table. A second table with the suffix `_seq` stores the next sequence number. Both are created when they are missing. A new mapping table holds masked values of up to 255 characters.

|Hash secret
|The mapping table stores an HMAC-SHA256 of each value, made with this secret, instead of the value itself. Use a variable such as `${MASKING_SECRET}`. Changing the secret starts a new mapping. Without a secret the table stores the original values and is itself personal data.
|===
14 changes: 11 additions & 3 deletions docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/maskfields.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ image::transforms/maskfields/maskfields-dialog.png[Mask fields dialog,width="60%
|Name of a Masking pattern metadata item. Leave this empty to skip the field.
|===

When a field cannot be masked and the transform has error handling, the error row has that field and every masked field after it set to null.

== What a pattern writes

*SYNTHETIC* builds `prefix + token + suffix` for a string. The token is a *SEQUENCE* (an integer that starts at *Sequence start* and steps by one) or a *UUID*. A number field can take a sequence only, with an empty prefix and suffix. A date cannot take a generated token.
Expand All @@ -67,11 +69,17 @@ image::transforms/maskfields/maskfields-dialog.png[Mask fields dialog,width="60%

*MEMORY* keeps the mapping for this pipeline execution. The next run starts again, so tokens from the previous run can be issued again.

*DATABASE* keeps the mapping in the table named on the pattern. A second table, that name plus `_seq`, stores the next sequence number. Both tables are created when they are missing. The map table has `pattern_name`, `source_key` and `masked_value`, with a primary key on the pattern name and the source key.
*DATABASE* keeps the mapping in the table named on the pattern. A second table, that name plus `_seq`, stores the next sequence number. Both tables are created when they are missing. The map table has `pattern_name`, `source_key` and `masked_value`, with a primary key on the pattern name and the source key. A new map table also has a unique key on the pattern name and the masked value, and holds masked values of up to 255 characters.

Give a database pattern a *Hash secret*. The map table then stores an HMAC-SHA256 of each value, made with that secret, instead of the value itself. Without a secret, `source_key` holds the original value, so the table is itself personal data, and the transform check warns about it. Keep the secret in a variable, out of the project. Changing the secret starts a new mapping. A row stored in plain text before the pattern had a secret is moved to its hashed key the next time its value is masked. Rows for values that do not come by again stay in plain text, so delete those once the mapping has been rebuilt.

Several Hop processes can share one map table. Each new sequence number is reserved with a row lock, so two processes never hand out the same value.

*Ignore surrounding spaces* and *Ignore case* make values that only differ in those ways share a replacement. A number is compared by its value, not by its conversion mask. Turning these options on for a database pattern that already holds mappings changes replacements: spellings such as `Matt` and `matt` that had their own replacement now share the one stored for whichever spelling comes by first, so data masked earlier no longer matches for the others.

The map table stores the original value, so it is itself personal data. Put it on a connection you protect, separate from the masked output.
A hash secret that resolves to nothing, or still holds a variable that is not set, stops the transform from starting. It never falls back to plain text or to the literal text of the variable.

Copies and partitions of one pipeline share one runtime, so a remembered value stays the same and a sequence does not start over on each copy. Which row receives which new sequence number follows whichever copy runs first. Beam and native Spark each run a worker in its own JVM, so they stay unsupported. The database store is what shares a mapping across pipelines.
Copies and partitions of one pipeline share one runtime, so a remembered value stays the same and a sequence does not start over on each copy. Which row receives which new sequence number follows whichever copy runs first. Beam and native Spark each run a worker in its own JVM, so they stay unsupported. The database store is what shares a mapping across pipelines. Copies share a database store when its connection resolves to the same URL and user.

== Related

Expand Down
186 changes: 186 additions & 0 deletions integration-tests/database/0046-mask-fields-hashed-database.hpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,186 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--

Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

-->
<pipeline>
<info>
<name>0046-mask-fields-hashed-database</name>
<name_sync_with_filename>Y</name_sync_with_filename>
<description/>
<extended_description/>
<pipeline_version/>
<pipeline_type>Normal</pipeline_type>
<parameters>
<parameter>
<name>RESULT_TABLE</name>
<default_value>mask_0046_run1</default_value>
<description>Table the masked rows are written to</description>
</parameter>
</parameters>
<capture_transform_performance>N</capture_transform_performance>
<transform_performance_capturing_delay>1000</transform_performance_capturing_delay>
<transform_performance_capturing_size_limit>100</transform_performance_capturing_size_limit>
<created_user>-</created_user>
<created_date>2026/10/07 20:00:00.000</created_date>
<modified_user>-</modified_user>
<modified_date>2026/10/07 20:00:00.000</modified_date>
</info>
<notepads>
</notepads>
<order>
<hop>
<from>people</from>
<to>Mask fields</to>
<enabled>Y</enabled>
</hop>
<hop>
<from>Mask fields</from>
<to>result</to>
<enabled>Y</enabled>
</hop>
</order>
<transform>
<name>people</name>
<type>DataGrid</type>
<description/>
<distribute>Y</distribute>
<custom_distribution/>
<copies>1</copies>
<partitioning>
<method>none</method>
<schema_name/>
</partitioning>
<fields>
<field>
<currency/>
<decimal/>
<group/>
<name>id</name>
<type>Integer</type>
<format></format>
<length>-1</length>
<precision>-1</precision>
<set_empty_string>N</set_empty_string>
</field>
<field>
<currency/>
<decimal/>
<group/>
<name>firstName</name>
<type>String</type>
<format></format>
<length>-1</length>
<precision>-1</precision>
<set_empty_string>N</set_empty_string>
</field>
</fields>
<data>
<line>
<item>1</item>
<item>Matt</item>
</line>
<line>
<item>2</item>
<item>Ann</item>
</line>
<line>
<item>3</item>
<item>Bart</item>
</line>
<line>
<item>4</item>
<item>Matt</item>
</line>
</data>
<attributes/>
<GUI>
<xloc>96</xloc>
<yloc>96</yloc>
</GUI>
</transform>
<transform>
<name>Mask fields</name>
<type>MaskFields</type>
<description/>
<distribute>Y</distribute>
<custom_distribution/>
<copies>1</copies>
<partitioning>
<method>none</method>
<schema_name/>
</partitioning>
<fields>
<field>
<name>firstName</name>
<pattern>0046-first-name</pattern>
</field>
</fields>
<attributes/>
<GUI>
<xloc>272</xloc>
<yloc>96</yloc>
</GUI>
</transform>
<transform>
<name>result</name>
<type>TableOutput</type>
<description/>
<distribute>Y</distribute>
<custom_distribution/>
<copies>1</copies>
<partitioning>
<method>none</method>
<schema_name/>
</partitioning>
<commit>1000</commit>
<connection>unit-test-db</connection>
<fields>
<field>
<column_name>id</column_name>
<stream_name>id</stream_name>
</field>
<field>
<column_name>first_name</column_name>
<stream_name>firstName</stream_name>
</field>
</fields>
<return_field/>
<ignore_errors>N</ignore_errors>
<partitioning_daily>N</partitioning_daily>
<partitioning_enabled>N</partitioning_enabled>
<partitioning_field/>
<partitioning_monthly>Y</partitioning_monthly>
<return_keys>N</return_keys>
<schema>public</schema>
<specify_fields>Y</specify_fields>
<table>${RESULT_TABLE}</table>
<tablename_field/>
<tablename_in_field>N</tablename_in_field>
<tablename_in_table>Y</tablename_in_table>
<truncate>Y</truncate>
<use_batch>Y</use_batch>
<attributes/>
<GUI>
<xloc>448</xloc>
<yloc>96</yloc>
</GUI>
</transform>
<transform_error_handling>
</transform_error_handling>
<attributes/>
</pipeline>
Loading
Loading