Sensitive Data Protection's built-in infoType detectors are effective at finding common types of sensitive data. Custom infoType detectors enable you to fully customize your own sensitive data detector. Inspection rules help refine the scan results that the Sensitive Data Protection returns by modifying the detection mechanism of a given infoType detector.
If you want to exclude or include more values from the results that are returned by a built-in infoType detector, you can create a new custom infoType from scratch and define all the criteria that Sensitive Data Protection should look for. Alternatively, you can refine the findings that Sensitive Data Protection's built-in or custom detectors return according to criteria that you specify. You can do this by adding inspection rules that can help reduce noise, increase precision and recall, or adjust likelihood certainty of scan findings.
This topic discusses how to use the two types of inspection rules to either exclude certain findings or add additional findings, all according to custom criteria that you specify. Presented in this topic are several scenarios in which you might want to alter an existing infoType detector.
The following types of inspection rules are available:
- Exclusion rules, which help exclude false or unwanted findings.
- Hotword rules, which help detect additional findings in text content.
- Adjustment rules, which help adjust the likelihood of findings based on the context in which they appear.
Target and context infoTypes
This document uses the following terms to refer to infoTypes in a ruleset.
- Target infoType: an infoType that Sensitive Data Protection excludes or adjusts when the conditions defined in the ruleset are met.
- Context infoType: an infoType that, if detected, provides context about a target infoType. Sensitive Data Protection uses context infoTypes to determine whether it must exclude or adjust a target infoType.
Rule ordering and chaining
Sensitive Data Protection applies the rules in the order you specify them in the ruleset. Therefore, the order of your rules can affect the results of the Sensitive Data Protection operation. For an example, see Boost the likelihood of person name in a health document and exclude the health document in this document.
Exclusion rules
Exclusion rules are useful in situations like the following:
- You want to exclude duplicate scan matches in results that are caused by overlapping infoType detectors. For example, you're scanning for email addresses and phone numbers, but you are receiving two hits for email addresses with phone numbers in them, such as "206-555-0764@example.org."
- You're experiencing noise in your scan results. For example, you're seeing the same dummy email address (such as example@example.com") or domain (such as "example.com") returned an inordinate number of times by a scan for legitimate email addresses.
- You have a list of terms, phrases, or combination of characters that you want to exclude from results.
- You want to exclude an entire column of data from results.
- You want to exclude findings that are near a string that matches a regular expression.
- You want to exclude findings in images based on their spatial relationship with other detected findings in the image.
Exclusion rules API overview
Sensitive Data Protection defines an exclusion rule in the
ExclusionRule
object. Within ExclusionRule you specify one of the following:
- A
Dictionaryobject, which contains a list of strings to exclude from the results. - A
Regexobject, which defines a regular expression pattern. Strings that match the pattern are excluded from the results. - An
ExcludeInfoTypesobject, which contains an array of infoType detectors. If a finding is matched by any of the infoType detectors listed here, the finding is excluded from the results. An
ExcludeByHotwordobject, which contains the following:- A regular expression that defines the hotword.
- A proximity value that defines how near the hotword must be to the finding.
If the finding is within the set proximity, that finding is excluded from the results. For tables, this exclusion rule type lets you exclude an entire column of data from the results.
An
ExcludeByImageFindingsobject, which contains the following:A list of context infoTypes, which are used to determine whether to exclude a target infoType finding.
An
ImageContainmentTypeobject that specifies the required spatial relationship between the bounding boxes of the target and context findings. If this requirement isn't met, Sensitive Data Protection doesn't exclude the target infoType finding.
Sensitive Data Protection excludes a target infoType finding if the bounding box of a context infoType has the specified relationship with the target infoType finding. When using
ExcludeByImageFindings, you must set thematchingTypefield toMATCHING_TYPE_RULE_SPECIFIC.
Exclusion rule example scenarios
Each of the following JSON snippets illustrates how to configure Sensitive Data Protection for the given scenario.
Omit specific email address from EMAIL_ADDRESS detector scan
The following JSON snippet and code in several languages illustrates how to
indicate to Sensitive Data Protection using an
InspectConfig
that it should avoid matching on
"example@example.com" in a scan that uses the infoType detector EMAIL_ADDRESS:
C#
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Go
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Java
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Node.js
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
PHP
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Python
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
REST
See the JSON quickstart for more information about using the DLP API with JSON.
...
"inspectConfig":{
"infoTypes":[
{
"name":"EMAIL_ADDRESS"
}
],
"ruleSet":[
{
"infoTypes":[
{
"name":"EMAIL_ADDRESS"
}
],
"rules":[
{
"exclusionRule":{
"dictionary":{
"wordList":{
"words":[
"example@example.com"
]
}
},
"matchingType": "MATCHING_TYPE_FULL_MATCH"
}
}
]
}
]
}
...
Omit email addresses ending with a specific domain from EMAIL_ADDRESS detector scan
The following JSON snippet and code in several languages illustrates how to
indicate to Sensitive Data Protection using an
InspectConfig
that it should avoid matching on any email addresses that end with
"@example.com" in a scan that uses the infoType detector EMAIL_ADDRESS:
C#
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Go
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Java
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Node.js
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
PHP
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Python
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
REST
See the JSON quickstart for more information about using the DLP API with JSON.
...
"inspectConfig":{
"infoTypes":[
{
"name":"EMAIL_ADDRESS"
}
],
"ruleSet":[
{
"infoTypes":[
{
"name":"EMAIL_ADDRESS"
}
],
"rules":[
{
"exclusionRule":{
"regex":{
"pattern":".+@example.com"
},
"matchingType": "MATCHING_TYPE_FULL_MATCH"
}
}
]
}
]
}
...
Omit scan matches that include the substring "TEST"
The following JSON snippet and code in several languages illustrates how to
indicate to Sensitive Data Protection using an
InspectConfig
that it should exclude any findings that include the token "TEST" from the
specified list of infoTypes.
Note that this matches on "TEST" as a token, not a substring, so that although something like "TEST@email.com" will match, "TESTER@email.com" will not. If matching on a substring is desired, use a regex in the exclusion rule instead of a dictionary.
C#
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Go
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.
Java
To learn how to install and use the client library for Sensitive Data Protection, see Sensitive Data Protection client libraries.
To authenticate to Sensitive Data Protection, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.