Creating a regular custom dictionary detector

Custom dictionaries provide the simple but powerful ability to match a list of words or phrases. You can use a custom dictionary as a detector or as an exception list for built-in detectors. You can also use custom dictionaries to augment built-in infoType detectors to match additional findings.

This section describes how to create a regular custom dictionary detector from a list of words.

Anatomy of a dictionary custom infoType detector

As summarized in API overview, to create a dictionary custom infoType detector, you define a CustomInfoType object that contains the following:

• The name you want to give the custom infoType detector, within in an InfoType object.
• An optional Likelihood value. If you omit this field, matches to the dictionary items will return a default likelihood of VERY_LIKELY.
• Optional DetectionRule objects, or hotword rules. These rules adjust the likelihood of findings within a given proximity of specified hotwords. Learn more about hotword rules in Customizing match likelihood.

• An optional SensitivityScore value. If you omit this field, matches to the dictionary items will return a default sensitivity level of HIGH.

  Sensitivity scores are used in data profiles. When profiling your data, Sensitive Data Protection uses the sensitivity scores of the infoTypes to calculate the sensitivity level.

• A Dictionary, as either a WordList containing a list of words to scan for or a CloudStoragePath to a single text file containing a newline-delimited list of words to scan for.

As a JSON object, a dictionary custom infoType detector that includes all optional components looks like the following. This JSON includes a path to a dictionary text file stored in Cloud Storage. To see an inline word list, see the Examples section, later in this topic.

{
  "customInfoTypes":[
    {
      "infoType":{
        "name":"CUSTOM_INFOTYPE_NAME"
      },
      "likelihood":"LIKELIHOOD_LEVEL",
      "detectionRules":[
        {
          "hotwordRule":{
            HOTWORD_RULE
          }
        },
        ...
      ],
      "sensitivityScore":{
          "score": "SENSITIVITY_SCORE"
        },
      "dictionary":
      {
        "cloudStoragePath":
        {
          "path": "gs://PATH_TO_TXT_FILE"
        }
      }
    }
  ],
  ...
}

Dictionary matching specifics

Following is guidance about how Sensitive Data Protection matches dictionary words and phrases. These points apply to both regular and large custom dictionaries:

• Dictionary words are case-insensitive. If your dictionary includes Abby, it will match on abby, ABBY, Abby, and so on.
• All characters—in dictionaries or in content to be scanned—other than letters, digits, and other alphabetic characters contained within the Unicode Basic Multilingual Plane are considered as whitespace when scanning for matches. If your dictionary scans for Abby Abernathy, it will match on abby abernathy, Abby, Abernathy, Abby (ABERNATHY), and so on.
• The characters surrounding any match must be of a different type (letters or digits) than the adjacent characters within the word. If your dictionary scans for Abi, it will match the first three characters of Abi904, but not of Abigail.
• Dictionary words containing characters in the Supplementary Multilingual Plane of the Unicode standard can yield unexpected findings. Examples of such characters are emojis, scientific symbols, and historical scripts.

Letters, digits, and other alphabetic characters are defined as follows:

• Letters: characters with general categories Lu, Ll, Lt, Lm, or Lo in the Unicode specification
• Digits: characters with general category Nd in the Unicode specification
• Other alphabetic characters: characters with general category Nl in the Unicode specification or with contributory property Other_Alphabetic as defined by the Unicode Standard

Examples

Simple word list

Suppose you have data that includes what hospital room a patient was treated in during a visit. These locations may be considered sensitive in a particular data set, but they are not something that would be picked up by Sensitive Data Protection's built-in detectors.

The rooms were listed as:

• "RM-Orange"
• "RM-Yellow"
• "RM-Green"

using System;
using Google.Api.Gax.ResourceNames;
using Google.Cloud.Dlp.V2;

public class DeidentifyWithSimpleWordList
{
    public static DeidentifyContentResponse Deidentify(string projectId, string text)
    {
        // Instantiate a client.
        var dlp = DlpServiceClient.Create();

        var contentItem = new ContentItem { Value = text };

        var wordList = new CustomInfoType.Types.Dictionary.Types.WordList
        {
            Words = {