Natural Language API Basics

This document provides a guide to the basics of using the Cloud Natural Language API. This conceptual guide covers the types of requests you can make to the Natural Language API, how to construct those requests, and how to handle their responses. We recommend that all users of the Natural Language API read this guide and one of the associated tutorials before diving into the API itself.

Natural Language features

The Natural Language API has several methods for performing analysis and annotation on your text. Each level of analysis provides valuable information for language understanding. These methods are listed below:

• Sentiment analysis inspects the given text and identifies the prevailing emotional opinion within the text, especially to determine a writer's attitude as positive, negative, or neutral. Sentiment analysis is performed through the analyzeSentiment method.

• Entity analysis inspects the given text for known entities (Proper nouns such as public figures, landmarks, and so on. Common nouns such as restaurant, stadium, and so on.) and returns information about those entities. Entity analysis is performed with the analyzeEntities method.

• Entity sentiment analysis inspects the given text for known entities (proper nouns and common nouns), returns information about those entities, and identifies the prevailing emotional opinion of the entity within the text, especially to determine a writer's attitude toward the entity as positive, negative, or neutral. Entity analysis is performed with the analyzeEntitySentiment method.

• Syntactic analysis extracts linguistic information, breaking up the given text into a series of sentences and tokens (generally, word boundaries), providing further analysis on those tokens. Syntactic Analysis is performed with the analyzeSyntax method.

• Content classification analyzes text content and returns a content category for the content. Content classification is performed by using the classifyText method.

Each API call also detects and returns the language, if a language is not specified by the caller in the initial request.

Additionally, if you wish to perform several natural language operations on given text using only one API call, the annotateText request can also be used to perform sentiment analysis and entity analysis.

Basic Natural Language requests

The Natural Language API is a REST API, and consists of JSON requests and response. A simple Natural Language JSON Entity Analysis request appears below:

{
  "document":{
    "type":"PLAIN_TEXT",
    "language_code": "EN",
    "content":"'Lawrence of Arabia' is a highly rated film biography about
                British Lieutenant T. E. Lawrence. Peter O'Toole plays
                Lawrence in the film."
  },
  "encodingType":"UTF8"
}

These fields are explained below:

• document contains the data for this request, which consists of the following sub-fields:
  • type - document type (HTML or PLAIN_TEXT)
  • language - (optional) the language of the text within the request. If not specified, language will be automatically detected. For information on which languages are supported by the Natural Language API, see Language Support. Unsupported languages will return an error in the JSON response.
    
  • Either content or gcsContentUri which contain the text to evaluate. If passing content, this text is included directly in the JSON request (as shown above). If passing gcsContentUri, the field must contain a URI pointing to text content within Google Cloud Storage.
• encodingType - (required) the encoding scheme in which returned character offsets into the text should be calculated, which must match the encoding of the passed text. If this parameter is not set, the request will not error, but all such offsets will be set to -1.

Specifying text content

When passing a Natural Language API request, you specify the text to process in one of two ways:

• Passing the text directly within a content field.
• Passing a Google Cloud Storage URI within a gcsContentUri field.

In either case, you should make sure not to pass more than the Content Limits allow. Note that these content limits are by byte, not by character; character length therefore depends on your text's encoding.

The request below refers to a Google Cloud Storage file containing the Gettysburg Address:

{
  "document":{
    "type":"PLAIN_TEXT",
    "language": "EN",
    "gcsContentUri":"gs://cloud-samples-tests/natural-language/gettysburg.txt"
  },
}

Sentiment analysis

Sentiment analysis attempts to determine the overall attitude (positive or negative) expressed within the text. Sentiment is represented by numerical score and magnitude values.

Sentiment analysis response fields

A sample analyzeSentiment response to the Gettysburg Address is shown below:

{
  "documentSentiment": {
    "score": 0.2,
    "magnitude": 3.6
  },
  "language_code": "en",
   "sentences": [
    {
      "text": {
        "content": "Four score and seven years ago our fathers brought forth
        on this continent a new nation, conceived in liberty and dedicated to
        the proposition that all men are created equal.",
        "beginOffset": 0
      },
      "sentiment": {
        "magnitude": 0.8,
        "score": 0.8
      }
    },
   ...
}

These field values are described below:

• documentSentiment contains the overall sentiment of the document, which consists of the following fields:
  • score of the sentiment ranges between -1.0 (negative) and 1.0 (positive) and corresponds to the overall emotional leaning of the text.
  • magnitude indicates the overall strength of emotion (both positive and negative) within the given text, between 0.0 and +inf. Unlike score, magnitude is not normalized for documentSentiment; each expression of emotion within the text (both positive and negative) contributes to the text's magnitude (so longer text blocks may have greater magnitudes).
• language_code contains the language of the document, either passed in the initial request, or automatically detected if absent.
• language_supported contains a boolean value to identify if the language is officially supported
• sentences contains a list of the sentences extracted from the original document, which contains:
  • sentiment contains the sentence level sentiment values attached to each sentence, which contain score between -1.0 (negative) and 1.0 (positive) as and magnitude values between 0.0 and 1.0. Note that magnitude for sentences is normalized.

A sentiment value of 0.2 for the Gettysburg Address indicates is slightly positive in emotion, while the magnitude value of 3.6 indicates a relatively emotional document, given its small size (of about a paragraph). Note that the first sentence of the Gettysburg address contains a very high positive score of 0.8.

Interpreting sentiment analysis values

The score of a document's sentiment indicates the overall emotion of a document. The magnitude of a document's sentiment indicates how much emotional content is present within the document, and this value is often proportional to the length of the document.

It is important to note that the Natural Language API indicates differences between positive and negative emotion in a document, but does not identify specific positive and negative emotions. For example, "angry" and "sad" are both considered negative emotions. However, when the Natural Language API analyzes text that is considered "angry", or text that is considered "sad", the response only indicates that the sentiment in the text is negative, not "sad" or "angry".

A document with a neutral score (around 0.0) may indicate a low-emotion document, or may indicate mixed emotions, with both high positive and negative values which cancel each out. Generally, you can use magnitude values to disambiguate these cases, as truly neutral documents will have a low magnitude value, while mixed documents will have higher magnitude values.

When comparing documents to each other (especially documents of different length), make sure to use the magnitude values to calibrate your scores, as they can help you gauge the relevant amount of emotional content.

The chart below shows some sample values and how to interpret them:

* “Clearly positive” and “clearly negative” sentiment varies for different use cases and customers. You might find differing results for your specific scenario. We recommend that you define a threshold that works for you, and then adjust the threshold after testing and verifying the results. For example, you may define a threshold of any score over 0.25 as clearly positive, and then modify the score threshold to 0.15 after reviewing your data and results and finding that scores from 0.15-0.25 should be considered positive as well.

Entity analysis

Entity Analysis provides information about entities in the text, which generally refer to named "things" such as famous individuals, landmarks, common objects, etc.

Entities broadly fall into two categories: proper nouns that map to unique entities (specific people, places, etc.) or common nouns (also called "nominals" in natural language processing). A good general practice to follow is that if something is a noun, it qualifies as an "entity." Entities are returned as indexed offsets into the original text.

An Entity Analysis request should pass an encodingType argument, so that the returned offsets can be properly interpreted.

Entity analysis response fields

Entity analysis returns a set of detected entities, and parameters associated with those entities, such as the entity's type, relevance of the entity to the overall text, and locations in the text that refer to the same entity.

An analyzeEntities response to the entity request is shown below: