Create an Amazon Kinesis Data Streams import topic

An Amazon Kinesis Data Streams import topic lets you continuously ingest data from Amazon Kinesis Data Streams as an external source and into Pub/Sub. Then you can stream the data into any of the destinations that Pub/Sub supports.

For more information about import topics, see About import topics.

Before you begin

Required roles and permissions

To get the permissions that you need to create and manage Amazon Kinesis Data Streams import topics, ask your administrator to grant you the Pub/Sub Editor (roles/pubsub.editor) IAM role on your topic or project. For more information about granting roles, see Manage access to projects, folders, and organizations.

This predefined role contains the permissions required to create and manage Amazon Kinesis Data Streams import topics. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

The following permissions are required to create and manage Amazon Kinesis Data Streams import topics:

  • Create an import topic: pubsub.topics.create
  • Delete an import topic: pubsub.topics.delete
  • Get an import topic: pubsub.topics.get
  • List an import topic: pubsub.topics.list
  • Publish to an import topic: pubsub.topics.publish and pubsub.serviceAgent
  • Update an import topic: pubsub.topics.update
  • Get the IAM policy for an import topic: pubsub.topics.getIamPolicy
  • Configure the IAM policy for an import topic: pubsub.topics.setIamPolicy

You might also be able to get these permissions with custom roles or other predefined roles.

You can configure access control at the project level and the individual resource level.

Set up federated identity to access Amazon Kinesis Data Streams

Workload Identity Federation lets Google Cloud services access workloads running outside of Google Cloud. With identity federation, you don't need to maintain or pass credentials to Google Cloud to access your resources in other clouds. Instead, you can use the identities of the workloads themselves to authenticate to Google Cloud and access resources.

Create a service account in Google Cloud

This is an optional step. If you already have a service account, you can use it in this procedure instead of creating a new service account. If you are using an existing service account, go to Record the service account unique ID for the next step.

For Amazon Kinesis Data Streams import topics, Pub/Sub uses the service account as the identity to access resources from AWS.

For more information about creating a service account, including prerequisites, required roles and permissions, and naming guidelines, see Create service accounts. After you create a service account, you might need to wait for 60 seconds or more before you use the service account. This behavior occurs because read operations are eventually consistent; it can take time for the new service account to become visible.

Record the service account unique ID

You need a service account unique ID to set up set up a role in AWS.

  1. In the Google Cloud console, go to the Service account details page.

    Go to service account

  2. Click the service account that you just created or the one that you are planning to use.

  3. From the Service account details page, record the Unique ID number.

    You need the ID as part of the workflow to set up set up a role in AWS.

Add the service account token creator role to the Pub/Sub service account

The Service account token creator role (roles/iam.serviceAccountTokenCreator) lets principals create short-lived credentials for a service account. These tokens or credentials are used to impersonate the service account.

For more information about service account impersonation, see Service account impersonation.

You can also add the Pub/Sub publisher role (roles/pubsub.publisher) during this procedure. For more information about the role and why you are adding it, see Add the Pub/Sub publisher role to the Pub/Sub service account.

  1. In the Google Cloud console, go to the IAM page.

    Go to IAM

  2. Click the Include Google-provided role grants checkbox.

  3. Look for the service account that has the format service-{PROJECT_NUMBER}@gcp-sa-pubsub.iam.gserviceaccount.com.

  4. For this service account, click the Edit Principal button.

  5. If required, click Add another role.

  6. Search and click the Service account token creator role (roles/iam.serviceAccountTokenCreator).

  7. Click Save.

Create a policy in AWS

You need a policy in AWS to let Pub/Sub authenticate to AWS so that Pub/Sub can ingest data from Amazon Kinesis Data Streams.

To create a policy in AWS, perform the following steps:

  1. Sign in to the AWS Management Console and open the IAM console.

  2. In the navigation pane of the console for IAM, click Access Management > Policies.

  3. Click Create policy.

  4. For Select a service, click Kinesis.

  5. For Action allowed, click the following:

    • List > ListShards.

      This action grants permission to list the shards in a stream and provides information about each shard.

    • Read > SubscribeToShard.

      This action grants permission to listen to a specific shard with enhanced fan-out.

    • Read > DescribeStreamConsumer.

      This action grants permission to get the description of a registered stream consumer.

    These permissions cover reading from the stream. Pub/Sub only supports reading from a Kinesis stream with Enhanced Fan-Out by using the streaming SubscribeToShard API.

  6. For Resources, if you'd like to restrict the policy to specific stream or consumer (recommended), specify the consumer ARN and the