Using subscription filters in Amazon CloudWatch Logs

This Python example shows you how to create and delete filters for log events in CloudWatch Logs.

The scenario

Subscriptions provide access to a real-time feed of log events from CloudWatch Logs and deliver that feed to other services, such as an Amazon Kinesis stream or AWS Lambda, for custom processing, analysis, or loading to other systems. A subscription filter defines the pattern to use for filtering which log events are delivered to your AWS resource.

In this example, Python code is used to list, create, and delete a subscription filter in CloudWatch Logs. The destination for the log events is a Lambda function. The code uses the AWS SDK for Python to manage subscription filters using these methods of the CloudWatchLogs client class:

• get_paginator('describe_subscription_filters').
• put_subscription_filter.
• delete_subscription_filter.

For more information about CloudWatch Logs subscriptions, see Real-time Processing of Log Data with Subscriptions in the Amazon CloudWatch Logs User Guide.

All the example code for the Amazon Web Services (AWS) SDK for Python is available here on GitHub.

Prerequisite tasks

• Configure your AWS credentials, as described in Quickstart.

• Create a Lambda function as the destination for log events. You will need to use the ARN of this function. For more information about setting up a Lambda function, see Subscription Filters with AWS Lambda in the Amazon CloudWatch Logs User Guide.

• Create an IAM role whose policy grants permission to invoke the Lambda function you created and grants full access to CloudWatch Logs or apply the following policy to the execution role you create for the Lambda function. For more information about creating an IAM role, see Creating a Role to Delegate Permissions to an AWS Service in the IAM User Guide.

  Use the following role policy when creating the IAM role.

{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Action": [
            "logs:CreateLogGroup",
            "logs:CreateLogStream",
            "logs:PutLogEvents"
         ],
         "Resource": "arn:aws:logs:*:*:*"
      },
      {
         "Effect": "Allow",
         "Action": [
            "lambda:InvokeFunction"
         ],
         "Resource": [
            "*"
         ]
      }
   ]
}

List existing subscription filters

List the subscription filters for the specified log group.

The example below shows how to:

• List subscription filters using get_paginator('describe_subscription_filters').

For more information about paginators see, Paginators

Example

import boto3

# Create CloudWatchLogs client
cloudwatch_logs = boto3.client('logs')

# List subscription filters through the pagination interface
paginator = cloudwatch_logs.get_paginator('describe_subscription_filters')
for response in paginator.paginate(logGroupName='GROUP_NAME'):
    print(response['subscriptionFilters'])

Create a subscription filter

Create or update a subscription filter and associate it with the specified log group.

The example below shows how to:

• Create a subscription filter using put_subscription_filter.

Example

import boto3

# Create CloudWatchLogs client
cloudwatch_logs = boto3.client('logs')

# Create a subscription filter
cloudwatch_logs.put_subscription_filter(
    destinationArn='LAMBDA_FUNCTION_ARN',
    filterName='FILTER_NAME',
    filterPattern='ERROR',
    logGroupName='LOG_GROUP',
)

Delete a subscription filter

The example below shows how to:

• Delete a subscription filter. using delete_subscription_filter.

Example

import boto3

# Create CloudWatchLogs client
cloudwatch_logs = boto3.client('logs')

# Delete a subscription filter
cloudwatch_logs.delete_subscription_filter(
    filterName='FILTER_NAME',
    logGroupName='LOG_GROUP',
)