OpenSearchIngestion / Client / create_pipeline

create_pipeline

OpenSearchIngestion.Client.create_pipeline(**kwargs)

Creates an OpenSearch Ingestion pipeline. For more information, see Creating Amazon OpenSearch Ingestion pipelines.

See also: AWS API Documentation

Request Syntax

response = client.create_pipeline(
    PipelineName='string',
    MinUnits=123,
    MaxUnits=123,
    PipelineConfigurationBody='string',
    LogPublishingOptions={
        'IsLoggingEnabled': True|False,
        'CloudWatchLogDestination': {
            'LogGroup': 'string'
        }
    },
    VpcOptions={
        'SubnetIds': [
            'string',
        ],
        'SecurityGroupIds': [
            'string',
        ]
    },
    Tags=[
        {
            'Key': 'string',
            'Value': 'string'
        },
    ]
)

Parameters:

• PipelineName (string) –

  [REQUIRED]

  The name of the OpenSearch Ingestion pipeline to create. Pipeline names are unique across the pipelines owned by an account within an Amazon Web Services Region.

• MinUnits (integer) –

  [REQUIRED]

  The minimum pipeline capacity, in Ingestion Compute Units (ICUs).

• MaxUnits (integer) –

  [REQUIRED]

  The maximum pipeline capacity, in Ingestion Compute Units (ICUs).

• PipelineConfigurationBody (string) –

  [REQUIRED]

  The pipeline configuration in YAML format. The command accepts the pipeline configuration as a string or within a .yaml file. If you provide the configuration as a string, each new line must be escaped with \n.

• LogPublishingOptions (dict) –

  Key-value pairs to configure log publishing.

  • IsLoggingEnabled (boolean) –

    Whether logs should be published.

  • CloudWatchLogDestination (dict) –

    The destination for OpenSearch Ingestion logs sent to Amazon CloudWatch Logs. This parameter is required if IsLoggingEnabled is set to true.

    • LogGroup (string) – [REQUIRED]

      The name of the CloudWatch Logs group to send pipeline logs to. You can specify an existing log group or create a new one. For example, /aws/OpenSearchService/IngestionService/my-pipeline.

• VpcOptions (dict) –

  Container for the values required to configure VPC access for the pipeline. If you don’t specify these values, OpenSearch Ingestion creates the pipeline with a public endpoint.

  • SubnetIds (list) – [REQUIRED]

    A list of subnet IDs associated with the VPC endpoint.

    • (string) –

  • SecurityGroupIds (list) –

    A list of security groups associated with the VPC endpoint.

    • (string) –

• Tags (list) –

  List of tags to add to the pipeline upon creation.

  • (dict) –

    A tag (key-value pair) for an OpenSearch Ingestion pipeline.

    • Key (string) – [REQUIRED]

      The tag key. Tag keys must be unique for the pipeline to which they are attached.

    • Value (string) – [REQUIRED]

      The value assigned to the corresponding tag key. Tag values can be null and don’t have to be unique in a tag set. For example, you can have a key value pair in a tag set of project : Trinity and cost-center : Trinity

Return type:

dict

Returns:

Response Syntax

{
    'Pipeline': {
        'PipelineName': 'string',
        'PipelineArn': 'string',
        'MinUnits': 123,
        'MaxUnits': 123,
        'Status': 'CREATING'|'ACTIVE'|'UPDATING'|'DELETING'|'CREATE_FAILED'|'UPDATE_FAILED'|'STARTING'|'START_FAILED'|'STOPPING'|'STOPPED',
        'StatusReason': {
            'Description': 'string'
        },
        'PipelineConfigurationBody': 'string',
        'CreatedAt': datetime(2015, 1, 1),
        'LastUpdatedAt': datetime(2015, 1, 1),
        'IngestEndpointUrls': [
            'string',
        ],
        'LogPublishingOptions': {
            'IsLoggingEnabled': True|False,
            'CloudWatchLogDestination': {
                'LogGroup': 'string'
            }
        },
        'VpcEndpoints': [
            {
                'VpcEndpointId': 'string',
                'VpcId': 'string',
                'VpcOptions': {
                    'SubnetIds': [
                        'string',
                    ],
                    'SecurityGroupIds': [
                        'string',
                    ]
                }
            },
        ]
    }
}

Response Structure

• (dict) –

  • Pipeline (dict) –

    Container for information about the created pipeline.

    • PipelineName (string) –

      The name of the pipeline.

    • PipelineArn (string) –

      The Amazon Resource Name (ARN) of the pipeline.

    • MinUnits (integer) –

      The minimum pipeline capacity, in Ingestion Compute Units (ICUs).

    • MaxUnits (integer) –

      The maximum pipeline capacity, in Ingestion Compute Units (ICUs).

    • Status (string) –

      The current status of the pipeline.

    • StatusReason (dict) –

      The reason for the current status of the pipeline.

      • Description (string) –

        A description of why a pipeline has a certain status.

    • PipelineConfigurationBody (string) –

      The Data Prepper pipeline configuration in YAML format.

    • CreatedAt (datetime) –

      The date and time when the pipeline was created.

    • LastUpdatedAt (datetime) –

      The date and time when the pipeline was last updated.

    • IngestEndpointUrls (list) –

      The ingestion endpoints for the pipeline, which you can send data to.

      • (string) –

    • LogPublishingOptions (dict) –

      Key-value pairs that represent log publishing settings.

      • IsLoggingEnabled (boolean) –

        Whether logs should be published.

      • CloudWatchLogDestination (dict) –

        The destination for OpenSearch Ingestion logs sent to Amazon CloudWatch Logs. This parameter is required if IsLoggingEnabled is set to true.

        • LogGroup (string) –

          The name of the CloudWatch Logs group to send pipeline logs to. You can specify an existing log group or create a new one. For example, /aws/OpenSearchService/IngestionService/my-pipeline.

    • VpcEndpoints (list) –

      The VPC interface endpoints that have access to the pipeline.

      • (dict) –

        An OpenSearch Ingestion-managed VPC endpoint that will access one or more pipelines.

        • VpcEndpointId (string) –

          The unique identifier of the endpoint.

        • VpcId (string) –

          The ID for your VPC. Amazon Web Services PrivateLink generates this value when you create a VPC.

        • VpcOptions (dict) –

          Information about the VPC, including associated subnets and security groups.

          • SubnetIds (list) –

            A list of subnet IDs associated with the VPC endpoint.

            • (string) –

          • SecurityGroupIds (list) –

            A list of security groups associated with the VPC endpoint.

            • (string) –

Exceptions

• OpenSearchIngestion.Client.exceptions.LimitExceededException

• OpenSearchIngestion.Client.exceptions.ValidationException

• OpenSearchIngestion.Client.exceptions.InternalException

• OpenSearchIngestion.Client.exceptions.AccessDeniedException

• OpenSearchIngestion.Client.exceptions.ResourceAlreadyExistsException