OpenSearchIngestion / Client / create_pipeline

create_pipeline#

OpenSearchIngestion.Client.create_pipeline(**kwargs)#

Creates an OpenSearch Ingestion pipeline. For more information, see Creating Amazon OpenSearch Ingestion pipelines.

See also: AWS API Documentation

Request Syntax

response = client.create_pipeline(
    PipelineName='string',
    MinUnits=123,
    MaxUnits=123,
    PipelineConfigurationBody='string',
    LogPublishingOptions={
        'IsLoggingEnabled': True|False,
        'CloudWatchLogDestination': {
            'LogGroup': 'string'
        }
    },
    VpcOptions={
        'SubnetIds': [
            'string',
        ],
        'SecurityGroupIds': [
            'string',
        ]
    },
    Tags=[
        {
            'Key': 'string',
            'Value': 'string'
        },
    ]
)
Parameters:
  • PipelineName (string) –

    [REQUIRED]

    The name of the OpenSearch Ingestion pipeline to create. Pipeline names are unique across the pipelines owned by an account within an Amazon Web Services Region.

  • MinUnits (integer) –

    [REQUIRED]

    The minimum pipeline capacity, in Ingestion Compute Units (ICUs).

  • MaxUnits (integer) –

    [REQUIRED]

    The maximum pipeline capacity, in Ingestion Compute Units (ICUs).

  • PipelineConfigurationBody (string) –

    [REQUIRED]

    The pipeline configuration in YAML format. The command accepts the pipeline configuration as a string or within a .yaml file. If you provide the configuration as a string, each new line must be escaped with \n.

  • LogPublishingOptions (dict) –

    Key-value pairs to configure log publishing.

    • IsLoggingEnabled (boolean) –

      Whether logs should be published.

    • CloudWatchLogDestination (dict) –

      The destination for OpenSearch Ingestion logs sent to Amazon CloudWatch Logs. This parameter is required if IsLoggingEnabled is set to true.

      • LogGroup (string) – [REQUIRED]

        The name of the CloudWatch Logs group to send pipeline logs to. You can specify an existing log group or create a new one. For example, /aws/OpenSearchService/IngestionService/my-pipeline.

  • VpcOptions (dict) –

    Container for the values required to configure VPC access for the pipeline. If you don’t specify these values, OpenSearch Ingestion creates the pipeline with a public endpoint.

    • SubnetIds (list) – [REQUIRED]

      A list of subnet IDs associated with the VPC endpoint.

      • (string) –

    • SecurityGroupIds (list) –

      A list of security groups associated with the VPC endpoint.

      • (string) –

  • Tags (list) –

    List of tags to add to the pipeline upon creation.

    • (dict) –

      A tag (key-value pair) for an OpenSearch Ingestion pipeline.

      • Key (string) – [REQUIRED]

        The tag key. Tag keys must be unique for the pipeline to which they are attached.

      • Value (string) – [REQUIRED]

        The value assigned to the corresponding tag key. Tag values can be null and don’t have to be unique in a tag set. For example, you can have a key value pair in a tag set of project : Trinity and cost-center : Trinity

Return type:

dict

Returns:

Response Syntax

{
    'Pipeline': {
        'PipelineName': 'string',
        'PipelineArn': 'string',
        'MinUnits': 123,
        'MaxUnits': 123,
        'Status': 'CREATING'|'ACTIVE'|'UPDATING'|'DELETING'|'CREATE_FAILED'|'UPDATE_FAILED'|'STARTING'|'START_FAILED'|'STOPPING'|'STOPPED',
        'StatusReason': {
            'Description': 'string'
        },
        'PipelineConfigurationBody': 'string',
        'CreatedAt': datetime(2015, 1, 1),
        'LastUpdatedAt': datetime(2015, 1, 1),
        'IngestEndpointUrls': [
            'string',
        ],
        'LogPublishingOptions': {
            'IsLoggingEnabled': True|False,
            'CloudWatchLogDestination': {
                'LogGroup': 'string'
            }
        },
        'VpcEndpoints': [
            {
                'VpcEndpointId': 'string',
                'VpcId': 'string',
                'VpcOptions': {
                    'SubnetIds': [
                        'string',
                    ],
                    'SecurityGroupIds': [
                        'string',
                    ]
                }
            },
        ]
    }
}

Response Structure

  • (dict) –

    • Pipeline (dict) –

      Container for information about the created pipeline.

      • PipelineName (string) –

        The name of the pipeline.

      • PipelineArn (string) –

        The Amazon Resource Name (ARN) of the pipeline.

      • MinUnits (integer) –

        The minimum pipeline capacity, in Ingestion Compute Units (ICUs).

      • MaxUnits (integer) –

        The maximum pipeline capacity, in Ingestion Compute Units (ICUs).

      • Status (string) –

        The current status of the pipeline.

      • StatusReason (dict) –

        The reason for the current status of the pipeline.

        • Description (string) –

          A description of why a pipeline has a certain status.

      • PipelineConfigurationBody (string) –

        The Data Prepper pipeline configuration in YAML format.

      • CreatedAt (datetime) –

        The date and time when the pipeline was created.

      • LastUpdatedAt (datetime) –

        The date and time when the pipeline was last updated.

      • IngestEndpointUrls (list) –

        The ingestion endpoints for the pipeline, which you can send data to.

        • (string) –

      • LogPublishingOptions (dict) –

        Key-value pairs that represent log publishing settings.

        • IsLoggingEnabled (boolean) –

          Whether logs should be published.

        • CloudWatchLogDestination (dict) –

          The destination for OpenSearch Ingestion logs sent to Amazon CloudWatch Logs. This parameter is required if IsLoggingEnabled is set to true.

          • LogGroup (string) –

            The name of the CloudWatch Logs group to send pipeline logs to. You can specify an existing log group or create a new one. For example, /aws/OpenSearchService/IngestionService/my-pipeline.

      • VpcEndpoints (list) –

        The VPC interface endpoints that have access to the pipeline.

        • (dict) –

          An OpenSearch Ingestion-managed VPC endpoint that will access one or more pipelines.

          • VpcEndpointId (string) –

            The unique identifier of the endpoint.

          • VpcId (string) –

            The ID for your VPC. Amazon Web Services PrivateLink generates this value when you create a VPC.

          • VpcOptions (dict) –

            Information about the VPC, including associated subnets and security groups.

            • SubnetIds (list) –

              A list of subnet IDs associated with the VPC endpoint.

              • (string) –

            • SecurityGroupIds (list) –

              A list of security groups associated with the VPC endpoint.

              • (string) –

Exceptions

  • OpenSearchIngestion.Client.exceptions.LimitExceededException

  • OpenSearchIngestion.Client.exceptions.ValidationException

  • OpenSearchIngestion.Client.exceptions.InternalException

  • OpenSearchIngestion.Client.exceptions.AccessDeniedException

  • OpenSearchIngestion.Client.exceptions.ResourceAlreadyExistsException