Glue / Client / get_unfiltered_partitions_metadata

get_unfiltered_partitions_metadata#

Glue.Client.get_unfiltered_partitions_metadata(**kwargs)#

Retrieves partition metadata from the Data Catalog that contains unfiltered metadata.

For IAM authorization, the public IAM action associated with this API is glue:GetPartitions.

See also: AWS API Documentation

Request Syntax

response = client.get_unfiltered_partitions_metadata(
    Region='string',
    CatalogId='string',
    DatabaseName='string',
    TableName='string',
    Expression='string',
    AuditContext={
        'AdditionalAuditContext': 'string',
        'RequestedColumns': [
            'string',
        ],
        'AllColumnsRequested': True|False
    },
    SupportedPermissionTypes=[
        'COLUMN_PERMISSION'|'CELL_FILTER_PERMISSION'|'NESTED_PERMISSION'|'NESTED_CELL_PERMISSION',
    ],
    NextToken='string',
    Segment={
        'SegmentNumber': 123,
        'TotalSegments': 123
    },
    MaxResults=123,
    QuerySessionContext={
        'QueryId': 'string',
        'QueryStartTime': datetime(2015, 1, 1),
        'ClusterId': 'string',
        'QueryAuthorizationId': 'string',
        'AdditionalContext': {
            'string': 'string'
        }
    }
)
Parameters:
  • Region (string) – Specified only if the base tables belong to a different Amazon Web Services Region.

  • CatalogId (string) –

    [REQUIRED]

    The ID of the Data Catalog where the partitions in question reside. If none is provided, the AWS account ID is used by default.

  • DatabaseName (string) –

    [REQUIRED]

    The name of the catalog database where the partitions reside.

  • TableName (string) –

    [REQUIRED]

    The name of the table that contains the partition.

  • Expression (string) –

    An expression that filters the partitions to be returned.

    The expression uses SQL syntax similar to the SQL WHERE filter clause. The SQL statement parser JSQLParser parses the expression.

    Operators: The following are the operators that you can use in the Expression API call:

    =

    Checks whether the values of the two operands are equal; if yes, then the condition becomes true.

    Example: Assume ‘variable a’ holds 10 and ‘variable b’ holds 20.

    (a = b) is not true.

    < >

    Checks whether the values of two operands are equal; if the values are not equal, then the condition becomes true.

    Example: (a < > b) is true.

    >

    Checks whether the value of the left operand is greater than the value of the right operand; if yes, then the condition becomes true.

    Example: (a > b) is not true.

    <

    Checks whether the value of the left operand is less than the value of the right operand; if yes, then the condition becomes true.

    Example: (a < b) is true.

    >=

    Checks whether the value of the left operand is greater than or equal to the value of the right operand; if yes, then the condition becomes true.

    Example: (a >= b) is not true.

    <=

    Checks whether the value of the left operand is less than or equal to the value of the right operand; if yes, then the condition becomes true.

    Example: (a <= b) is true.

    AND, OR, IN, BETWEEN, LIKE, NOT, IS NULL

    Logical operators.

    Supported Partition Key Types: The following are the supported partition keys.

    • string

    • date

    • timestamp

    • int

    • bigint

    • long

    • tinyint

    • smallint

    • decimal

    If an type is encountered that is not valid, an exception is thrown.

  • AuditContext (dict) –

    A structure containing Lake Formation audit context information.

    • AdditionalAuditContext (string) –

      A string containing the additional audit context information.

    • RequestedColumns (list) –

      The requested columns for audit.

      • (string) –

    • AllColumnsRequested (boolean) –

      All columns request for audit.

  • SupportedPermissionTypes (list) –

    [REQUIRED]

    A list of supported permission types.

    • (string) –

  • NextToken (string) – A continuation token, if this is not the first call to retrieve these partitions.

  • Segment (dict) –

    The segment of the table’s partitions to scan in this request.

    • SegmentNumber (integer) – [REQUIRED]

      The zero-based index number of the segment. For example, if the total number of segments is 4, SegmentNumber values range from 0 through 3.

    • TotalSegments (integer) – [REQUIRED]

      The total number of segments.

  • MaxResults (integer) – The maximum number of partitions to return in a single response.

  • QuerySessionContext (dict) –

    A structure used as a protocol between query engines and Lake Formation or Glue. Contains both a Lake Formation generated authorization identifier and information from the request’s authorization context.

    • QueryId (string) –

      A unique identifier generated by the query engine for the query.

    • QueryStartTime (datetime) –

      A timestamp provided by the query engine for when the query started.

    • ClusterId (string) –

      An identifier string for the consumer cluster.

    • QueryAuthorizationId (string) –

      A cryptographically generated query identifier generated by Glue or Lake Formation.

    • AdditionalContext (dict) –

      An opaque string-string map passed by the query engine.

      • (string) –

        • (string) –

Return type:

dict

Returns:

Response Syntax

{
    'UnfilteredPartitions': [
        {
            'Partition': {
                'Values': [
                    'string',
                ],
                'DatabaseName': 'string',
                'TableName': 'string',
                'CreationTime': datetime(2015, 1, 1),
                'LastAccessTime': datetime(2015, 1, 1),
                'StorageDescriptor': {
                    'Columns': [
                        {
                            'Name': 'string',
                            'Type': 'string',
                            'Comment': 'string',
                            'Parameters': {
                                'string': 'string'
                            }
                        },
                    ],
                    'Location': 'string',
                    'AdditionalLocations': [
                        'string',
                    ],
                    'InputFormat': 'string',
                    'OutputFormat': 'string',
                    'Compressed': True|False,
                    'NumberOfBuckets': 123,
                    'SerdeInfo': {
                        'Name': 'string',
                        'SerializationLibrary': 'string',
                        'Parameters': {
                            'string': 'string'
                        }
                    },
                    'BucketColumns': [
                        'string',
                    ],
                    'SortColumns': [
                        {
                            'Column': 'string',
                            'SortOrder': 123
                        },
                    ],
                    'Parameters': {
                        'string': 'string'
                    },
                    'SkewedInfo': {
                        'SkewedColumnNames': [
                            'string',
                        ],
                        'SkewedColumnValues': [
                            'string',
                        ],
                        'SkewedColumnValueLocationMaps': {
                            'string': 'string'
                        }
                    },
                    'StoredAsSubDirectories': True|False,
                    'SchemaReference': {
                        'SchemaId': {
                            'SchemaArn': 'string',
                            'SchemaName': 'string',
                            'RegistryName': 'string'
                        },
                        'SchemaVersionId': 'string',
                        'SchemaVersionNumber': 123
                    }
                },
                'Parameters': {
                    'string': 'string'
                },
                'LastAnalyzedTime': datetime(2015, 1, 1),
                'CatalogId': 'string'
            },
            'AuthorizedColumns': [
                'string',
            ],
            'IsRegisteredWithLakeFormation': True|False
        },
    ],
    'NextToken': 'string'
}

Response Structure

  • (dict) –

    • UnfilteredPartitions (list) –

      A list of requested partitions.

      • (dict) –

        A partition that contains unfiltered metadata.

        • Partition (dict) –

          The partition object.

          • Values (list) –

            The values of the partition.

            • (string) –

          • DatabaseName (string) –

            The name of the catalog database in which to create the partition.

          • TableName (string) –

            The name of the database table in which to create the partition.

          • CreationTime (datetime) –

            The time at which the partition was created.

          • LastAccessTime (datetime) –

            The last time at which the partition was accessed.

          • StorageDescriptor (dict) –

            Provides information about the physical location where the partition is stored.

            • Columns (list) –

              A list of the Columns in the table.

              • (dict) –

                A column in a Table.

                • Name (string) –

                  The name of the Column.

                • Type (string) –

                  The data type of the Column.

                • Comment (string) –

                  A free-form text comment.

                • Parameters (dict) –

                  These key-value pairs define properties associated with the column.

                  • (string) –

                    • (string) –

            • Location (string) –

              The physical location of the table. By default, this takes the form of the warehouse location, followed by the database location in the warehouse, followed by the table name.

            • AdditionalLocations (list) –

              A list of locations that point to the path where a Delta table is located.

              • (string) –

            • InputFormat (string) –

              The input format: SequenceFileInputFormat (binary), or TextInputFormat, or a custom format.

            • OutputFormat (string) –

              The output format: SequenceFileOutputFormat (binary), or IgnoreKeyTextOutputFormat, or a custom format.

            • Compressed (boolean) –

              True if the data in the table is compressed, or False if not.

            • NumberOfBuckets (integer) –

              Must be specified if the table contains any dimension columns.

            • SerdeInfo (dict) –

              The serialization/deserialization (SerDe) information.

              • Name (string) –

                Name of the SerDe.

              • SerializationLibrary (string) –

                Usually the class that implements the SerDe. An example is org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe.

              • Parameters (dict) –

                These key-value pairs define initialization parameters for the SerDe.

                • (string) –

                  • (string) –

            • BucketColumns (list) –

              A list of reducer grouping columns, clustering columns, and bucketing columns in the table.

              • (string) –

            • SortColumns (list) –

              A list specifying the sort order of each bucket in the table.

              • (dict) –

                Specifies the sort order of a sorted column.

                • Column (string) –

                  The name of the column.

                • SortOrder (integer) –

                  Indicates that the column is sorted in ascending order ( == 1), or in descending order ( ==0).

            • Parameters (dict) –

              The user-supplied properties in key-value form.

              • (string) –

                • (string) –

            • SkewedInfo (dict) –

              The information about values that appear frequently in a column (skewed values).

              • SkewedColumnNames (list) –

                A list of names of columns that contain skewed values.

                • (string) –

              • SkewedColumnValues (list) –

                A list of values that appear so frequently as to be considered skewed.

                • (string) –

              • SkewedColumnValueLocationMaps (dict) –

                A mapping of skewed values to the columns that contain them.

                • (string) –

                  • (string) –

            • StoredAsSubDirectories (boolean) –

              True if the table data is stored in subdirectories, or False if not.

            • SchemaReference (dict) –

              An object that references a schema stored in the Glue Schema Registry.

              When creating a table, you can pass an empty list of columns for the schema, and instead use a schema reference.

              • SchemaId (dict) –

                A structure that contains schema identity fields. Either this or the SchemaVersionId has to be provided.

                • SchemaArn (string) –

                  The Amazon Resource Name (ARN) of the schema. One of SchemaArn or SchemaName has to be provided.

                • SchemaName (string) –

                  The name of the schema. One of SchemaArn or SchemaName has to be provided.

                • RegistryName (string) –

                  The name of the schema registry that contains the schema.

              • SchemaVersionId (string) –

                The unique ID assigned to a version of the schema. Either this or the SchemaId has to be provided.

              • SchemaVersionNumber (integer) –

                The version number of the schema.

          • Parameters (dict) –

            These key-value pairs define partition parameters.

            • (string) –

              • (string) –

          • LastAnalyzedTime (datetime) –

            The last time at which column statistics were computed for this partition.

          • CatalogId (string) –

            The ID of the Data Catalog in which the partition resides.

        • AuthorizedColumns (list) –

          The list of columns the user has permissions to access.

          • (string) –

        • IsRegisteredWithLakeFormation (boolean) –

          A Boolean value indicating that the partition location is registered with Lake Formation.

    • NextToken (string) –

      A continuation token, if the returned list of partitions does not include the last one.

Exceptions

  • Glue.Client.exceptions.EntityNotFoundException

  • Glue.Client.exceptions.InvalidInputException

  • Glue.Client.exceptions.InternalServiceException

  • Glue.Client.exceptions.OperationTimeoutException

  • Glue.Client.exceptions.GlueEncryptionException

  • Glue.Client.exceptions.PermissionTypeMismatchException

  • Glue.Client.exceptions.FederationSourceException

  • Glue.Client.exceptions.FederationSourceRetryableException