SupplyChain / Client / update_data_integration_flow

update_data_integration_flow#

SupplyChain.Client.update_data_integration_flow(**kwargs)#

Enables you to programmatically update an existing data pipeline to ingest data from the source systems such as, Amazon S3 buckets, to a predefined Amazon Web Services Supply Chain dataset (product, inbound_order) or a temporary dataset along with the data transformation query provided with the API.

Request Syntax

response = client.update_data_integration_flow(
    instanceId='string',
    name='string',
    sources=[
        {
            'sourceType': 'S3'|'DATASET',
            'sourceName': 'string',
            's3Source': {
                'bucketName': 'string',
                'prefix': 'string',
                'options': {
                    'fileType': 'CSV'|'PARQUET'|'JSON'
                }
            },
            'datasetSource': {
                'datasetIdentifier': 'string',
                'options': {
                    'loadType': 'INCREMENTAL'|'REPLACE',
                    'dedupeRecords': True|False
                }
            }
        },
    ],
    transformation={
        'transformationType': 'SQL'|'NONE',
        'sqlTransformation': {
            'query': 'string'
        }
    },
    target={
        'targetType': 'S3'|'DATASET',
        's3Target': {
            'bucketName': 'string',
            'prefix': 'string',
            'options': {
                'fileType': 'CSV'|'PARQUET'|'JSON'
            }
        },
        'datasetTarget': {
            'datasetIdentifier': 'string',
            'options': {
                'loadType': 'INCREMENTAL'|'REPLACE',
                'dedupeRecords': True|False
            }
        }
    }
)

Parameters:

instanceId (string) –
[REQUIRED]

The Amazon Web Services Supply Chain instance identifier.
name (string) –
[REQUIRED]

The name of the DataIntegrationFlow to be updated.
sources (list) –
The new source configurations for the DataIntegrationFlow.
- (dict) –
  
  The DataIntegrationFlow source parameters.
  - sourceType (string) – [REQUIRED]
    
    The DataIntegrationFlow source type.
  - sourceName (string) – [REQUIRED]
    
    The DataIntegrationFlow source name that can be used as table alias in SQL transformation query.
  - s3Source (dict) –
    
    The S3 DataIntegrationFlow source.
    - bucketName (string) – [REQUIRED]
      
      The bucketName of the S3 source objects.
    - prefix (string) – [REQUIRED]
      
      The prefix of the S3 source objects.
    - options (dict) –
      
      The other options of the S3 DataIntegrationFlow source.
      - fileType (string) –
        
        The Amazon S3 file type in S3 options.
  - datasetSource (dict) –
    
    The dataset DataIntegrationFlow source.
    - datasetIdentifier (string) – [REQUIRED]
      
      The ARN of the dataset.
    - options (dict) –
      
      The dataset DataIntegrationFlow source options.
      - loadType (string) –
        
        The dataset data load type in dataset options.
      - dedupeRecords (boolean) –
        
        The dataset load option to remove duplicates.
transformation (dict) –
The new transformation configurations for the DataIntegrationFlow.
- transformationType (string) – [REQUIRED]
  
  The DataIntegrationFlow transformation type.
- sqlTransformation (dict) –
  
  The SQL DataIntegrationFlow transformation configuration.
  - query (string) – [REQUIRED]
    
    The transformation SQL query body based on SparkSQL.
target (dict) –
The new target configurations for the DataIntegrationFlow.
- targetType (string) – [REQUIRED]
  
  The DataIntegrationFlow target type.
- s3Target (dict) –
  
  The S3 DataIntegrationFlow target.
  - bucketName (string) – [REQUIRED]
    
    The bucketName of the S3 target objects.
  - prefix (string) – [REQUIRED]
    
    The prefix of the S3 target objects.
  - options (dict) –
    
    The S3 DataIntegrationFlow target options.
    - fileType (string) –
      
      The Amazon S3 file type in S3 options.
- datasetTarget (dict) –
  
  The dataset DataIntegrationFlow target.
  - datasetIdentifier (string) – [REQUIRED]
    
    The dataset ARN.
  - options (dict) –
    
    The dataset DataIntegrationFlow target options.
    - loadType (string) –
      
      The dataset data load type in dataset options.
    - dedupeRecords (boolean) –
      
      The dataset load option to remove duplicates.

Return type:

dict

Returns:

Response Syntax

{
    'flow': {
        'instanceId': 'string',
        'name': 'string',
        'sources': [
            {
                'sourceType': 'S3'|'DATASET',
                'sourceName': 'string',
                's3Source': {
                    'bucketName': 'string',
                    'prefix': 'string',
                    'options': {
                        'fileType': 'CSV'|'PARQUET'|'JSON'
                    }
                },
                'datasetSource': {
                    'datasetIdentifier': 'string',
                    'options': {
                        'loadType': 'INCREMENTAL'|'REPLACE',
                        'dedupeRecords': True|False
                    }
                }
            },
        ],
        'transformation': {
            'transformationType': 'SQL'|'NONE',
            'sqlTransformation': {
                'query': 'string'
            }
        },
        'target': {
            'targetType': 'S3'|'DATASET',
            's3Target': {
                'bucketName': 'string',
                'prefix': 'string',
                'options': {
                    'fileType': 'CSV'|'PARQUET'|'JSON'
                }
            },
            'datasetTarget': {
                'datasetIdentifier': 'string',
                'options': {
                    'loadType': 'INCREMENTAL'|'REPLACE',
                    'dedupeRecords': True|False
                }
            }
        },
        'createdTime': datetime(2015, 1, 1),
        'lastModifiedTime': datetime(2015, 1, 1)
    }
}

Response Structure

(dict) –

The response parameters for UpdateDataIntegrationFlow.
- flow (dict) –
  
  The details of the updated DataIntegrationFlow.
  - instanceId (string) –
    
    The DataIntegrationFlow instance ID.
  - name (string) –
    
    The DataIntegrationFlow name.
  - sources (list) –
    
    The DataIntegrationFlow source configurations.
    - (dict) –
      
      The DataIntegrationFlow source parameters.
      - sourceType (string) –
        
        The DataIntegrationFlow source type.
      - sourceName (string) –
        
        The DataIntegrationFlow source name that can be used as table alias in SQL transformation query.
      - s3Source (dict) –
        
        The S3 DataIntegrationFlow source.
        
        bucketName (string) –
        
        The bucketName of the S3 source objects.
        
        prefix (string) –
        
        The prefix of the S3 source objects.
        
        options (dict) –
        
        The other options of the S3 DataIntegrationFlow source.
        
        fileType (string) –
        
        The Amazon S3 file type in S3 options.
      - datasetSource (dict) –
        
        The dataset DataIntegrationFlow source.
        
        datasetIdentifier (string) –
        
        The ARN of the dataset.
        
        options (dict) –
        
        The dataset DataIntegrationFlow source options.
        
        loadType (string) –
        
        The dataset data load type in dataset options.
        
        dedupeRecords (boolean) –
        
        The dataset load option to remove duplicates.
  - transformation (dict) –
    
    The DataIntegrationFlow transformation configurations.
    - transformationType (string) –
      
      The DataIntegrationFlow transformation type.
    - sqlTransformation (dict) –
      
      The SQL DataIntegrationFlow transformation configuration.
      - query (string) –
        
        The transformation SQL query body based on SparkSQL.
  - target (dict) –
    
    The DataIntegrationFlow target configuration.
    - targetType (string) –
      
      The DataIntegrationFlow target type.
    - s3Target (dict) –
      
      The S3 DataIntegrationFlow target.
      - bucketName (string) –
        
        The bucketName of the S3 target objects.
      - prefix (string) –
        
        The prefix of the S3 target objects.
      - options (dict) –
        
        The S3 DataIntegrationFlow target options.
        
        fileType (string) –
        
        The Amazon S3 file type in S3 options.
    - datasetTarget (dict) –
      
      The dataset DataIntegrationFlow target.
      - datasetIdentifier (string) –
        
        The dataset ARN.
      - options (dict) –
        
        The dataset DataIntegrationFlow target options.
        
        loadType (string) –
        
        The dataset data load type in dataset options.
        
        dedupeRecords (boolean) –
        
        The dataset load option to remove duplicates.
  - createdTime (datetime) –
    
    The DataIntegrationFlow creation timestamp.
  - lastModifiedTime (datetime) –
    
    The DataIntegrationFlow last modified timestamp.

update_data_integration_flow#

Request Syntax

Response Syntax

Response Structure

Exceptions