NeptuneData / Client / get_propertygraph_stream

get_propertygraph_stream#

NeptuneData.Client.get_propertygraph_stream(**kwargs)#

Gets a stream for a property graph.

With the Neptune Streams feature, you can generate a complete sequence of change-log entries that record every change made to your graph data as it happens. GetPropertygraphStream lets you collect these change-log entries for a property graph.

The Neptune streams feature needs to be enabled on your Neptune DBcluster. To enable streams, set the neptune_streams DB cluster parameter to 1.

See Capturing graph changes in real time using Neptune streams.

When invoking this operation in a Neptune cluster that has IAM authentication enabled, the IAM user or role making the request must have a policy attached that allows the neptune-db:GetStreamRecords IAM action in that cluster.

When invoking this operation in a Neptune cluster that has IAM authentication enabled, the IAM user or role making the request must have a policy attached that enables one of the following IAM actions, depending on the query:

Note that you can restrict property-graph queries using the following IAM context keys:

See Condition keys available in Neptune IAM data-access policy statements).

Request Syntax

response = client.get_propertygraph_stream(
    limit=123,
    iteratorType='AT_SEQUENCE_NUMBER'|'AFTER_SEQUENCE_NUMBER'|'TRIM_HORIZON'|'LATEST',
    commitNum=123,
    opNum=123,
    encoding='gzip'
)

Parameters:

limit (integer) –
Specifies the maximum number of records to return. There is also a size limit of 10 MB on the response that can’t be modified and that takes precedence over the number of records specified in the limit parameter. The response does include a threshold-breaching record if the 10 MB limit was reached.

The range for limit is 1 to 100,000, with a default of 10.
iteratorType (string) –
Can be one of:
- AT_SEQUENCE_NUMBER – Indicates that reading should start from the event sequence number specified jointly by the commitNum and opNum parameters.
- AFTER_SEQUENCE_NUMBER – Indicates that reading should start right after the event sequence number specified jointly by the commitNum and opNum parameters.
- TRIM_HORIZON – Indicates that reading should start at the last untrimmed record in the system, which is the oldest unexpired (not yet deleted) record in the change-log stream.
- LATEST – Indicates that reading should start at the most recent record in the system, which is the latest unexpired (not yet deleted) record in the change-log stream.
commitNum (integer) – The commit number of the starting record to read from the change-log stream. This parameter is required when iteratorType is AT_SEQUENCE_NUMBER or AFTER_SEQUENCE_NUMBER, and ignored when iteratorType is TRIM_HORIZON or LATEST.
opNum (integer) – The operation sequence number within the specified commit to start reading from in the change-log stream data. The default is 1.
encoding (string) – If set to TRUE, Neptune compresses the response using gzip encoding.

Return type:

dict

Returns:

Response Syntax

{
    'lastEventId': {
        'string': 'string'
    },
    'lastTrxTimestampInMillis': 123,
    'format': 'string',
    'records': [
        {
            'commitTimestampInMillis': 123,
            'eventId': {
                'string': 'string'
            },
            'data': {
                'id': 'string',
                'type': 'string',
                'key': 'string',
                'value': {...}|[...]|123|123.4|'string'|True|None,
                'from': 'string',
                'to': 'string'
            },
            'op': 'string',
            'isLastOp': True|False
        },
    ],
    'totalRecords': 123
}

Response Structure

(dict) –
- lastEventId (dict) –
  
  Sequence identifier of the last change in the stream response.
  
  An event ID is composed of two fields: a commitNum, which identifies a transaction that changed the graph, and an opNum, which identifies a specific operation within that transaction:
  - (string) –
    - (string) –
- lastTrxTimestampInMillis (integer) –
  
  The time at which the commit for the transaction was requested, in milliseconds from the Unix epoch.
- format (string) –
  
  Serialization format for the change records being returned. Currently, the only supported value is PG_JSON.
- records (list) –
  
  An array of serialized change-log stream records included in the response.
  - (dict) –
    
    Structure of a property graph record.
    - commitTimestampInMillis (integer) –
      
      The time at which the commit for the transaction was requested, in milliseconds from the Unix epoch.
    - eventId (dict) –
      
      The sequence identifier of the stream change record.
      - (string) –
        
        (string) –
    - data (dict) –
      
      The serialized Gremlin or openCypher change record.
      - id (string) –
        
        The ID of the Gremlin or openCypher element.
      - type (string) –
        
        The type of this Gremlin or openCypher element. Must be one of:
        
        v1 - Vertex label for Gremlin, or node label for openCypher.
        
        vp - Vertex properties for Gremlin, or node properties for openCypher.
        
        e - Edge and edge label for Gremlin, or relationship and relationship type for openCypher.
        
        ep - Edge properties for Gremlin, or relationship properties for openCypher.
      - key (string) –
        
        The property name. For element labels, this is label.
      - value (document) –
        
        This is a JSON object that contains a value field for the value itself, and a datatype field for the JSON data type of that value:
      - from (string) –
        
        If this is an edge (type = e), the ID of the corresponding from vertex or source node.
      - to (string) –
        
        If this is an edge (type = e), the ID of the corresponding to vertex or target node.
    - op (string) –
      
      The operation that created the change.
    - isLastOp (boolean) –
      
      Only present if this operation is the last one in its transaction. If present, it is set to true. It is useful for ensuring that an entire transaction is consumed.
- totalRecords (integer) –
  
  The total number of records in the response.

Exceptions

NeptuneData.Client.exceptions.UnsupportedOperationException
NeptuneData.Client.exceptions.ExpiredStreamException
NeptuneData.Client.exceptions.InvalidParameterException
NeptuneData.Client.exceptions.MemoryLimitExceededException
NeptuneData.Client.exceptions.StreamRecordsNotFoundException
NeptuneData.Client.exceptions.ClientTimeoutException
NeptuneData.Client.exceptions.PreconditionsFailedException
NeptuneData.Client.exceptions.ThrottlingException
NeptuneData.Client.exceptions.ConstraintViolationException
NeptuneData.Client.exceptions.InvalidArgumentException
NeptuneData.Client.exceptions.IllegalArgumentException
NeptuneData.Client.exceptions.TooManyRequestsException