SageMaker / Client / batch_replace_cluster_nodes

batch_replace_cluster_nodes¶

SageMaker.Client.batch_replace_cluster_nodes(**kwargs)¶

Replaces specific nodes within a SageMaker HyperPod cluster with new hardware. BatchReplaceClusterNodes terminates the specified instances and provisions new replacement instances with the same configuration but fresh hardware. The Amazon Machine Image (AMI) and instance configuration remain the same.

This operation is useful for recovering from hardware failures or persistent issues that cannot be resolved through a reboot.

Warning

Data Loss Warning: Replacing nodes destroys all instance volumes, including both root and secondary volumes. All data stored on these volumes will be permanently lost and cannot be recovered.
To safeguard your work, back up your data to Amazon S3 or an FSx for Lustre file system before invoking the API on a worker node group. This will help prevent any potential data loss from the instance root volume. For more information about backup, see Use the backup script provided by SageMaker HyperPod.
If you want to invoke this API on an existing cluster, you’ll first need to patch the cluster by running the UpdateClusterSoftware API. For more information about patching a cluster, see Update the SageMaker HyperPod platform software of a cluster.
You can replace up to 25 nodes in a single request.

Request Syntax

response = client.batch_replace_cluster_nodes(
    ClusterName='string',
    NodeIds=[
        'string',
    ],
    NodeLogicalIds=[
        'string',
    ]
)

Parameters:

ClusterName (string) –
[REQUIRED]

The name or Amazon Resource Name (ARN) of the SageMaker HyperPod cluster containing the nodes to replace.
NodeIds (list) –
A list of EC2 instance IDs to replace with new hardware. You can specify between 1 and 25 instance IDs.

Warning
Replace operations destroy all instance volumes (root and secondary). Ensure you have backed up any important data before proceeding.
Note
- Either NodeIds or NodeLogicalIds must be provided (or both), but at least one is required.
- Each instance ID must follow the pattern i- followed by 17 hexadecimal characters (for example, i-0123456789abcdef0).
- For SageMaker HyperPod clusters using the Slurm workload manager, you cannot replace instances that are configured as Slurm controller nodes.
- (string) –
NodeLogicalIds (list) –
A list of logical node IDs to replace with new hardware. You can specify between 1 and 25 logical node IDs.

The NodeLogicalId is a unique identifier that persists throughout the node’s lifecycle and can be used to track nodes that are still being provisioned and don’t yet have an EC2 instance ID assigned.
Warning
- Replace operations destroy all instance volumes (root and secondary). Ensure you have backed up any important data before proceeding.
- This parameter is only supported for clusters using Continuous as the NodeProvisioningMode. For clusters using the default provisioning mode, use NodeIds instead.
- Either NodeIds or NodeLogicalIds must be provided (or both), but at least one is required.
- (string) –

Return type:

dict

Returns:

Response Syntax

{
    'Successful': [
        'string',
    ],
    'Failed': [
        {
            'NodeId': 'string',
            'ErrorCode': 'InstanceIdNotFound'|'InvalidInstanceStatus'|'InstanceIdInUse'|'InternalServerError',
            'Message': 'string'
        },
    ],
    'FailedNodeLogicalIds': [
        {
            'NodeLogicalId': 'string',
            'ErrorCode': 'InstanceIdNotFound'|'InvalidInstanceStatus'|'InstanceIdInUse'|'InternalServerError',
            'Message': 'string'
        },
    ],
    'SuccessfulNodeLogicalIds': [
        'string',
    ]
}

Response Structure

(dict) –
- Successful (list) –
  
  A list of EC2 instance IDs for which the replacement operation was successfully initiated.
  - (string) –
- Failed (list) –
  
  A list of errors encountered for EC2 instance IDs that could not be replaced. Each error includes the instance ID, an error code, and a descriptive message.
  - (dict) –
    
    Represents an error encountered when replacing a node in a SageMaker HyperPod cluster.
    - NodeId (string) –
      
      The EC2 instance ID of the node that encountered an error during the replacement operation.
    - ErrorCode (string) –
      
      The error code associated with the error encountered when replacing a node.
      
      Possible values:
      - InstanceIdNotFound: The instance does not exist in the specified cluster.
      - InvalidInstanceStatus: The instance is in a state that does not allow replacement. Wait for the instance to finish any ongoing changes before retrying.
      - InstanceIdInUse: Another operation is already in progress for this node. Wait for the operation to complete before retrying.
      - InternalServerError: An internal error occurred while processing this node.
    - Message (string) –
      
      A human-readable message describing the error encountered when replacing a node.
- FailedNodeLogicalIds (list) –
  
  A list of errors encountered for logical node IDs that could not be replaced. Each error includes the logical node ID, an error code, and a descriptive message. This field is only present when NodeLogicalIds were provided in the request.
  - (dict) –
    
    Represents an error encountered when replacing a node (identified by its logical node ID) in a SageMaker HyperPod cluster.
    - NodeLogicalId (string) –
      
      The logical node ID of the node that encountered an error during the replacement operation.
    - ErrorCode (string) –
      
      The error code associated with the error encountered when replacing a node by logical node ID.
      
      Possible values:
      - InstanceIdNotFound: The node does not exist in the specified cluster.
      - InvalidInstanceStatus: The node is in a state that does not allow replacement. Wait for the node to finish any ongoing changes before retrying.
      - InstanceIdInUse: Another operation is already in progress for this node. Wait for the operation to complete before retrying.
      - InternalServerError: An internal error occurred while processing this node.
    - Message (string) –
      
      A human-readable message describing the error encountered when replacing a node by logical node ID.
- SuccessfulNodeLogicalIds (list) –
  
  A list of logical node IDs for which the replacement operation was successfully initiated. This field is only present when NodeLogicalIds were provided in the request.
  - (string) –

Exceptions

SageMaker.Client.exceptions.ResourceNotFound