Salesforce Achieves Multi-AZ High Availability with SageMaker Inference Components

Salesforce has enhanced the resilience of its Agentforce AI models by leveraging Amazon SageMaker's Inference Components (ICs) placement controls. The company aimed to meet strict Multi-AZ high availability (HA) requirements, which are critical for production AI workloads.

Default IC placement in SageMaker optimizes for cost and distributes model copies evenly across instances but does not consider AZ balance, potentially creating single points of failure at the AZ level. To address this, Salesforce employed the new SchedulingConfig parameter within the CreateInferenceComponent API, enabling precise control over IC copy placement across instances and AZs.

The key parameters—AvailabilityZoneBalance and PlacementStrategy—allow Salesforce to specify whether copies should be spread across instances (SPREAD) or packed (BINPACK), and to enforce AZ balance with configurable tolerance. For example, deploying four IC copies across four instances in two AZs with SPREAD and MaxImbalance set to 1 ensures that copies are evenly distributed, with at most a one-copy difference between AZs.

This approach supports scaling operations while maintaining AZ balance, with SageMaker automatically adjusting placement during scale-in and scale-out activities. For ongoing maintenance, Salesforce configured the endpoint’s ScaleInPolicy with the CONSOLIDATION strategy, enabling background rebalancing of IC copies.

The new placement algorithm considers the final distribution, ensures availability-aware placement, and optimizes within-AZ distribution strategies, significantly improving fault isolation and HA compliance. Salesforce’s deployment architecture now guarantees that no single instance or AZ failure can take down their models, fulfilling their 2-AZ support mandate.

Monitoring is facilitated through SageMaker AI Insights, which provides metrics on AZ skew, IC copy distribution, rebalancing events, and capacity errors, enabling proactive management of deployment health.

This implementation demonstrates that enterprise AI teams can achieve high availability without compromising cost efficiency by explicitly controlling IC placement. Salesforce’s experience offers a valuable reference for organizations deploying critical AI workloads on SageMaker, emphasizing the importance of precise placement controls, capacity planning, and continuous observability.