SageMaker MME is like a coffee machine: one machine, many flavors — one endpoint, many models
AWS SageMaker
Multi Model Endpoint (MME)
Introduction
Imagine you wanted to try coffee from Starbucks for your morning ritual and Blue Tokai for afternoon meetings, but your coffee maker refuses to brew according to your whims. SageMaker MME(Multi-model-endpoint) is like the coffee machine — one machine that can brew different flavours but only pods designed for that system will fit. To get convenience of the same basic underlying shared infrastructure like memory, compute and operating system but host different models is what Amazon SageMaker Multi-model-endpoint about.
What is a Multi-Model-Endpoint?
Get Shashwati’s stories in your inbox
Join Medium for free to get updates from this writer.
A multi model endpoint is a single CPU/GPU backed endpoint which provides shared memory, compute and power to run the different categories of model for inference. For example, if we want to use a Logistic Regression or Decision Tree algorithm, we can pull the corresponding model from S3 bucket in memory without provisioning separate memory or compute for the same.
With different s3 buckets hosting multiple model types like ones specific to TensorFlow or PyTorch makes it a one-stop-shop for multi-purpose models. This reduces hosting cost by improving endpoint utilization instead of using multiple endpoints specifically designed for different model types. Yes, in that case there is no dedicated model-specific endpoint available all the time. However, we can bring frequently used models to cache and delete the Least Recently Used ones based on usage. The application trying to access an infrequently used model, should be tolerant to cold starts. There are several methods to eliminate cold start latency as well.
Different models like Ollama provides a ‘keep-alive’ control which can define how long until the model version is unloaded from memory so that subsequent requests produce consistent results with the same model version if needed for its computation.
Life Cycle of a Model in MME
What kind of metrics do we want to look into?
CloudWatch metrics for CPU/GPU backed Multi model endpoint are available to keep track on model loading time, model unloading time and CPU and disk utilization on model instances.
Conclusion
MMEs are not just about saving infrastructure costs but also useful for experimentation with wide range of models. This helps teams deploy smarter and faster models across diverse data sizes and types— combining both convenience and control. Monitoring key metrics and comparing model performance helps users decide on the most optimal solution — much like indulging in their favourite coffee brew, after all “Coffee is always a good idea”.
