Enterprise ML Architecture: Making Every Boundary Earn Its Place
Seven Workload Tests for Better Architecture Decisions
Enterprise ML architectures carry history. Many were built when the data platform handled storage, transformation and analytics, while a separate ML platform supplied notebooks, training infrastructure, registries, deployment pipelines, online endpoints, monitoring and specialist compute. Those choices created skills, controls and production services that may have improved over years. Snowflake now provides capabilities across more stages of the ML lifecycle. These capabilities now include feature management, development and job execution, model governance, warehouse and container-based inference, observability, explainability and lineage. The platform can also work with models trained elsewhere and with inference performed outside Snowflake. These broader capabilities merit reassessment, not automatic migration.
A platform gaining a capability is not, by itself, an architecture justification for moving a workload. Placement should follow workload evidence: data movement, execution pattern, latency, feature reuse, operating maturity, prediction consumption, governance and total cost. Here is a set of seven tests to structure that discussion and decision making.
Test 1: Does data movement materially affect the workload?
“The data is already in Snowflake” is not a sufficient reason to run a machine learning workload there. Locality matters when the volume, frequency and operational consequences of crossing the platform boundary are substantial.
As an example, consider a churn model that scores 50 million customer records each night. Source data and features reside in Snowflake, and the probabilities return there for campaign segmentation. External scoring creates a round trip; extract a large feature population, transfer it to another service, score it, move the predictions back, reconcile the result and recover from failures across two platforms. Running inference closer to the governed data may remove meaningful work.
Contrast that with a mature low-latency endpoint receiving a vector of 20 precomputed features from an operational application. The payload is small, the prediction is consumed immediately by the application, and the service already meets latency, scaling, monitoring and release requirements. Moving it merely because another platform can host inference may replace a well-run boundary without solving the material problem.
Measure the boundary rather than treating locality as an architectural virtue.
Test 2: What are we paying to simplify?
Compute rates are only one part of the comparison. A credible ML total-cost view includes compute, data movement, duplicated feature logic, integration, platform operations, governance, skills and change cost.
Direct infrastructure may be economical when utilisation is high, commercial commitments are favourable and the operating team is effective. Managed execution may cost more per unit yet reduce orchestration, staging, reconciliation and support effort. Neither result can be assumed from a price list.
For container workloads, Snowpark Container Services uses managed compute pools, while storage and data transfer remain separate cost considerations. Warehouse inference has a different operating and pricing profile again. Each option must be tested against the workload’s region, service objective, concurrency, scaling pattern and utilisation.
Change cost deserves equal weight. Rebuilding a mature service can require retraining teams, recreating security and approval controls, altering CI/CD and on-call ownership, and accepting transition risk. Conversely, a batch pipeline may justify change if it removes repeated extraction, feature duplication and cross-platform reconciliation. Benchmark the workload, not an abstract platform configuration.
Test 3: Is the workload batch, online or specialised?
Inference covers operating patterns with different architecture needs. A nightly propensity run, a millisecond-sensitive fraud decision and a GPU-intensive vision model should not share a placement rule.
Snowflake is a candidate for large batch scoring over Snowflake tables, forecasting across many entities, anomaly detection embedded in data pipelines and model inference executed from Snowflake compute. Warehouse inference can reduce movement when the scoring population and downstream consumers already sit in Snowflake. Models do not need to be trained there first; externally trained models can be registered and invoked where supported.
Online and specialised workloads need harder evidence. Latency, throughput, concurrency, availability, autoscaling, network path, regional deployment, accelerator needs, custom runtimes and burst behaviour can outweigh locality. Snowpark Container Services expands the available execution choices, including specialised compute, but there is a need to establish workload suitability and price-performance.
Existing production services face a higher hurdle for change. Proven latency, effective monitoring, mature deployment automation, security controls and clear operational ownership are assets. A migration must produce measurable value beyond architectural symmetry.
The result may differ by execution mode. One model can use warehouse inference for scheduled portfolio scoring while retaining a real-time endpoint for transactional decisions. Shared model logic does not require a single serving pattern.
Test 4: Where should features live?
Feature placement can be more consequential than model placement. Recreating features independently produces inconsistent definitions, point-in-time leakage, stale values, training-serving skew and unclear lineage.
A governed feature layer in Snowflake can be valuable when the underlying data already resides there and several teams reuse the same transformations. Feature Store can centralise feature definitions and governed access while supporting configured refresh for feature data. These capabilities change the decision only if they reduce duplicate engineering and establish reliable ownership.
Centralising features does not require centralising the whole lifecycle. A sound hybrid design can keep governed data and reusable features in Snowflake, use a specialised training environment, deploy to the serving environment that meets operational needs, and return predictions or ground truth to Snowflake for governed consumption and analysis.
Get Anupama Gangadhar’s stories in your inbox
Join Medium for free to get updates from this writer.
Online features require separate scrutiny. Their freshness, access path and latency may call for a dedicated serving layer even when offline definitions originate in Snowflake. The architecture should preserve semantic consistency without forcing every read through the same runtime. Consolidate feature logic when it creates reuse, point-in-time correctness and accountable lineage. Do not build a feature store merely because the platform offers one.
Test 5: Where should training, governance, execution and observation occur?
Training, governance, execution and observation are separate placement decisions. Treating them as one platform choice hides the requirements that differ across the lifecycle.
Training may depend on experiment velocity, distributed compute, accelerators, open-source dependencies and existing team practice. Governance depends on ownership, versioning, approval, access control, lineage and promotion between environments. Model Registry can govern supported models, including models trained externally, without requiring their development history to move.
Execution should follow the serving mode. Warehouse inference and Snowpark Container Services provide different execution options, but both remain workload-specific choices.
Observation depends on where predictions and outcomes can be joined reliably. For models monitored through Snowflake ML Observability, stored predictions, features and ground truth can be used to assess model performance, drift and prediction volume. That supports a useful hybrid pattern: an external service produces online predictions, Snowflake stores those predictions, ground truth arrives later, and monitoring runs against governed data. Explainability and lineage are useful where they support review, investigation or regulation; they should not be added as generic platform features.
As an example, the lifecycle can be deliberately split: specialised training outside Snowflake, governance in Model Registry, batch inference in a warehouse, online serving externally, and observation against data in Snowflake.
Test 6: Where are predictions consumed?
Prediction destination can matter as much as training-data location. If a batch score returns immediately to a Snowflake customer table for segmentation, analytics or governed activation, an external inference boundary should demonstrate the value it adds. Otherwise it may create movement and orchestration without changing the business outcome.
The answer is different when an operational application sends a small request to a real-time endpoint and acts on the response immediately. If Snowflake is neither the feature-serving path nor the prediction consumer, relocating inference may add a network hop or new dependency with little offsetting benefit.
Consumption also determines the recovery model. Batch predictions written into governed tables can be reconciled, rerun and audited with data pipelines. Transactional predictions may need synchronous fallbacks, strict availability and application-level telemetry. Those are different operating contracts even when the model artefact is related.
A single model can therefore support more than one execution path. A risk model might score a full portfolio overnight near the data while an external service handles new transactions. Consistent model and feature governance matters; identical infrastructure does not. Always follow the prediction into the business process. Its destination reveals the meaningful latency, security, orchestration, reconciliation and monitoring requirements.
Test 7: Does an agentic workflow change the placement decision?
Predictive models estimate probabilities, forecasts, rankings or expected outcomes from historical patterns. CoCo can improve development work such as exploration, feature engineering, training, evaluation, debugging and deployment preparation. CoCo capabilities are evolving, so its influence must be carefully considered after validating its availability for the target environment.
None of this dictates inference placement. An agent operating near Snowflake data may call a model served elsewhere because that endpoint still provides the right latency, runtime or operating model. Conversely, a Snowflake-resident predictive signal may reduce movement when the agent and its governed context already consume data there.
The placement question remains unchanged: which boundary best satisfies the predictive workload and the business action? Agentic workflows increase the value of reliable, governed signals; they do not remove the need to benchmark how those signals are produced and served.
Turning the tests into an architecture decision
Evidence should include observed workload behaviour rather than stakeholder preference: representative volumes, latency distributions, utilisation, failure and recovery experience, operational effort, control requirements and commercial assumptions. Where the evidence is incomplete, the recommendation should remain conditional rather than being forced into a platform decision.
The seven tests structure a conversation; they do not produce a generic suitability percentage. Constraints are asymmetric. A hard latency objective, regulatory restriction, specialised runtime or mature production service cannot sensibly be averaged away by several weaker benefits.
Four outcomes provide a useful decision vocabulary:
- Keep — Retain the architecture when the existing boundary earns its cost through proven service levels, mature operations, suitable specialist infrastructure, effective governance or predominantly external consumption. A credible review must be able to recommend no migration.
- Integrate — Improve selected interfaces without replacing working capabilities. Govern features in Snowflake while retaining specialised training; keep an online service while returning predictions and ground truth for observation; or use warehouse inference for batch scoring while preserving a separate transactional path.
- Consolidate — Move the lifecycle components that show measurable benefit. Strong candidates combine a large Snowflake-resident scoring population, repeated boundary movement, batch execution, Snowflake-derived features, Snowflake-centric consumption and manageable change cost.
- New Build — For a new workload centred on governed Snowflake data, reusable features, batch execution and Snowflake consumption, evaluate a Snowflake-based design before adding another platform boundary. Another platform may still win, but the boundary should have a demonstrated purpose.
Snowflake’s expanding ML capabilities create more placement choices, not a reason to consolidate by default. Existing investments, workload characteristics and operating constraints still matter, and the right architecture may be consolidated, external or deliberately hybrid. The decision should follow evidence from the workload tests and the value created by changing the boundary, not the availability of a platform capability. The responsibility is not to choose a platform first. It is to make every architecture boundary earn its cost.
