This is why you should use the NEW AI Gateway — Azure API Management!
Are you sure your AI models are secure? Who has access to them? Can you control which applications interact with your models? And do you know how much data and how many tokens are flowing through your AI infrastructure? As organizations increasingly integrate AI into their applications, agents, and business processes, managing and securing access to these models becomes just as important. This is where AI Gateway comes into play. Built on Azure API Management, it introduces a dedicated gateway layer designed to bring control, security, and visibility to your AI ecosystem. While Microsoft Foundry provides the platform to build, manage, and deploy AI models and applications, AI Gateway takes care of what happens around them. It provides centralized governance over how applications access models and tools, allowing platform teams to enforce policies, manage traffic, control consumption, and monitor usage. In this blog, we’ll explore how to deploy Azure AI Gateway and connect it to Microsoft Foundry, creating a more controlled and governed foundation for your AI workloads.
For this blog, I aimed to automate as much of the deployment process as possible. However, since AI Gateway is currently in public preview, Microsoft is still working on the underlying API specifications. Therefore, for this demo, we will focus on deploying Microsoft Foundry, the Foundry project, and the model deployment using Bicep. The AI Gateway configuration itself will be configured through the AI Gateway Portal. Once the APIs become more mature and future-proof, I’ll publish a second part covering this aspect in more detail. As I work exclusively with Microsoft Azure, I use Azure Bicep for infrastructure deployment. The main.bicep template serves as the entry point for our Bicep configuration, defining the parameters and variables used throughout the Microsoft Foundry infrastructure module.
using 'main.bicep'
param parAlzName = 'corp'
param parEnvironment = 'prod'
param parLocation = 'swedencentral'
param parDeploymentNumber = '001'The main.bicep template serves as the entry point for our Bicep configuration and defines the parameters and variables used across the Microsoft Foundry module.
metadata name = 'Microsoft Foundry & Models'
metadata description = 'This is a Bicep template to deploys Microsoft Foundry, Including Foundry Project and two AI-models <3'
metadata owner = 'Brian Veldman'
metadata version = '1.0.0'
/* TARGET SCOPE */
targetScope = 'subscription'
@description('Defining our input parameters')
param parEnvironment string
@allowed([
'swedencentral'
])
param parLocation string
param parAlzName string
param parDeploymentNumber string
@description('Defining our variables')
var shortEnvironmentMapping = {
swedencentral: 'sdc'
}
var shortEnvironment = shortEnvironmentMapping[parLocation] ?? parLocation
var resourceGroupName = 'rg-${parAlzName}-${parEnvironment}-${shortEnvironment}-${parDeploymentNumber}'
@description('Deploying our resourcegroup')
resource rg 'Microsoft.Resources/resourceGroups@2025-04-01' = {
name: resourceGroupName
location: parLocation
}
@description('Deploying our AI module')
module ai 'modules/ai.bicep' = {
params: {
parAlzName: parAlzName
parEnvironment: parEnvironment
parLocation: parLocation
parDeploymentNumber: parDeploymentNumber
shortEnvironment: shortEnvironment
}
scope: rg
}The ai.bicep module automates the deployment of Microsoft Foundry, ❤️ including the Foundry resource, a dedicated Foundry project, and multiple AI model deployments. In this example, we’re deploying two models: GPT-6 and Grok 4.7. Both models are deployed using the Global Standard deployment type, with configurable capacity to handle our expected workloads.
param parLocation string
param parAlzName string
param parEnvironment string
param shortEnvironment string
param parDeploymentNumber string
@description('Deployment of Foundry')
resource aif 'Microsoft.CognitiveServices/accounts@2025-06-01' = {
name: 'aif-${parAlzName}-${parEnvironment}-${shortEnvironment}-${parDeploymentNumber}'
location: parLocation
kind: 'AIServices'
sku: {
name: 'S0'
}
properties: {
networkAcls: {
defaultAction: 'Allow'
}
allowProjectManagement: true
customSubDomainName: 'sub-aif-${parAlzName}-${parEnvironment}-${shortEnvironment}-${parDeploymentNumber}'
}
}
@description('Deployment of Foundry Project')
resource aip 'Microsoft.CognitiveServices/accounts/projects@2025-06-01' = {
name: 'proj-aif-${parAlzName}-${parEnvironment}-${shortEnvironment}-${parDeploymentNumber}'
parent: aif
location: parLocation
identity: {
type: 'SystemAssigned'
}
}
@description('Deployment of GPT 6 Sol Model')
resource gptModel 'Microsoft.CognitiveServices/accounts/deployments@2025-06-01' = {
name: 'gpt-6-sol-deployment'
parent: aif
sku: {
name: 'GlobalStandard'
capacity: 499
}
properties: {
model: {
name: 'gpt-6-sol'
format: 'OpenAI'
version: '2026-09-22'
}
}
dependsOn: [
aip
]
}
@description('Deployment of Grok 4.7 Model')
resource grokModel 'Microsoft.CognitiveServices/accounts/deployments@2025-06-01' = {
name: 'grok-4-7-deployment'
parent: aif
sku: {
name: 'GlobalStandard'
capacity: 499
}
properties: {
model: {
name: 'grok-4.7'
format: 'xAI'
version: '1'
}
}
dependsOn: [
aip
]
}You can deploy the Bicep Infra using the following PowerShell script. 💪🏻
New-AzSubscriptionDeployment `
-Name 'ai-foundry-lab' `
-Location 'swedencentral' `
-TemplateFile '.\main.bicep' `
-TemplateParameterFile '.\main.bicepparam'When the Microsoft Foundry resources are deployed it is time to set up the AI Gateway, which will interact with our Microsoft Foundry resources, including our GPT-6 and Grok AI model. To get started, navigate to the AI Gateway Portal and sign in using your account.
Fill in the required details, including the name of your AI Gateway. In this example, we’re using corp-r-prod-sdc-001 to follow our existing resource naming conventions. Keep in mind that the AI Gateway resource name is limited to 24 characters, so make sure your chosen name stays within this limit.
Now that we’re in the AI Gateway portal, it’s time to add an AI model. We’ve already deployed Microsoft Foundry with a project containing two model deployments. We’ll add these models to the AI Gateway by selecting “Add from Microsoft Foundry”.
The AI Gateway automatically discovers the available models deployed in Microsoft Foundry within your Azure subscription. All we need to do is select the models we want to expose and click “Add models”.
Get Brian Veldman’s stories in your inbox
Join Medium for free to get updates from this writer.
Now we can manage aliases, backend models, and model providers. It’s time to put our AI Gateway to the test using the Grok 4.7 model we’ve deployed.
Best-practice is to use Entra ID Authentication, for that we need a role assignment within the AI Gateway.
Next, you can define the scope of the role assignment. In this example, I want to grant a specific Microsoft Entra ID user, myself, access to selected resources and a specific model. You can also choose which role to assign, depending on the level of access required. For example, you can grant permission to invoke models only, or extend access to include MCP servers configured in the AI Gateway. 🔥
To test the AI Gateway using cURL with Entra ID authentication, I asked the Grok model the following question: “Why is pizza so delicious? Answer me in three words.” After waiting a few seconds, we received the response: “Cheese sauce perfection.” This confirms that our AI Gateway is working as expected and successfully routing requests to the Grok model. We can also verify this by checking the model property in the API response, which confirms which model (Grok-4.7) handled the request. 😎
Now that our AI Gateway is up and running, we can configure policies based on our organization’s requirements. For example, if we want to restrict access to the AI Gateway based on IP addresses, we can configure IP filtering to control which clients are allowed to send requests.
We can for example scope our IP Filter on our entire AI Gateway or individual assets, such as selected models, aliases or MCP servers.
In this example, I’m allowing traffic from my HQ IP address using a /32 prefix. By selecting the “Allow” action, only the IP addresses explicitly defined in the configuration are permitted to interact with the backend resources accessed through the AI Gateway.
When testing the AI Gateway again, I’m unable to invoke the AI model, even though I’m using the Microsoft Entra ID user that has been granted access. The reason is that my current outbound IP address does not match the IP address configured in the IP filter, which confirms our configuration is working as expected!
I also explored whether we could prevent clients from accessing the model endpoints directly through Microsoft Foundry by configuring Private Endpoints between the AI Gateway and Microsoft Foundry. However, Private Endpoint support for the AI Gateway is not available yet. This is definitely something worth exploring in a future blog!
What about pricing? Since AI Gateway is currently in public preview, it’s available free of charge in the East US 2 and Sweden Central Azure regions. Of course, this will likely change in the future, but for now, I can experiment with it completely free of charge. Isn’t that nice? 😍
From now on, you know how to deploy Microsoft Foundry and configure an AI Gateway to centrally manage access to your AI models. By combining Microsoft Foundry with the AI Gateway, you can introduce an additional layer of governance, security, and control over your AI workloads. From managing model access through Microsoft Entra ID to configuring IP filtering and controlling model usage with policies, you have the building blocks to create a more secure and manageable AI environment. As organizations increasingly adopt generative AI, having control over how models are accessed and consumed becomes more important than ever. The Azure Bicep code is available in my GitHub repository: GitHub — brianveldman/azure-bicep. Feel free to star the repository and clap for this story. That’s all for today. I’d love to hear your feedback, and I look forward to our next session. Until then, take care. ⭐️
