AI and ML infrastructure is the compute, storage, networking, data and platform layer that AI workloads run on. Most Australian mid-sized businesses will never build it. They consume it through Azure, and what they actually need in place is clean data, sound permissions and a decision about where their data is allowed to go.
Key Takeaways
- Building AI infrastructure is the exception, not the rule. Under a few hundred staff, the question is which Azure services to use, not which GPUs to buy.
- The hard part is rarely the infrastructure. It is data quality, permissions and governance, which are also the cheapest things to get wrong.
- Azure deployment type decides whether your data stays in Australia. This catches people out, and it is covered below.
- Residency has a cost. Keeping inference onshore limits which models you can use, because not every model is available in Australia East.
- Residency is not retention. Azure stores prompts for up to 30 days by default, and turning that off is an application, not a setting.
- Start with the readiness assessment, not the architecture. You cannot size infrastructure for a use case nobody has chosen yet.
What Is AI and ML Infrastructure?
It is four jobs: store the data, process it, move it between the two, and keep track of what the models do. The hardware and software handling those jobs are the infrastructure.
That covers everything from preparing data through to running a model in production.
The terms move around. AI infrastructure, ML infrastructure, AI/ML infrastructure and machine learning infrastructure all describe the same thing.
Where people do draw a line, ML infrastructure means the training and model lifecycle side. AI infrastructure means the wider stack, including the applications built on top.
How It Differs From Traditional IT Infrastructure
Conventional business systems are built for steady, predictable loads. AI workloads are not.
- The compute is bursty. Training demands enormous parallel processing for a period, then nothing. Traditional servers are sized for averages.
- The access patterns differ. Models read entire datasets repeatedly, which punishes storage built for occasional record lookups.
- The pipeline matters more than the hardware. Data has to be collected, cleaned and versioned first. That is usually where projects stall.
Do You Need to Build AI Infrastructure?
Probably not, and it is worth settling early.
Most published material on AI infrastructure is written for organisations building large training environments. GPU clusters. High-bandwidth interconnects. CUDA environments. That is a real discipline, and the writing about it is good.
It is also a different problem from the one most Australian businesses have.
The division is reasonably clear.
| Your situation | What you need |
| Using Copilot, or AI features inside software you already pay for | Nothing. The infrastructure is the vendor’s problem |
| Building applications on existing models through Azure OpenAI or Foundry | A subscription, a region decision, and governance. No hardware |
| Fine-tuning a model on your own data | Managed Azure services. Still no hardware |
| Training models from scratch, or running AI on data that is not permitted to leave your premises | Actual infrastructure, and a genuine reason for it |
Only the last row involves building anything. If you are in the first three, the useful work is data, permissions and governance, not architecture.
This matters most for cost. Buying hardware for an unvalidated use case is the most expensive mistake available here. Pay-as-you-go Azure services let you test the use case before committing capital.
AI Infrastructure Architecture and Components
AI infrastructure architecture has six layers. You will meet all six, whether you build them or rent them.
| Layer | What it does | Who provides it if you use Azure |
| Compute | Processes training and inference workloads | Azure VMs, Azure Machine Learning compute, serverless model endpoints |
| Storage | Holds datasets, model versions and outputs | Azure Blob Storage, Azure Data Lake |
| Networking | Moves data between storage, compute and users | Azure virtual networks, private endpoints |
| Data platform | Prepares, cleans and governs the data | Microsoft Fabric, Azure Data Factory, Azure Databricks |
| Model platform | Trains, tunes, deploys and monitors models | Azure Machine Learning, Microsoft Foundry |
| Governance and security | Controls access, protects data, records decisions | Microsoft Entra ID, Microsoft Purview, Azure Policy |
The layer that causes most trouble is the data platform, not the compute. A model trained on messy data gives confident, wrong answers, and no amount of compute fixes that. If you are deciding where to start, our data analytics and reporting work usually comes first.
AI Infrastructure in the Cloud with Azure
For Australian businesses, the practical route to AI infrastructure is Azure, because the identity, data and security layers are already there.
The services that do the work:
- Microsoft Foundry, formerly Azure AI Foundry, for building and deploying applications on existing models. Microsoft’s region support documentation lists Australia East among the regions where Foundry projects can be created.
- Azure Machine Learning for training, tuning and managing your own models.
- Microsoft Fabric for the data platform underneath.
- Azure Databricks where large-scale data engineering is involved.
- Microsoft Purview for governing what any of it can reach.
Most businesses starting out need the first and the third. The rest arrives later, if at all.
The Deployment Type That Catches Australian Businesses Out
This is the detail worth knowing before you commit, and it is not obvious from the product pages.
When you deploy a model in Microsoft Foundry, you choose a deployment type, and that choice decides where your prompts are processed. Microsoft’s documentation on region availability for Foundry models sets out three.
- Global: may be processed in any Azure region where the model is deployed.
- Data Zone: processed anywhere within the Microsoft-specified data zone, which is the United States, the European Union, or Asia Pacific.
- Standard or Regional: processed in the region tied to your deployment.
The trap is the middle one. “Data Zone” sounds like a residency setting. For Australia it is not, because the Asia Pacific data zone spans multiple countries. Choosing it does not keep processing in Australia.
If your requirement is that data stays onshore, you need a Standard or Regional deployment in an Australian region. Data stored at rest stays in the designated Azure geography across all three types, but processing does not. Source: Microsoft Learn, region availability for Foundry Models sold by Azure.
What Residency Costs You
The trade-off is model availability. Not every model is offered in Australia East. Microsoft’s own guidance records realtime model families as unavailable there, with those models documented for United States and European regions instead. Model availability changes, so treat any specific example as a snapshot.
So the choice is real: onshore processing, or access to the newest models. For a business handling personal or regulated information, the first usually wins.
What matters is that it is a decision, not something you discover three months in. Check the Foundry model catalogue from your own subscription in Australia East before designing anything around a specific model. Microsoft publishes availability by region and deployment type, and that page is the one to trust.
This connects directly to your obligations under the Privacy Act 1988 and Australian Privacy Principle 8, which governs disclosure of personal information overseas.
What Microsoft Keeps, and for How Long
Residency answers where your data is processed. It does not answer what is kept afterwards, and that is a separate question worth asking.
By default, Azure OpenAI retains prompts and completions for up to 30 days for abuse monitoring. Microsoft’s abuse monitoring documentation sets out the detail.
The stored data sits in your Azure region and is isolated to your subscription. It is not used to train Microsoft’s models, OpenAI’s models or anyone else’s. Automated systems review it, and a human reviewer may see a sample where something is flagged.
For most businesses that is acceptable. For some it is not, and the distinction matters if you handle health records, NDIS participant information or financial data.
The way out is a program called modified abuse monitoring. Approved subscriptions do not have prompts and completions stored in that monitoring path, which is what people usually mean by zero data retention on Azure.
Two things to know before you plan around it.
It is an application, not a setting. There is no toggle in the portal, and Microsoft approves it through its Limited Access process.
It is also reported to require an Enterprise Agreement or Microsoft Customer Agreement rather than pay-as-you-go, which puts it out of reach for many smaller organisations. Confirm your own eligibility before promising it to a compliance team.
One more trap. Approval covers Microsoft’s abuse monitoring store. It does not cover anything your own application writes to Application Insights, Azure Monitor or a storage account. Plenty of organisations secure the model and then log the prompts themselves.
AI for Infrastructure, or Infrastructure for AI?
These two phrases get used interchangeably. They mean opposite things.
Infrastructure for AI is the subject of this page: the compute, storage and platform layer that AI workloads run on.
AI for infrastructure is the reverse. It means using AI to run IT operations: spotting anomalies in monitoring data, predicting capacity problems, triaging alerts, automating routine fixes. The industry term is AIOps.
They need different things from you. Infrastructure for AI starts with a data and governance question. AI for infrastructure starts with whether your monitoring and logging are good enough to learn from, which in most mid-sized environments they are not.
If you came looking for the second, this page is about the first.
Running AI Locally: When On-Premises Makes Sense
Local AI infrastructure, sometimes called local-first AI, means running models on hardware you control rather than in a public cloud.
It makes sense in a few cases. Where data cannot leave your premises for contract or regulatory reasons. Where latency rules out a round trip. Or where steady, high usage makes paying per call worse than owning the hardware.
It is a poor default. You take on the capital cost, the capacity planning, the patching and the physical security, and you lose the ability to change your mind cheaply. For most Australian mid-sized businesses, an Australian-region cloud deployment answers the sovereignty question without any of that.
If you are weighing it up, our cloud migration and modernisation work covers where workloads are best placed.
What Australian Businesses Need in Place Before Adopting AI
Before infrastructure becomes the question, five things need to be true.
- Your data is somewhere you can find it. Scattered across file shares, mailboxes and someone’s desktop is the normal starting point and the first thing to fix.
- Permissions reflect reality. AI tools surface whatever the signed-in user can already open. Years of permission drift become visible the moment you switch one on.
- Someone owns the outcome. Who is accountable when an AI output is wrong, and who decides what data may be used.
- You know which processes are worth the effort. Frequent, slow and tolerant of review. Not rare and expensive to get wrong.
- You have decided where data may go. See the deployment type section above.
Four of the five are governance, not technology. That is the shape of AI readiness at this size, and it is why the assessment comes before the architecture. Our guide to Microsoft 365 workflows covers the adoption side, where most practical value shows up first.
AI and ML Development Services: What to Look For
Looking for an AI and ML development service rather than building internally? The selection criteria matter more than the technology list.
- Ask what they will not do. A provider who claims every capability is describing a brochure, not a team.
- Ask where your data will be processed, and expect a specific region and deployment type in the answer, not reassurance.
- Ask who owns the model and the pipeline at the end of the engagement.
- Ask what happens when it breaks. Who monitors it, who is called, and under what agreement.
- Ask for the readiness assessment first. Any provider willing to start building before assessing your data is selling you risk.
Planning and Advisory Support
The sequence that works is unglamorous. Assess what your environment can support. Pick two or three use cases worth the effort. Settle the governance and residency position. Then size the infrastructure.
Done in that order, the infrastructure question tends to answer itself. The answer is usually a managed Azure service, not something you need to own.
Our Data and AI services cover the data platform and analytics side of that work.
AI strategy consulting is where that assessment happens. If you would rather start with a conversation than a proposal, get in touch.
Frequently Asked Questions
AI infrastructure is the compute, storage, networking, data platform and model platform layer that AI workloads run on, along with the governance controls around it. It can be built and owned, or consumed as managed cloud services. For most businesses the second is the practical option.
In practice they describe the same stack. Where people draw a line, ML infrastructure refers to the training and model lifecycle side, including data pipelines, experiment tracking and model deployment. AI infrastructure is used more broadly, taking in the applications built on top of those models.
Usually not. Using AI features in software you already pay for makes the infrastructure the vendor’s problem. Building on existing models through Azure makes it a managed service. Either way the real work is governance and data preparation, and building matters only when you train from scratch or data cannot leave your premises.
Only if you choose the right deployment type. Standard and Regional deployments in an Australian region process data in that region. Global deployments may process anywhere the model runs, and Data Zone deployments process anywhere in the Asia Pacific zone, which spans several countries. Data at rest stays in the designated Azure geography either way, but processing does not.
Six layers: compute, storage, networking, the data platform, the model platform, and governance and security. On Azure each maps to a managed service, so the design question is usually which services and which region rather than which hardware.
Five things. Data you can find, permissions that match who should see what, and a named owner for AI outputs. Then a shortlist of use cases worth the effort, and a decision on where data may be processed. Four of the five are governance rather than technology, which is why the readiness assessment comes before any infrastructure decision.




