Development of an AI Platform based on Microsoft Azure

Inside the rollout

We designed and implemented an enterprise-wide AI platform based on Azure AI Foundry and AI Factory. The architecture allows all business units to securely use AI services – with central governance, uniform guardrails, and full cost transparency. The platform was built according to the "Secure by Default" principle: Private Endpoints, Entra ID integration, role-based access control, and Azure Policies ensure that compliance requirements are automatically met.

Before implementation, we conducted extensive benchmarks: Different LLM models - including Azure OpenAI GPT variants and Microsoft Copilot - were tested for response quality, latency, cost, and suitability for the specific use cases. The results flowed directly into the architectural decisions. As the first productive solution, we developed an Enterprise Search based on LangChain. SharePoint, ServiceNow, and SAP were connected as source systems and indexed in Azure AI Search. Via a chat-based web app, employees can ask questions in natural language and receive answers from all connected systems – with source citations and taking existing permissions into account. The result: A scalable AI platform with well-founded model selection and an immediately noticeable added value – enterprise knowledge accessible in seconds, based on the optimal LLM for the respective use case.

Results at a glance

Connected source systems

Three systems

SharePoint, ServiceNow, and SAP in the enterprise search

Central AI platform

One platform

uniform governance and guardrails for all business units

Cost transparency

100 %

central control of all AI services through the platform

35% more accurate forecasts

Learnings

• LLM benchmarks before the architecture decision pay off: response quality, latency, and cost differ considerably per use case.
• "Secure by default" prevents shadow IT more effectively than any ban — a secure self-service offering makes isolated AI experiments unnecessary.
• Enterprise search is the ideal first use case: immediately noticeable value for all business units, without having to replace source systems.
• RAG only works in the enterprise with permissions: answers must respect the existing access rights of the source systems.

What's next

The platform has been in production since December 2024 and is being expanded continuously. Based on the central governance, additional business units can implement their own AI use cases without re-clarifying security and compliance questions every time. Connecting additional source systems to the enterprise search follows the same pattern as with SharePoint, ServiceNow, and SAP. New LLM generations are evaluated against the established benchmarks and introduced deliberately — model selection thus remains an informed decision rather than a gut feeling.

Service: AI & Custom Development