The Case for Custom, Sovereign AI in the Public Sector
Agentic AI has entered the workplace across sectors, meaning instead of simply responding to prompts, AI systems are now using context engineering to plan, reason and act within real operational workflows. This entrance is perhaps the most consequential and challenging in regulated environments (think finance, life sciences, government, defense and public institutions), where trust, compliance and accountability are required.
Even in the public sector, organizations are struggling to correctly adopt agentic AI due to a lack of tooling. But standalone models and “bolt-on” AI features that are popular in the public eye completely fail to address the deep operational, governance and regulatory integrations that public sector environments demand.
While businesses for profit can assume the risk of adopting off-the-shelf solutions for basic workflows, enterprise CIOs and public leaders often will find these solutions to be unclear about exactly what intelligent products they’re introducing into sensitive ecosystems. This is where sovereign AI, a nation's or organization's ability to produce and control its own artificial intelligence, using locally owned infrastructure, data, and personnel, becomes a distinguishing factor in product.
Want more from The CPO Club?
Sign up for a free membership to complete reading this article:
Have an account? Log In
At deepset, our open-source work with Haystack reflects broader shifts in the enterprise and public sector. Organizations are moving away from pilots and pre-packaged AI toward a focus on AI orchestration for their transformative use cases, the process of coordinating multiple models, tools and systems into unified workflows that deliver consistent, accountable results. In regulated industries, these look like custom, mission-aligned systems and agents that deliver measurable outcomes while staying compliant and resilient to changes.
As Europe takes the lead in defining sovereign AI ecosystems, we have built Haystack to enable this future. Based on my experience, this is how organizations can build production-ready sovereign, agentic AI solutions for regulated environments, and why customization is the key to attaining trustworthiness and sovereignty responsibly.
The Three Pillars of Sovereign AI Systems
For product leaders, sovereign AI is neither a political stance nor merely a compliance checkbox, but a product operating model for regulated environments. As agentic AI is introduced into these ecosystems, it must be thoughtfully implemented in a way that meets the workflows where they are.
The three pillars of sovereign AI are customizability, deployment flexibility and an open-source foundation. These principles should guide any AI implementation strategy in a regulated industry, no matter where they are located geographically.
● Customizability: Tailoring AI pipelines and agents to an enterprise or public organization’s specific domain knowledge, rules and governance requirements to be trustworthy.
● Deployment flexibility: The AI should run from anywhere, such as cloud, secure on-prem systems, hybrid, virtual private cloud or even air-gapped environments.
● Open-source foundation: Ensuring transparency, auditability and freedom from vendor lock-in, running systems across different sovereignty boundaries.
Together, these three pillars enable organizations to build AI they can trust, control and truly own while fostering tighter collaboration between product teams, end users, security and legal stakeholders.
Public Sector Distinctions
In regulated environments, there are many risks that come with agentic AI adoption – security breaches, compliance failures and “black box” decisions that cannot be audited. Since organizations are on the hook for every decision their AI agents make and all the sensitive data these systems touch, public institutions must prioritize control, compliance and trust from day one upon implementation.
More so, to aid the process and change management within the organization, these AI systems must help create industry standards for similar workflows. A single, one-size-fits-all AI solution simply doesn’t work when AI and AI agents must operate under distinct legal frameworks, data classifications, and accountability models. To pull a “big box solution” off the shelf will impact these regulated workflows and require employees to adapt, with an unclear risk and liability of the data and decision-making. That is why for AI to succeed in regulated industries, customization is imperative.
Customization is often perceived as slow or expensive. With the right levels and layers of open orchestration, teams can move from prototyping to iterative testing and production deployment in a few weeks. Orchestration allows AI systems to evolve safely and incrementally, without sacrificing governance. These systems then become standards for other departments or public sector levels to adopt and customize to their needs. This often described by hands-on Enterprise Architects and Platform Teams as the ability to standardize repeatable architectures.
What Customized, AI Sovereign Systems Look Like
When looking into sovereign AI options, successful sovereign AI systems include:
- Control over models, data and deployment: Modular orchestration of AI pipelines avoid vendor lock-in, securely integrating sensitive data and deployed across self-hosted environments.
- Security-first infrastructure and auditability: Built-in logging, access controls and compliance readiness supporting public trust and transparency.
- Human-in-the-loop design: Humans remain accountable for decisions, with AI automating the repetitive work, not replacing employee judgment or legacy knowledge, but rather building that into workflows.
- Explainability and traceability: Every output must be visible and justifiable during audits and governance reviews.
This is why government agencies are moving beyond big box AI models and point solutions toward production-ready, secure infrastructure designed specifically for their real-world workflows.
Due to AI agent “hype,” organizations have focused on finding an AI model to work with and deploy. Yet to build trustworthy AI agents in regulated environments, teams must start with orchestration. Models and system designs will naturally change.
Modular orchestration platforms allow organizations to integrate models, domain logic and data as swappable components within governed AI pipelines. These pipelines are built with granular data controls, testing procedures for user feedback, audit trails, identity and access management and CI/CD integration for continuous improvement and risk management. This brings AI into standard software development and deployment practice, which makes it possible for the architecture to enable agent autonomy while enforcing strict compliance and meeting the workflows where they are.
From Policy to Practice with Haystack: Real-World Use Cases
If organizations are not sure in what capacity custom AI can be deployed, at deepset we have worked within regulated, high-stakes settings to prove that customized AI orchestration is essential as a horizontal management layer across infrastructure.
For example, at the German Federal Ministry of Research, Technology, and Space, staff previously struggled to navigate a 200-page manual governing funding programs, causing delays and inconsistencies across subsequent tasks. This was solved with a custom AI system built on Haystack, which now delivers verified answers to employee questions about the manual in under three seconds, with full source attribution. This has improved efficiency without sacrificing compliance, traceability and human-in-the-loop design.
In defense contexts, the KITCH project led by Airbus Defense and Space have used orchestrated, RAG-based agentic systems to integrate simulations and advanced tools to support real-time decisions in rapidly changing situations. They have used these programs to develop context-aware insights in mission-critical environments that support faster, more reliable decision-making for leaders.
At the broader EU level, the European Commission has built its sovereign AI platform on Haystack to enable the customization required to meet different departments’ needs, while ensuring control over infrastructure and architectures that meet standards requirements, and creating an ecosystem for innovation and repeatability.
Toward an AI-Integrated Future in the Public Sector
Regulated industries have distinct agentic AI needs. They must be able to govern, audit, customize and trust their AI, meaning generic solutions will never suffice. Sovereign systems built with production readiness in mind will define the next wave of public-sector AI adoption.
At the center of this shift is AI orchestration. Sovereign AI isn’t built by choosing the “right” model. It’s built by designing the right context through a resilient, orchestrated system. As AI evolves in these workplaces, true transformation in regulated environments will come from platforms that tie agent autonomy to control, compliance, traceability and human accountability. That is the future of responsible, sovereign AI and the opportunity ahead for public institutions willing to thoughtfully build the future of their workflows.
Stay in the Loop
Don't forget to subscribe to our newsletter for more product leadership resources and guides, plus the latest podcasts, interviews, and other insights from industry leaders and experts.
Hook 1: The State of Agentic AI Today
Hook 2: Public Sector Challenges
Hook 3: Three Pillars of Sovereign AI Systems
Hook 4: The Profile of Customized AI Systems
Hook 5: Real-World Outcomes
