Recent statements by leaders of frontier AI companies have brought renewed urgency to the debate about whether the capabilities and autonomy of general-purpose AI are developing faster than our ability to fully understand, govern and control their risks. These concerns go beyond the familiar challenges of bias, transparency and responsible use: they raise questions about increasingly autonomous AI systems, the adequacy of current testing and evaluation, and whether sufficient safeguards can keep pace with rapidly advancing capabilities.
AI guardrails – an open letter from i~HD
For i~HD, these developments reinforce the principles that have guided our work on trustworthy AI in health. AI has enormous potential to improve healthcare, but it must remain a tool in the service of people—not a substitute for human responsibility and judgement. Human agency and meaningful oversight, robust testing and validation, transparency, accountability and independent scrutiny therefore need to be designed into AI systems and their governance from the outset. As AI systems become more capable and autonomous, however, these safeguards themselves must evolve: keeping the human “in the driving seat” requires not only oversight at the point of use, but effective governance throughout the development, deployment and monitoring of AI. Guardrails should therefore extend beyond regulation and technical controls to encompass human judgment, critical thinking leadership and accountability, ensuring AI remains powerful, but firmly in the backseat, and we should explore innovative models of regulation, adapted to the challenges of regulating an evolving technology, rather than instinctively reaching for the idea of a ‘kill switch’ that is highly unlikely to exist.
The importance of AI for healthcare
i~HD has long championed the potential value of artificial intelligence embedded within digital health technologies for use by clinicians and patients, the uses of AI to analyse large volumes of complex health data to better stratify patients for risk, diagnostic or personalised treatment, as well as in life sciences research. AI can therefore be an accelerator of digital health transformation to improve healthcare efficiency, improve outcomes for patients, reduce healthcare and drug development costs, accelerate vaccine development, improve resilience against future possible pandemics or other threats, and to strengthen the empowerment of patients in their own self-care and prevention.
Europe’s regulatory stance on AI
Europe has taken world leadership in developing a comprehensive framework for the development of trustworthy AI in Europe, progressing from the voluntary, ethics-based approach of ALTAI in 2020 to the legally binding, risk-based framework of the EU AI Act setting a foundational standard through legislation to manage the associated risks. The EU AI Act applies very stringent obligations on AI developers and on AI users of high-risk AI systems such as those used in patient care. It also recognises that the most advanced general-purpose AI models may create systemic risks extending well beyond an individual application, which adopting organisations will need to manage. Providers of such models in must evaluate and test them, assess and mitigate systemic risks, report serious incidents and maintain appropriate cybersecurity safeguards.
Through the European AI Office, the Commission has powers to request information, conduct its own evaluations of general-purpose AI models -including through independent experts -and require access to models where necessary to investigate compliance or systemic risk. It can require corrective and risk-mitigation measures and, impose measures which seek to restrict availability of a model on the European market. These provisions apply to providers placing their models on the EU market irrespective of whether they are established in Europe or elsewhere. The EU is therefore combining trustworthy-AI principles, binding obligations, independent regulatory scrutiny and enforcement in a framework capable of responding as AI capabilities and risks evolve.
How i~HD is supporting trustworthy AI innovation in health
i~HD has been working on the regulation, safety and governance of AI for nearly a decade now. At the core of our approach is the principle that innovation, trust and governance must develop together: AI can deliver value in healthcare only where patients, professionals and healthcare organisations can have confidence that it is safe, trustworthy and used responsibly. This has been at the core of our mission on AI: to enhance safe, trustworthy reuse of health data to improve outcomes within an understandable and appropriate governance framework. We advocate robust, proportionate governance that enables rather than constrains responsible innovation. Well-designed governance can help create the conditions for adoption and scale by providing clarity, accountability and confidence for developers, healthcare organisations, professionals and patients. We also believe that safe and successful AI use within healthcare will propagate confidence, discernment as well as experience amongst healthcare professionals, patients and healthcare decision makers.
This is an important area of innovation where much learning is still needed. Our work is directed at both the level of individual AI solutions and at the wider organisational and health-system level. i~HD is actively working in research and innovation projects that are pioneering areas of AI innovation for different medical challenges. At governance level, we work with healthcare organisations, policymakers, professionals, patients and international bodies on the conditions needed for responsible AI adoption: clear accountability and human oversight; independent evaluation and assurance; appropriate procurement and deployment practices; data quality, representativeness and bias; transparency and explainability; workforce AI literacy; and continuous monitoring as technologies and risks evolve. Through our Horizon Europe, IHI and PERMED projects: these principles are being applied and tested in real innovation settings, while the lessons learned feed back into our policy and governance work.
By analysing the ALTAI requirements and AI Act provisions, we have developed a toolkit of criteria that software innovations need to reach in order to meet regulatory requirements. These criteria and this proposed compliance approach draw from our work on GDPR. Taking the Data Protection by Design and Default principle, we introduce compliance adherence as early as possible, regardless of whether the regulatory requirements are enforceable yet or not.
Across our European research portfolio, i~HD is applying and testing trustworthy-AI principles in very different healthcare settings. These projects provide practical environments in which questions of safety, data quality, fairness, transparency, accountability, human oversight as well as responsible and accountable deployment can be addressed alongside technological innovation.
- SMASH-HCM is identifying which patients with concerning cardiovascular symptoms who are early in the development path of cardiomyopathy and should have intensive interventions and those who are simply healthy people who should not be treated at all.
- GENOMED4ALL has used AI with genomic and clinical data to better understand sub-types of haematological cancer which might respond better to different treatments.
- IDERHA is developing novel artificial intelligence algorithms to risk stratify and treatment stratified patients with lung cancer, to help improve outcomes.
- PARADISE has researched into an algorithm that could provide early prediction of a possible relapse in vasculitis, allowing shorter and lower dose courses of corrective treatment than is currently possible.
- SYNTHEMA has developed synthetic data generation techniques for supporting rare disease research in cases where source data availability is limited and privacy concerns are paramount.
- AIDAVA has developed AI tools that can help improve the quality of routinely collected EHR data by identifying inconsistent entries and proposing values for missing data.
Through these projects, and more broadly in the community, we are developing and supporting good practice in ethical AI development and preparing their solutions for AI Act compliance.
i~HD engagement with key stakeholders
Regulation and technical safeguards alone will not be enough; and because it is unlikely that a ‘kill switch’ will ever exist, it is crucial that the human judgement, critical thinking, organisational accountability and leadership skills needed to develop understanding both the opportunities and the limitations of increasingly powerful AI systems are developed.
These questions of human agency and responsible leadership are becoming just as important as the technologies themselves. Initiatives such as CLAIR - conference of leadership for AI in research - provide an important forum for bringing together leaders, researchers and practitioners to examine how these capabilities must evolve alongside AI.
i~HD is actively working with the OECD, the European Commission and the World Economic Forum on the good practices and guardrails that governments need to put in place in order for AI in health to flourish safely. These transnational bodies offer trusted ways of bringing countries together to collaborate on aligned policies and measures that they should take to balance the benefits that AI can bring with an appropriate level of governance and transparency to safeguard individuals and society. We value participation in their expert and advisory groups, and contributions to their policy reports.
i~HD recognises the challenges that AI developers have in gaining access to high-quality and representative data on the target populations for their solutions. We provide guidance on data quality and data representativeness assessments, bias mitigation strategies, ethical AI development and validation practices and on AI transparency to users. We are working with stakeholders and transnational bodies to formulate recommendations for how healthcare systems could collaborate to make pre-competitive health data available to the AI development community.
We are helping healthcare systems to make more effective, wiser and safer uses of AI, for example by improving the AI literacy of the health workforce, consulting across stakeholders on how healthcare systems could become more influential in shaping the AI developer marketplace, to target its most highly prioritised healthcare transformation needs, good AI procurement practice and how healthcare professionals and patients should be equipped with the right knowledge to become effective and safe AI users.