Enriching knowledge and enhancing care through health data

Full workshop report


Report on the two-day workshop (March 5-6, 2018 - PARIS) organised by the i~HD
Hospital Network of Excellence and Data Quality Task Force,
in collaboration with

Gaining the Benefits of Improved Health Data Quality


i~HD Hospital Network Workshop participants join forces

Day 1

The first day began with an overview by our hosts of the health data repositories and their use at AP-HP by Gilles Chatellier. The AP-HP is a large research organisation, connecting multiple hospitals throughout Paris, and has one of the largest databases aggregating data that is accessible for research. The data are used for interventional and epidemiological research on hospital patient cohorts. We learned of the data driven innovations they utilise and some of their challenges regarding data quality. He presented several examples of studies that have highlighted different data quality issues. At times, it may be hard to tell if the data are correctly reflecting variations in the quality of care or variations amongst the patients.

Research on data is different from research on individuals. The quality is a challenge. Regularly there is a change in the method for measuring a biological quantity. If you do not know that there is a change you may have some differences in your data. We have examples of some quantative [data] becoming completely qualitative. How do we manage that ?

Gilles Chatellier, Director Clinical Research and Innovation, AT-HP, Paris

Christel Daniel then presented the technical pillars of AP-HP’s clinical data repository (CDR). They have used four research pillars: data integration & access, anonymization and pseudo-anonymization, quality assessment and business modelling with open data. She spoke of their Learning Health System mission that utilises data-driven analytics for better care. She presented details of fracture prediction through opportunistic screening for osteoporosis.
There was discussion with the audience about patient acceptance of their data being used for research. AP-HP had contacted many thousands of patients and found very few wished to opt out. The culture difference between countries was discussed: differing cultures having varying opt out rates. France is presently in a nation-wide discussion regarding access to the data from the national databases for hospitals.


Dipak Kalra next introduced the conference explaining that there is a real hunger from hospitals to do more with their data. He sketched out what assistance i~HD can offer to support hospitals. This includes privacy protection initiatives, and the aim of establishing a balance between the readiness of the public to share their data and the need for large scale data analysis to discover new disease trajectories and treatments. i~HD is helping to improve data quality, for hospitals to have better data and better EHR systems. Lastly, he stressed the importance of value based care. He linked value to outcome and to delivering outcome based care. The approach to progress has been fragmented and he emphasised i~HD’s mission to bring perspectives together, to share learning on improving outcomes. He welcomed the collaboration on this workshop with ICHOM and EFP, and thanked our sponsors ICON and EFPIA.


ICHOM presenter Rosna Mortuzza spoke about the rationale of value based healthcare. We learned of the global problem of increasing healthcare spend with diminishing returns, the variation in health outcomes and lack of outcome measurement. Generally, hospitals are unable to learn from each other because they do not collect comparable outcome data. The focus of healthcare must shift to value; outcomes measurement drives value improvements for all stakeholders. ICHOM was formed to drive the health sector towards value based healthcare by defining global outcome standards. We learned in detail of the standard sets that they have created, which (currently) cover over 50% of global disease, and how these are created. The Martini Klinik in Germany had demonstrated improvements in care as a direct result of using outcome data. Discussion included the standardisation of outcomes, ICHOM’s development of its standard sets which include international working groups and international patient representatives.

ICHOM and i~HD are working towards the same goal around providing value to healthcare in terms of patients, providers, payers and the wider system. I think we can work synergistically to achieve that.

Rosna Mortuzza, ICHOM

Pascal Coorevits presented the Data Quality Task Force and i~HD’s data quality service. He explained i~HD’s nine data quality dimensions: completeness, consistency, correctness, uniqueness, timeliness, stability, relevance, contextualisation and trustworthiness. We heard about innovative tools and applications for measuring data quality, with examples. These dimensions have been developed in cooperation with many hospitals, and the new service will now be piloted in Gent University Hospital, Madrid’s 12 de Octobre Hospital and Hospital del mar Barcelona, before rolling this service out to the rest of our hospital network. i~HD is in discussion with an ISQua certified organisation to support this scaling up.
There was lively discussing on funding streams, the potential for alignment with other organisations, the impact of data quality cleaning, the cost and incentives for hospitals investing in this service and the potential benefits to a hospital for raising its game on health data.

The i~HD Data Quality Task Force now offers a concrete service for hospitals to assess and monitor their data quality, giving guidance on how data quality can be measured and improved and to give feedback on data quality issues.

Pascal Coorevits, Professor Ghent University

We then heard 3 presentations of hospital case studies regarding defining, collecting, improving outcomes and the use of routinely collected health data. The first presenter was Christel Daniel who spoke of improving outcomes and using routinely collected health data at AP-HP’s clinical data repository (CDR). Firstly, she presented the CRD’s value proposition: its uses and benefits including the wide range of studies being undertaken. She then presented the results of several data quality assessments, highlighting different kinds of quality problem. Christel concluded that the quality of data sources ultimately determines their utility and value.

Ronald Cornet, from the Amsterdam public health research institute, presented a study at the GI department of the Amsterdam Medical Centre. This study examined whether routinely collected clinical data was of a sufficient quality to enable outcome indicators to be measured. He drew the audience attention to the privacy protection difficulties they had with undertaking this study. He outlined some of their data quality findings, stressing the challenges of the data entry burden on clinicians. He also emphasised the need for research data to be FAIR.

Morten Kildal, lead for value based healthcare, Uppsala Sweden, discussed the gains and benefits of improved health quality, both improving the quality of medical care and improving patient-flow. He raised the often-heard patient concern that clinicians need to communicate with each other in a better way. He had found that inviting multidisciplinary teams to regularly review their progress towards achieving good outcomes – spending only six minutes on this each session – led to team-driven quality improvements. We learned that they are including patient representatives within the team, to ensure that this is a patient centred process.
There followed a lively Q&A panel discussion where many queries were raised. The topics included:

  • The critical importance of data quality.
  • The concern about length of time taken to use data was raised.
  • How possible is it to adopt standardised outcomes in a collaborative way?
  • To what extent are standardised outcomes the way to work in the future?
  • What kinds of challenge are there for data quality when patient entered data or sensor data is involved?
  • What is the perspective on working on one’s own hospital data, to reap benefits and compare with others?
  • Responding to Ronald’s data access challenge, will this situation get worse with the introduction of the GDPR? We need more guidance on using the GDPR especially with real world data.
  • We need to improve health data to improve healthcare, so we need to work with patients. We also need to involve patients to make their data available for research.

Then we heard from Nicola Bedlington Secretary General of the EPF. We learned about patient reported outcomes, why are they important and why the patient experience matters. Nicola discussed the EPF’s mission and strategy and explained what patient empowerment should be. The EPF welcomed the OECD “PaRIS” initiative, an important political step. They are contributing to PaRIS, bringing a cross-disease perspective and facilitating patient participation. PaRIS has potential to drive real change in healthcare practice and mindsets, supporting health systems to become more patient-centred. Patient involvement is key. An important area will be the “generic” PROMs, as well as PREMs. Regarding the use of new technology, it must be patient driven not technology driven. This ensures it adds value. Patient engagement is vital for the entire life cycle of medicines development. Change happens bottom-up …. but needs an enabling environment, support, plus top-down investment.

The European Patients’ Forum has been very welcomed by i~HD right from the early stages. This is key as we see ourselves helping to shape the institute’s objectives; to make sure that the Institute itself is a very patient centred, patient focussed organisation.

Nicola Bedlington, Secretary General, European Patients’ Forum


After lunch, Carlos Sáes presented i~HD’s data quality dimensions in detail.
i~HD’s initial data quality service is focussing on three dimensions: completeness, consistency correctness. He presented examples of each, highlighting that one study showed 29% of errors in Spanish health data.

  • Correctness: values are true and unbiased.
  • Consistency: data satisfy general and domain constraints related to format, allowed ranges, multivariate rules, inter-table relationships etc.
  • Correctness: likelihood of recorded values with respect to their real-world state or under a specific context.

Carlos presented the ways in which data quality tools and applications can generate visualisations and reports to inform a hospital about its quality issues. He illustrated what reports could be offered by the new data quality service.

One of the key challenges in the reuse of data and monitoring of outcomes is the variability of data among the different sources or hospitals. This problem is not addressed in the data quality literature and you may find that you are relying on indicators derived from multi-source data that may be biased for even a single source providing this data.

Carlos Sáes, Senior Researcher, Universitat Politècnica de València


The afternoon finished with 3 parallel breakout groups focussing on how to optimise the collection and use of high-quality and routinely collected data.

  • Group 1 examined the assessment and improvements of clinical outcomes for direct patient benefit within the healthcare organisation facilitated by Nicola Beddington and Dipak Kalra.
  • Group 2 discussed how to maximise the quality of data used for outcomes recording and reporting facilitated by Geert Byttebier, Pascal Coorevits and Carlos Sáez.
  • Group 3 learned how to combine outcomes data across healthcare organisations to support planning, public health decision-making and health service strategy, facilitated by Rishi Hazarika from ICHOM.


After plenary feedback from the sessions Dipak discussed with the audience i~HD’s proposal for collecting, publishing and maintaining an inventory of outcomes relevant data items linked to data quality benchmarks. cross referencing the ICHOM standards where applicable. It was strongly suggested that this inventory should also include case studies of where hospitals have shown value from measuring their outcomes.

Day 2.

Dipak Kalra introduced and welcomed everyone. He spoke of the imminent GDPR with emphasis on raising everyone’s data quality game. The ongoing challenge for healthcare is the limitation of funds and investment: revenue cannot be directly increased which is acting as a barrier to improvements. The theme of the day was about making a convincing case that there is value in investing in better data. Why is it in the interests of healthcare organisations to invest in data quality and better data?

Neil O'Hare was the first presenter and gave an overview of steps of IT system deployment. He emphasised the importance of building a relationship across stakeholders and decision-makers, building trust and increasing understanding through education. He outlined the elements of a business case and raised the question of who owns the data and how can it be obtained. He noted the cost of data: it accounts for 29% of clinicians’ time. He stressed the importance of including the clinical team in discussions about IT systems.
During discussion with the audience some queries were raised including: the costs of legal procurement and the necessity of brainstorming the potential tangible benefits of an IT system. It was noted that decision makers only perceive costs and the spending of more money on IT, they often do not see it as an asset or perceive the opportunities of data benefits or how it can deliver benefits. He stated that health IT brings a slow gain: it is a 10-15-year journey. Identify the key stakeholders, the key enthusiastic people and educate them as to the benefits.

Rishi Hazarika, ICHOM took the next session. ICHOM develops, and helps hospitals to measure, standard sets of outcomes. They are currently supporting 43 hospitals, globally. They benchmark data for sharing and develop value creation strategies that will have profound effects on all healthcare stakeholders. Hospitals can deliver higher quality and lower cost care. Measurement was discussed as a process of cultural change. He discussed clinical outcomes and measurement and emphasised that the biggest factor is clinician leadership. Clinicians provide valuable insight and make the project a success. There can be concern about outcomes measurement amongst clinicians, and it is important that they are utilised for improvements and reflection rather than management enforced change. He discussed how measurements integrate within the workflow and need to become perceived as a “business as usual” activity.
Informatics can enable benefits and provides opportunity for feedback and interfaces, it encourages completeness and accuracy, bringing interoperability and export functions. He expressed the maintenance of momentum as important, which is assisted by sharing and communicating successes and benefits. He emphasised their approach of aligning with clinicians to convince them to measure outcomes, then building on that and giving clear motivation.
The second part of his talk considered the importance of measuring and reporting meaningful outcomes. The example given was the Martini Klinik, Germany. They shared outcome data, mentored each other and used an innovative adoption of consistent measurements which led to rapid prototyping and rapid adoption of outcome measurements. They implemented a reverse mentoring technique, a non- hierarchical approach. It created a virtuous learning cycle and brought benefits.
He concluded that generally, the level of spend does not correlate to better outcomes. Data use and data transparency is changing; it is happening in health care now. Leadership is important as are team effort for outcomes measurement. There is “no one size fits all” approach. Measurement does not have to be a burden to clinicians. It is important that data is accessible and actionable if it is to have an impact.
Discussion included scale of adoption, the importance of involving any hospital organisation, not just the centres of excellence and how patient representative groups are involved. ICHOM facilitate, recruit patients. He discussed that hospitals will publicise their data, using measurement outcomes, which will give patients choice.

Teus Linsen, was next to speak about incentivising healthcare organisations to improve outcomes in the Dutch healthcare system It is a regulated market, a complex patient- insurer- provider network. He asserted that transparency of outcomes is the most powerful driver in healthcare and talked through some of the initiatives they have used. There is a need to move to new payment models, to value based models. Value can be created at different levels including the medical conditions and the patient experience. There are opportunities to improve quality of care, increase income and have a no limits volume contract.
At the level of the population, value can include: the total health cost of the population, shared savings, shared losses, savings dependent on quality of care and patient satisfaction. He spoke of working on incentives and value based health contracts together, but stressed that registration (data entry) needs to be done by healthcare professionals and patients (PROM). Data entry burden and indicators not being patient orientated are challenges to value based health care.
Audience discussion was around the challenges of dealing with patients with multi morbidity which, due to its complexity, is still work in process, and the incorporation of complications in outcomes.

The next speaker was Thomas Allvin who spoke on outcomes-based healthcare and the contribution of pharmaceutical companies. We learned that there is a projected increase in public expenditure in healthcare due to the demographic shift towards older persons, which raises the challenge of healthcare sustainability. There are variations in morbidity rates between and within countries. There is currently a huge practice variation between OECD counties. It is possible to identify best clinical practice, for example, around prescriptions.
It is estimated that there is a 20-40% waste in healthcare systems. How can money in the system be better used? There are four key sources of waste: over treatment, failures of care delivery & coordination, pricing and payment failures and lastly administration complexity. He discussed five elements that have been identified, in models, that have proved successful at increasing HC outcomes. When paying for outcomes there needs to be an assurance that we have value for money and incentives. Performance linked reimbursements are in place but are exceptional. We saw, with examples, that Italy has the most scaled up value-based system. The main barriers to implementation and the lessons learned were discussed. Medications costs are a small part of a healthcare service budget. In conclusion, he announced that EFPIA, ICHOM and i~HD are in partnership to promote outcomes based healthcare and explore data requirements through evidence pilots.

i~HD is in a good spot, with its network of hospitals, to understand the challenges that they are facing, with organisation, IT infrastructure etc. It is extremely important to have an independent partner that can work with hospitals and others to find the best practices to implement these models.

Thomas Allvin, EFPIA

Gurparkash Singh presented the opportunities and challenges of the reuse of EHRs from a clinical trial point of view. Clinical trials are costly, lengthy and complex. EHRs are an enabler for clinical research and there is lots of under-used data in EHRs that can be better utilised. However, there is a need for collaborations with different stakeholders, in a non- competitive space. EHRs can bring benefit to all stakeholders, but there are issues with data quality. He addressed both data quality and data gaps: not all recorded values are realistic, there is missing data, incorrect values and incorrect measurement units. He emphasised that the completeness of lab values should be obligatory. There can be missing values for common variables. We heard that the further the data is away from its source the more difficult it is to correct. So, the data needs corrected right away, at the point of its creation. Each department in a hospital has their own way of inputting information: we need to standardise this, as there is too much variability. This will enable interoperability & harmonisation, to make sure the systems can talk to each other and combine data. There is a need for data tools and data specialists.

In discussion, it was agreed that patients with chronic diseases are very interested in EHRs for their treatment, management and empowerment. The EHR does present some fundamental challenges, currently we have a plethora of data and a thirst for knowledge.

Diane Whitehouse led the panel session. We heard three Presentations from real situations in hospitals and how they are achieving value through data.


Ricardo Garcia de Leon Chocano presented an information-based best practice for mother and child care outcomes. He spoke of the milestones for improving outcomes and the importance of improved data quality.
Gregory Katz is working with ICHOM and he emphasised that transparency brought about improvements. Transparency drives change, the sparks of change reflect this. He spoke of the challenges that had been encountered, primarily that transparency needs a standardised approach and the need for a standardised language. PROMS and COMS data integrated. It is a true organisational challenge.
Agustin Gomez de la Camara then presented how benefits have been realised from targeted interventions on outcomes, emphasising the role of outcomes standards and data. He raised the importance of data quality and the desire to turn the data into benefits which will need more data to be available.

In the afternoon Nathan Lea spoke about the imminent enforcement of the General Data Protection Regulation and the launch of i~HD’s GDPR task force. It will be focussing on the opportunity around the GDPR. It is a ground-breaking regulation and there is a need to better understand it. He views the GDPR positively as it will realise opportunities, establish a clear communications and data use network and it will drive better data value within the sector. He discussed what he thinks is needed, where our current understanding is and talked about the GDPR being a living document requiring a living task force, with regular discussion. The GDPR will furnish the hunger that patients have to be involved in their health data, offering a chance for academics and others to educate. i~HD will provide education on the GDPR in a variety of formats, in accordance with the needs of its members and community.
Our aim is to publish a White paper, so people can refer to it and see the opportunity space.
In conclusion, the GDPR is a living document, implementation of it will be required. Privacy is a requirement for all with the consideration that technologies are advancing. 25th May is the beginning of the GDPR not the deadline.

i~HD is uniquely placed to help the innovation community and our members to fully understand the requirements of the GDPR, in terms of improving data protection, transparency around use and understanding why it is important to share healthcare data.

Nathan Lea, Senior Research Associate, UCL

We then had three breakout sessions that covered three targets for business case development.

  • Group 1 discussed EHR system procurement, facilitated by Thomas Allvin and Dipak Kalra.
  • Group 2 looked at data quality improvement, facilitated by Veli Stroetmann.
  • Group 3 considered the promoting the adoption of value based care, facilitated by Diane Whitehouse.

A plenary feedback session followed where the thoughts and discussions from each of the groups were presented.

ICON’s mission is to help improve patients’ lives. Collaborating with i~HD, health care providers, hospitals and pharmaceutical companies is important to us because we know that this is the data that would help us save patients’ lives.

Almenia Garvey, ICON


Dipak Kalra concluded the workshop by thanking the hosts, the speakers and the attendees for their participation. He briefly explained about i~HD supporting #datasaveslives, including the campaign’s mission and why our involvement is important. He said: “Let's help to explain to the world how health data can help.”
Finally, he outlined the proposed themes for the upcoming 2018 annual i~HD conference.


Published on: 13 April 2018