eMeasure Feasibility Testing 

Access the Full Report 

The Opportunity

The National Quality Forum (NQF) relies on four criteria for evaluating the suitability of quality measures for endorsement as voluntary consensus standards: Importance to Measure and Report, Scientific Acceptability of Measure Properties, Usability, and Feasibility. The second criterion, Scientific Acceptability of Measure Properties, requires testing to demonstrate empirically a measure’s reliability and validity, which are critical elements for use of measures for accountability. Evidence that a clearly specified measure produces credible results on performance comes from the basic measurement principles of reliability and validity. In 2010, NQF convened a task force to provide further guidance on 1) the scope of testing, 2) what tests of reliability and validity could be performed, and 3) how to weigh the results of this testing. The resulting NQF Measure Testing Task Force report has added clarity to the measure development enterprise.

As quality measurement shifts to measures derived from electronic health records (EHRs), there is less clarity about the testing needed to assure that eMeasures can be used for a range of accountability applications, which require both precision, and results that are reliable and valid. While the concepts of reliability and validity apply equally to measures derived from EHRs, the electronic health record presents additional challenges related to measure testing: widespread EHR data are not yet available for measure development and testing; there is a lack of comparability across vendor products; and data elements needed for advanced measures currently may not be feasible to capture in EHRs. Realizing the promise of EHRs as a tool for quality reporting will rest on the ability of providers, payers, vendors and other users of the information to know that e-measures provide valid and reliable data. During the public and member comment for NQF’s eMeasure Review and Testing Proposal in early 2012, several organizations expressed support that eMeasure testing should incorporate the feasibility of data capture for the data elements utilized in addition to reliability and validity testing. This requirement is significantly hampered by the lack of clarity and definition in the field as to what constitutes feasibility testing for EHRs.

About the Project

This project will assess the current state of feasibility testing for new and retooled eMeasures and identify an important set of principles and criteria for adequate feasibility testing.

Process

This initiative will gather current approaches used to assess measure feasibility by conducting an environmental scan of approaches to feasibility testing from measure developers, government contractors and EHR vendors. NQF will convene a panel of experts to review the results of the environment scan and propose a set of feasibility recommendations. Additionally, the expert panel will identify a starter set of criteria for eMeasure feasibility testing that will address the following considerations:

• Timing of feasibility testing in the course of measure development (e.g., iterative testing with development, feasibility testing of fully specified measures)
• Potential differences in feasibility testing requirements for de novo measures and retooled measures;
• Number and diversity of EHRs used for testing and relation to comparability across EHRs in terms of data feasibility, validity, and reliability; and
• Interrelationship of feasibility and validity testing of new measures.

This work will culminate in an eMeasure feasibility testing guidance report and should provide important guidance that can shape future measure development contracts as well as certification requirements. The report will be available for Public and Member comment and for the Consensus Standards Approval Committee (CSAC) and Board of Directors’ input and approval.

NQF Related Work

Measure Testing Task Force 

Funding

This project is funded under a contract provided by the Department of Health and Human Services.

For information about the availability of auxiliary aids and services for NQF’s federally funded projects, please visit: http://www.medicare.gov/about-us/nondiscrimination/nondiscrimination-notice.html.

Contact Information

For further information, contact Reva Winkler or Kathryn Streeter at 202-783-1300 or via email at emeasurefeasibility@qualityforum.org.