[This Transcript is Unedited]

Department of Health and Human Services

National Center for Health Statistics

Subcommittee on Quality

June 15, 2011

Double Tree Hotel
300 Army Navy Drive
Arlington, VA 22202

Proceedings by:
CASET Associates, Ltd.
Fairfax, Virginia 22030
(703) 266-8402


Agenda Item: Welcome

DR. MIDDLETON: We will now convene the Quality Subcommittee of the NCVHS. This is Blackford Middleton. And I think we have done introductions all the way around of everybody already once unless there is someone who —

PARTICIPANT: We do want to do it so we know who is attending.

DR. MIDDLETON: We will go around the table and just announce yourself and any conflicts if they exist. Blackford Middleton. No conflicts.

MR. QUINN: Matt Quinn, NIST, staff to Quality Subcommittee.

DR. FITZMAURICE: Michael Fitzmaurice, Agency for Healthcare Research and Quality, staff to the subcommittee.

DR. NICHOLS: Len Nichols, George Mason University, new to the committee and no conflicts.

DR. KAPLAN: Bob Kaplan from the National Institutes of Health. I hope I don’t have any conflicts.

DR. KLOSS: Linda Kloss, new to the committee and no conflicts.

DR. COHEN: Bruce Cohen, new to the committee, and no conflicts.

DR. GREEN: Larry Green, old to the committee, no conflicts.

DR. WARREN: Judy Warren, older than he is, no conflicts.

DR. CARR: Justine Carr, chair of the committee, member of the subcommittee, no conflicts.

DR. JACKSON: Debbie Jackson, National Center for Health Statistics, committee staff.

DR. FRANCIS: Leslie Francis, member of the full committee and sitting in here because of the PCAST letter.

DR. MIDDLETON: And we want to go through the back of the room please.

(Intros around the room)

DR. MIDDLETON: Anybody who hasn’t had a chance to say hello? And is there anybody on the phone? Is the phone line open? Well, thanks to all for coming and particularly to the new members of the Quality Committee. I am really very excited to have such talented folks coming to help us with this agenda and look forward to working with all of you on all the ideas and things that will be coming. It is very exciting.

We have a revised agenda which I think has been sent around or handed out. We wanted to go over briefly any remaining quality-related issues with the PCAST letter and then have some very open discussion and brainstorming around how we move forward with other sources of data relevant to quality and health, consumer source data capture, for example, and other sources of data that might be relevant to quality assessment. We can have discussion about that. Review any other quality-related issues surrounding the Community Health Data Initiative feedback and report we have just heard and then have some again open discussion.

The question I have in my mind which I think will guide us for the next hour and we will aim to finish at 5:30 on the dot is really what is next for the quality agenda. I will ask Matt Quinn to summarize some of the recent work products to start the conversation, but I hope by the end of the meeting my goal would be that we have some ideas about workshops or efforts we might want to undertake, what hearings you might want to hold, and how do we wish to relate to the activities of the other committees in the different topics or agenda items you come up with.

DR. CARR: Just one thing. I see PCAST is on here and I would ask that we defer that until the end or even until we get to the full committee. I don’t think there was anything immediate for the quality subcommittee on that letter unless you had something because it will allow more time for the discussion that you recommend. You mentioned knowledge management today. I really think that is very much in the future and coming soon to a health care facility near you. I think that really is an important concept that we want to bring forward as a learning health system.

DR. MIDDLETON: Thank you, Justine. And let me make an announcement also. Of course Paul Tang would be here except that he is somewhere in Tuscany. Paul is the co-chair with me of the quality committee.

DR. FITZMAURICE: Chuck Friedman has just announced that he has accepted an offer of a professorship at the University of Michigan. He will be in the School of Information and the School of Public Health. He will direct the university’s new program in Health Informatics.

DR. MIDDLETON: To Justine’s recommendation let me just open the floor — if there is any unmentioned or undiscussed quality issue pertaining to PCAST. Otherwise the motion on the floor basically is to dispense with that conversation.

DR. GREENBERG: I think the reason we weren’t sure how this morning was going to go and we knew that Matt who is a lead staff had taken the lead on that letter, but I think we have gone another direction here.

DR. MIDDLETON: Then if nothing further we will dispense with that agenda item. I will turn to Matt now and ask just to summarize for new members the recent history of the quality committee and some of our more important products.

MR. QUINN: In the past year or so we have had a couple of really important hearings for the quality subcommittee and both of them were ironically around quality measurement. And both of these letters are available on the NCVHS website or I can send you the link.

The first one and we published a letter December 1, 2009. It was titled “Meaningful Measurement of Quality Healthcare Using Electronic Health Records”. And really the focus of this was coming out of meaningful use. We wanted to talk around meaningful measurement and what does that mean.

We brought together a wide array of people to talk about quality measurement initiatives. At the end of the day we heard about a lot of different things and it was sort of like measures, measures everywhere, but nothing tying it all together, which we knew but we were hoping to hear a little bit more coordination.

What came out of this and this was actually reflected in the PCAST letter, we recommend looking at this, was three pretty specific NCVHS recommendations. One was HHS should develop a national quality and performance measure strategy and designate or establish an oversight structure to coordinate and align existing initiatives in the national strategy. We have recently published a nationally quality strategy, not necessarily with the oversight, et cetera. This recommendation — we referenced the IOM 2006 performance measurement report.

The second observation is that — it was based on observing that lack of standard definitions for measures and their underlying data elements and EHRs using an array of measures. It recommended establishing a quality and performance reporting specification library and perhaps using NQF’s AHRQ-supported quality dataset as the initial basis for this library of data elements and making it public and well specified. This could serve as a foundation for quality measurement that was going to be reported in an aggregate way.

And then the third observation was that if you want EHRs to do this, it should be a certification requirement or otherwise we will have the array of disparate data models that exist today. This isn’t saying replace existing data models with the standard one. It is just saying that there should be a standard chunk to support aggregate data and quality reporting in this way. That letter was December of 2009.

The next hearing that we held was post-health reform and we published it actually February 17 of this year. And the title of the hearing was “Aligning Quality Measurement with the Needs of Health Reform”. This was building on that meaningful measurement hearing. Really the aha moment here was that again measures, measures everywhere, but none that anybody actually uses themselves. What we are reporting, for example, for insurance or to CMS or to this or that are different than measures that are used internally as well as for maintenance and certification and some other aspects. Acutely there was a lack of measures that were actually used by consumers so meaningful to consumers.

The recommendations from this hearing were prioritize and fund development of measures that are meaningful to consumers. It makes a lot of sense, but doesn’t exist.

The second is funded research development of improved assessments of the value of health care based on measures and information about cost and quality that are relevant to all health care stakeholders. What this means is that try to have reporting of quality measures that are used internally God forbid for quality measurement or quality improvement initiatives.

And then third, focus on accountability and care coordination, fund research and development to provide information and measures that enhance the ability to assess accountability and care coordination health care whether this is ACO or patient-centered medical home or something else. There was noted a real lack of good measures in this area and the desire to look at different techniques.

And finally, the final recommendation was to convene quality measurement stakeholders to formulate a strategy to coordinate development of quality measures using standard meta-data. This actually builds. We reference that prior hearing.

We have not held a hearing since then. There are various directions that we could go. One is building on these recommendations, looking to the consumer side, other areas. I just wanted to provide a little bit of context as to what we had done and move forward. A little more context is the hearing that we held and this was actually a joint quality and populations hearing was on the patient-centered medical home and this was I believe in March 2008 that we held that. This was really on quality measurement for patient-centered medical home and it identified some of the gaps there.

With that I am happy to answer any questions about what we have done and then facilitate the conversation about what we should do next.

DR. MIDDLETON: Terrific. Thank you, Matt.

DR. COHEN: Matt, can you give me some examples of quality measures that would be good for consumers? How much does it cost?

MR. QUINN: We had some great discussion. The woman from Oregon Health and Science University on measures of patient activation and also the fellow from New Hampshire —

DR. MIDDLETON: I think the personalization was the aha thinking about turning, if you will, quality measurement on its head from population-oriented statistics to really personally oriented measures for the individual. We thought that was kind of a C change and a little bit of a paradigm shift.

Other examples might be — not is what is the population outcome expectation or risk assessment for the total hip I am about to have, but what is the risk assessment for me given my profile, attributes, et cetera. These were kind of big ideas which I think we need to come back to and I have some things that will I think will help us flush them out and maybe be topics that we can pursue.

MR. QUINN: The aha of this — the aha of the previous one was that everybody is doing lots of things. There are tons of activity around quality measurement, but none of it coordinated or little of it. And the aha for this is there are lots of measures, but nobody really wants to use any of them. Our statement here was pretty blunt. Consumers currently do not use available quality reports. Information on patient experiences or outcomes for the average patient is not useful or actionable by consumers. Consumers want to know about the impact of health care interventions for themselves as unique individuals. That is really saying — you can change consumer to physician probably too.

DR. MIDDLETON: Other questions for Matt on some of the recent reports or letter?

DR. NICHOLS: I would just like to ask is there shall we say a conversation going on between all of the learning you all have been doing in these workshops particularly around consumers; and the people who are writing the regs for the exchange which is going to be tasked with ranking health plans by price and quality where presumably although I am not positive the quality will reach all the way down to the delivery system underneath the plan as opposed to just consumer service.

MR. QUINN: That would be an excellent hearing for us.

DR. NICHOLS: Well, that is where I am headed, but is that conversation going on now? Are you allowed to tell me? Are you actually paid by the Federal Government?

MR. QUINN: That is not a conversation that we have had. Our letter to the secretary tried to highlight these issues and we have not been re-engaged by them to address it.

DR. NICHOLS: — that reg has been almost out now for eight months so it could appear tomorrow. It in clearance I presume. That just seems like a natural place to really bring this — particularly when you just told me they don’t use any of them and we are about to require new creatures that don’t exist to rank plans based on them. Good Lord, have mercy. We could do better.

PARTICIPANT: Or require as part of meaningful use.

DR. NICHOLS: Anyway that seems to be my candidate for the first hearing.

DR. FITZMAURICE: I guess it is not to say that employers and purchasers don’t use them. It is to say that people who receive the benefits of health care don’t use them.

DR. NICHOLS: Yes, but the rankings are precisely designed to go to help consumers make choices about plans.

DR. FITZMAURICE: I agree. That is a very fertile area and one that we should put it near the top of our list.

DR. GREENBERG: And didn’t you also hear that providers or physicians or whatever didn’t use them either?

MR. QUINN: The measures that they report externally are different than the measures that they use internally. That dissidence is just fascinating.

DR. KLOSS: I have heard — recently I have actually been doing more work in the clinical analytics area and Dr. Mullen from Fairview in his speaking, he makes a distinction between measures for accountability and measures for improvement. I think that for me the light bulbs went on because the standard measures have to be so clean and crisp to be used for improvement. We delete all patients who with diabetes who are over 70 years so that the severity adjustment issues aren’t compounding or confounding the analysis of data. But the measures that are needed for population improvement that will be needed for accountable care and other internal really need the complexity of the patient populations. I think that so many people in working quality in healthcare organizations — all their efforts around measures and perhaps maybe the more advanced organizations understand that that serves one purpose, but it isn’t the improvement purpose.

PARTICIPANT: That is a great thought. Thank you.

MR. QUINN: Where those two things come together almost is in the — we heard about maintenance of certification from the boards and where physicians are required or one of the ways to maintenance of certification is to take on quality improvement projects and how those are designed, measures that are used in there and the level of improvement required as part of it or demonstrated as part of that touches on both of those.

DR. KAPLAN: Is this because of the reporting of an aggregate for quality purposes?

DR. KLOSS: Yes because it is being used for public reporting. The measures have to be so clean and you have to take out the complexity. They are not very useful for internal quality improvement.

DR. MIDDLETON: I would like to go around the room and brainstorm for about 30 minutes on what are the key topics like these, the quality issues bedeviling the country inhibiting our ability to measure and assess and improve quality and then really in a free willing way no idea is a bad idea, try to refine then after we have that set into a set of things we might want to look at in the next 18 months. Does that sound reasonable?

MR. QUINN: Does it make sense to describe what an NCVHS hearing —

DR. MIDDLETON: For newcomers? Sure. Please.

MR. QUINN: For both of these hearings we had approximately 20 to 25 participants, most of them in person doing somewhere between 15 and 30 or 45-minute presentation. We seek to have people who are going to really provide diverse perspectives on it. We start with a room like this and usually over the course of a day or day and a half tries to bring folks together. And then based on what we hear at the end of the day based on what we hear and what we don’t hear we try to come up with a letter to the secretary providing observations and recommendations about what should come from it.

Sometimes the goals going in aren’t necessarily what we produce going out based on what we hear and what we don’t hear. Usually we try to have rich dialogue and then some time for the committee, not just people from the quality subcommittee or whoever is hosting it, but some dialogue around what we really want to tell the secretary about what we have heard.

DR. CARR: One time we got testimony about the measures — I mean for measurement and our letter to the secretary was that the absence of coordination made it impossible to appreciate the individual efforts because each individual had developed measures which were great, but nobody else was using them and they were idiosyncratic although people came saying I have this and I have that. The message to us was this is a problem because there is no coordination.

DR. MIDDLETON: Let me build on that and sort of start the brainstorming exercise. One of the things that I had thought of and we have this item on the agenda is analysis by this committee and the hearing potentially on how do we address the issue of consumer source data capture both at the clinical level for health care delivery purposes as well as what survey data or consumer source or personal data source date capture can apply to surveys.

The idea would be looking at how consumers are supplying data from home environment, from work environments to what kind of tools and resources, what is happening with these data as they are rolled up into community health data initiatives or other types of population reports.

There is increasingly of course that the acquisition of these remote data from the home environment, the workplace, even the car, et cetera can be very instrumental in influencing clinical decisions and patient care utilization and outcomes. This whole area is what I have been calling consumer source data, i.e. those data from the consumer him or herself. Let me open up that one for discussion.

DR. COHEN: You mean self-reported data.


PARTICIPANT: It could be remote monitoring.

DR. MIDDLETON: That too.

PARTICIPANT: Halter monitors, blood pressures.

DR. COHEN: I really like that idea. When I think about the emerging uses of EHRs to link and to populate existing data streams, we have surveillance systems in place whether they are self report or from other areas. What is the best source of the data? For instance, where should we get race data? Should we get it from the birth certificate or should we get it from the medical record or hospital intake form? Where should we get information on cancer incidents? Should it be aggregated from path reports or from cancer registries? I am very interested in evaluating the quality of data by source. If we are going to be promoting the use of indicators, we need to understand the limits of the quality of the data from the different sources.

DR. CARR: Is this about meta-data tagging — what you are saying is right that the source of the data whether your race and ethnicity came from the registration clerk or the physician or the patient himself or herself is different. I think we do have to be cognizant of that and maybe it is a part of that grand scheme is just really acknowledging that the data source impacts the reliability of the data and whether cataloguing that because whether it is meta-data or some other data cataloguing those sources is important.

DR. WARREN: One of the things as I was listening to you reminded me of Jim Scanlon’s report this morning that we are beginning to look for standards for reporting those things. That would be something that I think would be appropriate to explore is where do we get the standards for developing the quality measures. We all say it comes out of evidence, et cetera, but if I have organization one deriving a measure on hypertension or something and I have another one with one that has totally different data elements and we still can’t compare anything. How do we determine the source of goodness?

And it also kind of goes along with an assumption that we have that if it is self-reported, it has to be erroneous data. And I am not always sure about that. I think maybe the patient may be the most accurate source of the data. And the other places who make assumptions about the patient may be the ones who are reporting erroneously.

DR. MIDDLETON: We find it partners, for example, assignment of a patient who his or her primary care doctor is exceeding difficult and it is much easier just to ask the patient. Who is your doctor?

MR. QUINN: The best source of information for kids is their parents oddly enough.

DR. GREENBERG: I know that we have been asked — I particularly remember a conversation we had with Ed Sondik saying is this quality subcommittee about quality of care or quality of data. I think what we are saying is it really is about both and in that way can really — I see the linkages with what we were talking about the last hour and those questions about what are the best sources of the data. What can you realistically think you could use patient reported data? What about electronic record data? What about survey data? All of those – the population health record, the minimum data, all of that.

I think we maybe — I would love to actually see this subcommittee or this focus area because I know Larry is pushing us not to siloed here with subcommittee, but I think at least we agreed a few meetings ago for a while at least we would say with these subcommittees or these focal areas really embrace that part of the quality equation and the quality of the data. A lot of people are making assumptions. You can use certain types of data for purposes that it maybe you can’t or that you can’t use data for certain purposes which you could. There is a lot of lack of clarity on that.

MR. QUINN: I was going to say one hearing that was more of a mini hearing that I forgot to mention was remember we had the innovative data sources discussion and so we brought in Matthew Holt, John Wonka(?) and someone else who talked about PatientsLikeMe and Health 2.0 type world.

MS. GREENBERG: That was at a full committee meeting.

MR. QUINN: Yes. It wasn’t 25 people talking. It was more three or four. But it really sort of got the juices flowing. The genesis of that was some discussion in our patient-centered medical home larger hearing that said we need to look into other sources of data. I remember some of the conversation around it was what can we use this for. Is it as rigorous as a survey? I think that it might — what I have heard is that we want to — as this has become more mature in probably the last couple of years how do we incorporate this into both perhaps accountability as well as improvement efforts in the context of health reform, ACOs, patient-centered medical home, et cetera.

DR. KAPLAN: I just want to say I just think this is really important. In addition to that in doing it — in addition to just doing it — the harmonization is really crucial that everybody goes about this in their own way. What we are discovering is that is it most important for these psychosocial measures and behavioral things; something as simple as do you smoke cigarettes is asked in all kinds of different ways and questions about alcohol use and depression and so forth typically are not included anywhere.

PARTICIPANT: Those data won’t roll up unless we get at this level.

DR. KLOSS: I like your idea of the consumer sourced data. I almost think that there is a need for a focus and it may be a separate initiative somewhere on that time table for looking at current state of data quality measurement and what the models and techniques for doing that because it is not being done except ad hoc and for specific purposes. I think there is an assumption that data quality has improved in the electronic health record and not necessarily so.

When you think about data quality, it really is one of the hallmarks of information governance. We have talked about governance and I think quality is a really important part that gets less attention than some of the other policy areas.

DR. HORNBROOK: One of the things we sometimes forget to do is ask about who is the specialists in any given variable. We used to think that there were only two answers to the word sex or gender. Now of course there are places in the country where geopolitically or politically there are only two answers that are allowed on the questionnaire. There are other places of course where they want four answers or six answers. The same thing with race and ethnicity. These questions have experts out there who have done research on how communities view themselves and how they want to report themselves and how you can flex your instruments to not take off communities who you hope are going to give you the best answers they are willing to give you.

DR. NICHOLS: This comes under the heading of thinking about the future, but one of the virtues of having recently come back to academia is all the dissertation talks I have seen. One of them was this really fascinating study of social media and how it impacted cap(?)32:13 scores because the people who used the Johns Hopkins chat room for cancer survivors have a more favorable impression of Johns Hopkins — it really does have a — my point is simply think about the techniques and Lord knows they are all brand new of capturing what is really being discussed in that social media. You could even pick up things about — look if it is driving things like patient compliance then it is relevant to our enterprise because we are going to be measuring patient compliance. It is way developmental, but it is something we should — that technique for, if you will, sampling the cloud is really going to be useful.

DR. WARREN: To follow up on that I just heard my first dissertation proposal for data collection was done by Twitter and the analysis of the responses in health care. There is new stuff that is out there. Twitter is being collected at the Library of Congress, all the messages.

PARTICIPANT: Is that right? I will be more careful now.

DR. MIDDLETON: Judy, this line of thinking you are going down is — I have heard Jeremy Wyatt say that he doesn’t feel he is going to need to run an RCT ever again because he can access in three days or three weeks enough data on any particular question from these social or Internet-based surveys and sort of queries and what not that it is sufficient. It is a totally different paradigm of investigation.

DR. WARREN: I think too the kids that are coming up — of course anymore the kids that are coming up are younger than 30. They have a different view of interacting with technology. Everything is with Facebook. Everything is through social networking. If you want to dole out surveys, it is not the old stuff that we have been used to. It is the stuff that is dynamic and engaging, et cetera. And most of it is collected electronically. We really are coming into a new world of how we can collect health data both to make policy decisions with as well as research with.

DR. CARR: I think that it is important to realize that where there used to be one or two answers just as we have heard many times today we are in a dynamic fluid state. And getting back to this concept of a learning health system in some ways we imagine all of the checklists or forms, on EHR the box that says race and ethnicity will march up and it may or may not be right. But in aggregate it tells a story. But the aggregate comes from many different sources.

I think that a theme is aggregate data that does not come out of the statistics world but yet has already demonstrated. I have to just mention that learning that Marc Overhage shared with us last year with his very rich Indianapolis information exchange and his finding was that there was a growing incidence of no cough. It just shows we need to understand the incentives for recording the data.

DR. MIDDLETON: What I have heard so far and I think there has been a lot of resonance around the idea of potentially constructing some hearings around this topical area, self-reported, consumer-oriented, remote monitoring, those data which come essentially from the patient, consumer, client, whomever, whatever you want to call it, but that are increasing in relevance to both clinical care and public health population management. What is the quality of those data? How do we distinguish those data from other sources of data? What are standards for reporting among those data types? How do we address the quality of those data? How do we harmonize these data and use them in perhaps measures of quality that are using a variety of sources of data? And beyond the consumer sort of intimately him or herself in what are those social data relating to that individual and that is why I see there might be actually a bit of a threshold to having a separate set of hearings about social data, social media, and those data, but we can perhaps touch on that and think through whether or not that is part of this. This is a rich bucket. I can easily imagine some hearings here.

DR. WARREN: — computing — start reading in the area. I am not all together sure what we will throw one into the cloud or how we access it or what it is going to do.

DR. MIDDLETON: Without making it a technology issue there is a set of issues around does Facebook run health care surveys for individuals? Does Amazon or Google run these things and where does this consumer data come from?

DR. WARREN: That is more of what I am getting at is the infrastructure kind of stuff.

PARTICIPANT: Are we going on to another topic or are we still on —

DR. MIDDLETON: I am trying to gently nudge us that way.

MR. QUINN: One last point on this one is that it is not just about the data. We really want in this testimony if we go this direction to find examples, best practices, shining stars of where the health care system has incorporated this into improving care.

DR. CARR: I would say what we are really looking for are strategies on how to use crowd data. When I go on to Amazon, they are pretty right on about what I like to read. And when I go to the Globe, they also seem to know what Amazon knows. But when I hear Twitter, I know you don’t have my perspective and maybe some of my type of people generational or otherwise. I think the real question is not what should go into the cloud or what not, but when it is there what do you do, how do you validate, and go back and say it looks like the Google thing. Millions of people asked about nausea and vomiting. Do you think there could be an outbreak of flu? In fact, oh yes, there is. Or the Marc Overhage thing. It looks like there is an outbreak of no cough. Is that true? No, that was an alternative incentive or something like that.

But I think it is really how do we learn as a learning health system. How do we — given that we have this disparate data that is flowing and we will never be able to validate which is exactly right or not right, but then how do you use enormous datasets that may be directional and then how do you go back and validate those.

MR. QUINN: Care delivery organizations or individual practices or whoever who have organically figured out how to do this on their own or in groups. I don’t know. Are there examples?

DR. CARR: Maybe we need to be hearing from Amazon and Google and people like that. That is where the learning is. They have taken this cloud mentality.

DR. FRANCIS: Just as the privacy fly on the wall however way you want to weigh any of those questions, but they are very different kinds of questions that are posed by using aggregate data to find out that maybe there is flu in the community versus looking to see whether you might want to use feedback loops to clinical care, which is some of the quality question. Patients might very easily share on Google or Facebook or whatever information they would never want their doctors to know.

DR. CARR: And similarly in PatientsLikeMe there are incentives. The more you give the more you get. There is kind of a subtle pressure to give up more information.

DR. GREEN: I wanted to go back and consolidate the last 10 minutes or so with one of this morning’s discussions. It has to do with building the delta. It was Mark’s addition. Mark Hornbrook said there is a public education and awareness issue here about the what, why, and how you use this and what it means. That is a feature or a dimension to this discussion about quality measures that are meaningful to people that hasn’t been brought into the discussion this afternoon.

DR. KAPLAN: I am just changing topics, but it is actually sort of related to Larry’s comment. This has to do with how quality data are reported publicly and becoming intrigued by whether or not you adjust. For example, if there is a poorly performing organization that they always say well of course. We are performing poorly. We serve the toughest patients. It turns out that NCQA doesn’t adjust because that they argue that that takes organizations off the hook where other people feel and particularly provider groups feel very strongly that you should adjust for demographic or other characteristics of their population. There doesn’t appear to be much consensus on this issue.

DR. MIDDLETON: It is a great topic. One might even extend that in very interesting ways with the variety of new data types coming to bear. For example, we not only increase genotypic understanding of the patient. What level of adjustment is correct? In the end do we have a set of orphan diseases, patients with a disease of their own? This is very interesting methodological question.

DR. GREEN: — the letter that the secretary already has — you guys repair this. My memory is we said what we are looking for is stratified, unadjusted data.

DR. KAPLAN: Larry, as a provider — I have dealt with this in our quality project in California that provider groups will say that among other things providing a disincentive to care for the toughest populations.

DR. GREEN: That is exactly where that recommendation came from. Those are quality measures for providers and health plans. Our issue is measures that matter to people. And people didn’t want to know that they have homogenized out. They wanted to be their unique particular self and where they fit into California into the health care delivery system as opposed to having been adjusted to out of the review because they were too sick.

DR. COHEN: I don’t think the issue is whether you adjust or you don’t adjust. You need to link the measure to its use and we have been awful at linking the purpose for the use of particular measures. All these measures have appropriate uses, but if we just put them all out there without understanding the context then we end up with obscuring the messages we tried to send. I think the real focus is not choosing a particular measure, but clarifying its use and its limitations.

DR. KAPLAN: I will give you a really interesting example. This is the clinical example. Suppose you have — take something like systolic blood pressure. You are going to pay for performance. You are going to pay when somebody goes from being hypertensive or having high blood pressure to having normal blood pressure. You have one person in there. They start out with a systolic blood pressure of 190 and you get them down to 145. Well, you are not going to get paid because you didn’t cross that magic 140, but you really dramatically reduced the probability they are going to have a stroke a heart attack or somebody else is at 141 and you get them to 139. That one you get paid for because that you crossed the line, but it is only 2 millimeters of mercury.

The issue that was raised when we talked to some clinicians about this was they were saying you know something. We just give up on those people that started at 190 because we know we are not going to get them to 140 where those people that are near the borderline kind of are worthy of our attention. I don’t think clinicians really think that way although people sort of are aware of this and that was kind of the issue that I was getting at. Are you disincentivizing attending to people who are in most need of help because they are not going to get you your good scores?

DR. COHEN: My response to that is you need to change the incentive system to reward people who do 190 to 145. We had a similar problem several weeks ago. We released cesarean section rates by hospital and we tried to look at low risk as opposed to overall C section rates. It was very interesting from the clinical perspective the way the hospitals received the data looking at their level of hospital and where they fell in low risk.

I think the lesson I learned there is again the measure needs to meet the purpose and you need to carefully explain it. And the reality of it is once these data get out there somebody is going to get it wrong, but that shouldn’t stop us from trying to get it right.

DR. KAPLAN: NCQA uses these big broad categories and thresholds.

DR. MIDDLETON: Any other thoughts on this topic which if I can briefly summarize is do we want to now re-examine the methodology around adjustment as it is applied to quality measurement considering the new myriad array of data that are available and used in quality assessment, examining what level of adjustments are appropriate for the purpose of the measure and its intent. I am going to suggest —

I was at a meeting last week with Arnie Milstein and he said if you talk about incentives — you are not allowed to talk about incentives. They are toxic. Until we repair the health care system I don’t think we can address the incentives, misalignment, or problems. Accepting discussion of the incentives around measurement and disincentives that might apply as an unintended consequence let’s talk about the methods of adjustment given the new types of data that we are using to assess quality. Does that sound right?

The one I do want to throw out, Justine is giving me permission, is this idea of how do we think knowledge management really plays into the mission of quality assessment, the management of populations, the efficacy of patients, and even the mission of NCVHS. In many ways what we have experienced in the partners provider environment is that of course the knowledge base is extraordinarily variable across physicians. It is applied in a wide variety of ways with extraordinary variation in care locally. Of course this is seen then regionally and nationally. And that failure to attend to the knowledge management problem itself leads to perpetuation, unintended consequences, and the inability to actually affect this learning system. The learning system has to feedback into the practices of care and the knowledge that is put into clinical systems and what not. Something in this space seems to me to be relevant at some point for NCVHS. I will leave it open for discussion.

DR. CARR: Knowledge management — getting back to Marjorie’s point. If we are advising on data and quality of data and so on, I think of knowledge management as after we build all these EHRs, how do we put clinical decision support in so that we can make best practices available. Let me be really clear. There is a gigantic need for that. It is a huge enterprise and it is a fundamental piece of the success of the EHR, the value of it. I would just put out there is that within the purview should we be thinking about — that is really quality of care more than quality of data.

DR. GREENBERG: The data implications of quality of care we are always talking about. Philosophically I am right with you, but I am trying to understand what you are suggesting the committee might do in relationship to knowledge management or knowledge —

DR. MIDDLETON: It is sort of at the most practical level. Right now there is little standardization of knowledge as it is implemented into clinical information systems. There is a lot of — research. ONC has sponsored a number of projects. AHRQ has sponsored a number of projects. This has been something in the clinical informatics world that has been of interest for decades. There has never, however, been previously kind of the mandate explicit or not explicit to actually sort of get our act together and build repositories of dollars that enable HIT to influence behavior in all the ways that CDS has been shown to do.

In some ways I think the current HIT policy framework at least from ONC suggest the destination of meaningful use, but doesn’t really describe the journey and many people are getting hung up on the journey and this is the chasm step, if you will, no pun intended, but most people upon implementing HIT never get to the — don’t have the ability or the facility with which to translate the guidelines into the HIT. As Chuck said, some evidence would suggest it takes 17 years. That is not even considering HIT. It might take longer when one considers the HIT side of the problem. Maybe it is better in some cases.

The basic question is if we worry about data and we worry about information in the NCVHS, we worry about health statistics at large; this is kind of the unspoken elephant in the room. Do we want to actually begin to think about how to address it and how to make some guidance or policy statements or directional statements that would help the secretary understand the problem and think about ways to address it?

DR. HORNBROOK: We are trying to deal with that in a micro sense in Kaiser because NIH is holding us to the accountability standard of what difference did you make in your health care system from your research findings. And of course every time we want to talk to the physicians about our research findings they are too busy to talk to us. Anything we suggest to them they say they don’t have any money to make the change anyway.

One of the questions of course here is that part of the learning organizations as the data get better used, you get better data and it is a continuing process forever. It is not just a one-time thing. And the problem we are having right now is the quantity and quality of data has gotten better, but the physicians aren’t ready to put their scientific minds around it quite yet. There are vanguard leaders out there who understand this. They are working nights and weekends in between patients to build databases to convince their peers of how to use the clinical data. I know we are doing that in heart, disease, diabetes, and cancer.

I think eventually we are going to get a whole generation of physicians who will grow up with these data systems and will think that self teaching automatic to this learning is the way to go, but we are having transition problems and we have to figure out the way to break the cultural barriers.

DR. WARREN: I would depend on that and this is a conversation you and I have had several times Blackford. As I look around and most of the students that I have are in organizations that are implementing or refining their EHRs, very few are at the point of no EHR or purchasing. –

I am constantly amazed. These people are usually the leaders of their organizations and yet they have no idea where to go get a content for their EHRs. Their idea of getting content is to bring their friends together around a table and they all vote for what they would like best. That is not the way that we need to be doing evidence-based practice in this country.

This is where knowledge management gets in. There are two components. One, figuring out the tools and stuff, to adequately do this and also educating the workforce that this is what we do and what the tools are and where the sources are.

MR. QUINN: The hearing that we had on meaningful measures — I have a feeling if we brought folks in to talk about clinical decision support that we would have a similar discussion that we had where we are hearing about a million great initiatives where there is people working really hard around the table and each one is slightly different and consuming massive resources. And we could probably write the same letter and just cut and paste quality measures and clinical decision support. That would be easy for me.

PARTICIPANT: One of the things I was thinking other than it being easy for you, Matt, is earlier today we were talking about an order catalog being standardized. That is a prime example of where knowledge management is needed.

PARTICIPANT: Maybe it is approaching it from best practices.

PARTICIPANT: And some exemplar institutions.

DR. NICHOLS: The member new to the committee who is not here apparently James Walker from Geisinger actually is quite the expert on this issue and had the privilege doing ground rounds up there and the deal you do as you do ground rounds since you have to drive so far then they give you the briefing on everything Geisinger is doing. And Walker talked about how they just basically decided to invent their own decision support because what goes out there was so bad, therefore, built into their electronic system quite the elaborate apparatus. There is a Hell of a lot of good stuff going on.

DR. COHEN: When you talk about decision support, it should only be clinical decision support. This is certainly relevant to our early discussion about population health data and community initiatives. We generate all these indicator data for communities and they have all this linked data. What do they do with it? What do people do with it? I think the key is developing the tools and as well as the data and helping provide guidance around options for use.

DR. MIDDLETON: Maybe actually the nomenclature is not quite right. Maybe it is more about clinical decision support or best practices that we could have some discussion around and as it pertains to then the data input to decision support would be something that we are very comfortable with and how the privacy and population and individual issues all have to be addressed. I am trying a little reframing. Does that help or does that hurt?

DR. COHEN: It helps me understand when we talk about knowledge. You create knowledge with information and told us to use it I guess and then helping people decide what to use and how to use it.

DR. MIDDLETON: One thing the NQF has done and has been working very actively on is the whole quality data framework, quality data measures framework. The quality dataset is the set of defined — it is a defined object model and defined ways in which to make quality measures and what not. That is pretty well along the path and I think pretty sophisticated. They recognize it of course that quality data framework is just the flip side of clinical decision support. The same logic that you are trying to assess in the hemoglobin A1C compliance rate is going to be same logic, if you will, implemented to remind the provider to do the hemoglobin A1C.

PARTICIPANT: It is retrospective in one’s real time or prospective.

DR. FITZMAURICE: One thing that we could look at is what is the path from data to research findings to guidelines to quality measures to clinical decision support. It is when to follow a finding all the way through. But practically what do we have to do to move from research findings to get a guideline out? You have to work with the specialty society. Once you get the guideline out, what does it take to put it into a clinical decision support module and can that module be put from one place to another to another?

Second idea is how do we store knowledge? Is it in clinical decision support modules? Is it in knowledge trees where we know this for certain, but it branches and we don’t know what to do with those branches? Where is the uncertainty in a given bit of knowledge that requires action — judgment resolves — the differences where you don’t know the probabilities or you don’t have very good guesses at them. How do we store this information? And then can you do a — kind of accumulation of here is the sum of professional judgments. Now does this beg for some research to say is the professional judgment right and then everybody should be doing it?

DR. WARREN: You are talking about knowledge management and you are linking it with clinical decision support. I would say it is much broader than that. It is also about what needs to go into your order catalog. What needs to go into your assessment forms? What needs to go into all the other components that we have in EHR? We need to take those guidelines and make them actionable to where they are totally invisible to the user. And yes there do need to be some rules that will fire that will screen things so like on admission certain things happen to that patient that research tells us needs to have happen and things are automatically put on a problem list or an order set comes up for you to view and determine which ones of those orders need to be personalized for your patient. Which things do you need to continue to document so that you do generate your outcomes so that you can then make your quality indicators be a by-product of care? They shouldn’t be something you go out and search for or query for or do extra work.

DR. MIDDLETON: And when is your clinical decision support a partnership with your patient? The two of you are in there together playing around with the data.

DR. WARREN: Especially if you want patient preferences.

DR. GREENBERG: One of the things that is sort of percolating in my mind probably because Larry is two people over there from me. Also, what is the data that we are not routinely collecting that can contribute to what Mike is talking about those probabilities and everything and that gets to this primary care data model? And knowing systematically collecting the reason the patient actually sought care. I don’t know whether that is systematically being collected in most electronic health records and what they are using as the terminology, but I do know that that is not systematically collected in administrative data. There is just so much research that doesn’t so much come from the US. It comes from the Dutch system and others that you can build a lot of knowledge from that and decide which way you are going to go based on what the patient initially presented with from the patient’s point of view and then what the outcomes were, what kind of tests were done and all of that. I just wanted to throw that in.

MR. QUINN: That would be the — Michael Klinkman spokesman of that at our medical home hearing, the ICPC, which is used as I understand everywhere except here.

DR. GREEN: There is a beautiful linkage — that happened here is that quality measures that matter to people. There is a very strong case to be made of what people care about is whether their goals for their health are being pursued and met and if the problem they think they have has been addressed satisfactorily. We collect neither. What we do is we behave as if all patients arrive tattooed right across your forehead I have an L4 herniated disc that is one of those that would respond and is — going to require neurosurgery which is .7 out of 100 or something like that. They don’t come that way.

They come out and say it is my grandkid. I was playing with my grandkid. And ever since I started playing with my grandkid, my back hurts. Their goal is to be able to keep playing with their grandkid and they are satisfied when the health care system does something that helps them meet that goal. And when you explain to them that the reason their back hurt is that is fine.

So much of our conversation is so far downstream that when we start talking about measures that matter to people, we can’t recently expect to be responsive because we missed the start.

DR. MIDDLETON: And those measures that matter to people if they aren’t created for those conditions where there are preference-sensitive outcomes, we will never know. Very shortly we are going to be asked to take patient preferences in as part of meaningful use. No one knows how to measure patient preferences and there are no standards in all the rest of it. And then care is supposed to be stratified based upon patient preferences and all the rest of it.

DR. GREEN: You could put this on a CCD, Blackford. CCD could have as a data standard the patient’s goal.

DR. MIDDLETON: I love it. I am with you. I want to be mindful of the time because I did promise to close on time. That is very important because we are going to get to go out to dinner and have some fun. I think there is a sort of a gemish of ideas here. I will try to capture in a paragraph or two, share it with Matt and Paul maybe for an internal sanity check, and then share it with a subcommittee and we can take it forward with the other two ideas I think have come forward about quality assessment and adjustments and all the questions there that arise with new data sources and types.

And then perhaps the biggest one and maybe the one to do sooner than later, but let’s put them all together is about how to — what are the issues surrounding patient source, self-report, consumer source data, remote monitoring, all the outside of the care system data which are relevant in there.

MS. GREENBERG: We kind of came full circle. We started with patient source data. Then we talked about getting from the patients what they really feel about —

MR. QUINN: To go back to what Larry mentioned before about measures that are relevant to patients and to go to what Len said before about information supplied to health information or to the insurance exchanges on rating and ranking and things. It would be great to tie those together.

DR. CARR: One of the things is that we meet four times a year and then we have some hearings and some phone calls. We have to be cognizant of the resources available and the tasks we set out to do. What is not good is when we have a very ambitious thing that we couldn’t possibly do or that it is very ambitious and we stack all the resources of the committee so nothing else gets done. We have done all of the above.

I think what we want to do is to remember the precepts that I put forward today that we want to get out in front of what is going on. One caution with this agenda is if there are five other agencies working on it, we have to think very clearly about what is the value that we add.

And then secondly, the more we resonate with what the work that we are doing as a whole, the greater the benefit will come from it because it will be really a step forward coming from all sectors.

DR. MIDDLETON: Okay. Let’s aim to break in the next 90 seconds or so. The only other questions I have for the subcommittee and I am very pleased of course with now more panel members and more horsepower and brilliance to draw upon for all of our work.

I do want to ask the question do we have the right repreesntation from relevant agencies for this last question. And I am just struck that actually no one from CMS seems to be coming unless I have missed that.

MS. GREENBERG: I know Lorraine Doo is going to be here tomorrow — and then of course Friday. This is kind of one of those three-day meetings certainly for CMS. Do we have someone from CMS on the staff to this subcommittee — to the quality subcommittee?

DR. FITZMAURICE: Maybe somebody from OCSQ which deals with quality measures.

MS. GREENBERG: It looks like we don’t actually have someone at this point. We have in the past —

DR. MIDDLETON: Here is a suggestion. Let’s raise the question. Do we have the right liaisons from the relevant agencies — across the government in this committee? We have a lion here and he is strong, but we want to make sure we have —

MS. GREENBERG: Absolutely. We have people at NCHS who have worked on these — and with AHRQ on the quality reports.

DR. MIDDLETON: It is a question to be answered after we close today.

MS. GREENBERG: We need to reach out to our liaisons. It would be best perhaps since we don’t have them right now from some of these to see what direction you really want to focus on say in the next year or two so we can try to get people who can really contribute to that.

DR. MIDDLETON: And then the very last thing is I love the summary work plan in the back. I was a little chagrin that I think the quality section of this summary work plan on the back is the weakest. I would like to fix that.

MS. GREENBERG: That is why we put it out there.

DR. MIDDLETON: Let’s try to flesh this out with some of these ideas and then share that with the subcommittee and make sure we got the 18-month vision and roadmap in line which is always subject to revision.

PARTICIPANT: Maybe we could consolidate with the populations’ one —

DR. MIDDLETON: I am going to leave that topic until tomorrow. Any other business for today’s meeting? Thank you all. We are adjourned.

(Whereupon, at 5:36 pm, the meeting was adjourned.)