[This Transcript is Unedited]
Department of Health and Human Services
The National Committee on Vital and Health Statistics
March 1, 2013
Hubert H. Humphrey Building
200 Independence Ave., SW
Washington, D.C. 20201
CASET Associates, Ltd.
Fairfax, Virginia 22030
TABLE OF CONTENTS
- Welcome – Dr. Green
- NCVHS Strategic Session – Dr. Carr
- Population Health – Empowering Communities Follow-Up – Dr. Cohen and Ms. Kanaan
- HHS Update: Health System Measurement Project – Dr. Queen
- Remarks by the Chair and Feedback from the Membership – Dr. Green
P R O C E E D I N G S (9:09 a.m.)
DR. GREEN: Good morning, I am Larry Green, chair of the NCVHS. We’re about
to convene our first meeting ever in March of 2013, so a propitious moment,
unique event. Welcome back, those of you who were here yesterday or the day
before. We have a very busy morning and also what I think will prove to be a
very interesting morning. Let’s go around the table with introductions of who’s
in the room.
DR. GREEN: Thank you all. Our first item of business today is going to be
hearing from Dr. Ed Sondik.
DR. SONDIK: I just want to take a few minutes to, well, to be straight out
about it, to reassure the committee of its continuing support from the National
Center for Health Statistics. The committee, and I say this in light of
Marjorie’s coming departure, Marjorie, of course, has been an institution here
and with the committee. She has had, at last count, four roles, depending on
how you count, actually, within the National Center for Health Statistics. She
is hell-bent on retiring, not going to change that I gather. I ask her each
time we meet. Replacing her is a bit of a challenge. It really won’t be
replacing her; it’s going to be replacing the several hers who are involved in
this. I just wanted to underscore the importance of the committee to NCHS and
underscore what I hope is the importance of NCHS to the committee.
For many, many years, this was the sole advisory committee that NCHS had. As
you all know and know what your vision is, you advise the Secretary, but that
advice that you give the Secretary has a good deal to do with how we function
and the direction of NCHS. For members of the committee, NCHS is not just
another center in Centers for Disease Control and Prevention. It’s the
designated federal statistical agency that focuses on health. We are one of the
group of agencies like the Census Bureau, the Bureau of Labor Statistics,
Education, and I could run through all 13 or 14 of them but I won’t, that
operate under a somewhat different set of principles and practices than other
We collect information for research and policy. We do not sit on the data.
As soon as it’s ready to go out, it goes out. We collect it without fear or
favor, regardless of who has helped support it. This sometimes gets us in some
hot water. I’d say we’re pretty much independent in terms of editorial content,
in terms of what it is we publish and how we do it. The guidance from this
committee and now from our Board of Scientific Counselors, which tends to focus
more on the way we actually do our work, the technical aspects of it, whereas
this committee in the past, has focused more on the policy aspects, on the
direction. The guidance from this committee has been very important to us.
The agenda of this committee continues to be very, very full. I really
encourage the committee to periodically reserve some time to hear from the
Center, and whenever I’ve wanted to say something I’ve not been refused, but I
think a regular time to hear from the Center and where it’s going, is very
important, coupled with, and I say this looking at Jim Scanlon, the data
direction and policy direction of the Department as well.
What we do is very much related to what others do. The more that we can
integrate these data activities across the department, the better. Minimizing
but not eliminating duplication is very important. We don’t want to eliminate.
There can be very productive duplication, collecting information in different
ways from different populations, so judicious duplication, I think, is really
In my view, the committee hasn’t spent a lot of time on that aspect of the
Department. I think that would be really useful in the future. Related to that,
if I could be so bold, I’m not sure Larry completely agrees with this and maybe
he can voice his opinion, but the agenda of this committee is important
obviously to the Secretary, and therefore to the Department, but it is
particularly important to– the technical term—the optives, the agencies
within the department: NIH, CDC, HRSA, SAMHSA, AHRQ, without a doubt, it really
cuts across. Yet, I don’t think that the advisory committees for those agencies
know about this advisory committee. I think it would be very useful to
periodically share the agenda and themes that are being addressed here with
them. I think that would be very helpful.
There are an uncountable number, I think it’s an alpha-zero actually, an
uncountable number, out of zero, of such committees in the department. There
are a lot. I’m not suggesting interacting with necessarily all of them, but
there are a few key ones. The CDC Advisory Committee to the Director, the NIH
Advisory Committee to the Director, these are really extremely important. I
think communicating the direction that this committee is taking and the issues
that you see particularly related to the community use of data and access to
the federal resources and melding that with local resources is really
It’s really important to this committee no question, but I think it’s
important to the rest of the Department as well. The more that this could be
stimulated and coordinated the better. In terms of the support of this
committee, we’re working out the plan for that. I can assure you that within
the next month or so that will become clear and that this committee will
continue to have support. Even though there might be changes, there will be
changes, that support will not let up and there will be a smooth transition to
I’ve taken a lot of time here, but I want to take if I could, just two more
minutes and talk about NCHS. I can’t remember the last time I actually came
here and gave an update on where we are. I thought I would take this
opportunity since I grabbed the microphone, to do that really quickly. I did
want to run down where we are. I think the Center—I can’t get straight
whether sequester begins after today or it already began. I was feeling a cold
blade close to my neck, but that must be a psychic connection. In spite of
that, it turns out that from a budget point of view, and I shouldn’t say this
too loudly, we’re actually in a very good place.
The Center has never had as large a budget as it has now. It never had as
much in terms of the contributions from our collaborators– she says not to say
this. I think it’s important. I think it shows the support for data in the
department, which is really very strong. We get an appropriation that goes
through a department at CDC. On top of that, we get a large number of
collaborators, some of whom assist the data collection with funds, some of whom
don’t. Whether we do something is not necessarily a function of whether there
is money attached to it. It really is a function of priority. When you pool all
of that together, we really are at a point—we’ve really never been this
We are prepared, though, a la that knife and the cut, to make the necessary
cuts if they come. We have plans for draconian cuts. We can and will do that,
but we don’t want to make those cuts in a way in which there’s a significant
gap in the data that we collect. There will be some. There’s no question about
it. The detail, some of it will go by the boards. We don’t need to go into that
right now, because I have it on really good authority that the cut isn’t right
now. It’s coming later.
In terms of our programs, the health interview survey, which is the core of
the department’s data collection activity, has a sample size this year of
125,000, which is 50 percent above of what’s normal level, which has been
87,500. This is extremely important, because it enables us to do more in terms
of state estimates. We now can actually do state estimates from the HIS for the
great majority of the states, not all but the great majority. I think that’s
really crucial. Any cutback in funds will cut that back, continue the HIS, but
it will cut back on that detail.
NHANES continues to be the most critical support of clinical measures of
health in the country. This year it did even more than it’s done in the past,
including a youth fitness survey that’s very important to the First Lady’s
efforts, but in general to the health and fitness of kids. I give great credit
to them for taking this on and in really short order coming up with a really
rigorous protocol and then conducting it for the year.
Our healthcare surveys have been revamped considerably to be even more
responsive, but on the ambulatory side. We get a lot of hospital information
from a number of sources, including AHRQ, but the ambulatory side is really
where we think we have a gap. We’re filling that even more effectively, and the
sample size there has been boosted enormously this year. I hope that we can
keep it up in the future.
Vital statistics I think of as the core of the health information we have in
the country, even though it’s only information on both sides of our ledgers so
to speak, speaking metaphorically. It is crucial. When we compare the U.S. to
other countries, we do it using the vital statistics in large part. The
timeliness there and the use of electronic methods is increasing enormously.
There’s a great priority on doing more with that. When we put it all together,
I think we’re doing well, but I think that it’s important that we continue to
get guidance from this committee and others on the direction and on priorities,
on special focuses, on what you all see as coming issues.
Collecting data on something that’s a past issue isn’t particularly
effective. The trick here is to figure out what the issue is before it’s an
issue to be able to collect information on it and track it over time. I think
we have a pretty good staff who are able to do these kinds of things. The more
guidance that we can get on this the better.
There are some challenges to all the data efforts that are coming along. I
think these challenges have to do with the use of surveys. Surveys in some
aspects are really a technology that is kind of passé. In a way, it’s
the best if we don’t know how to do it better, but there are other data
sources. We have to figure out how we’re going to use those data sources,
everything from electronic medical records, which are not an immediate panacea
here, but will be valuable I’m convinced down the road when we figure out how
to use them.
Crowd-sourcing, and my colleagues in the federal statistical area get very
jittery when they hear about this because it’s the height of imprecision, and
at the same time we know that we can learn a good deal about that. We have to
figure out how we can harness that. We also have to be aware of and be wary of
commercial sources of information that with their budgets are able to publicize
their information very heavily but don’t necessarily have the rigor and the
precision that goes along with the federal sources.
All of this needs to be put together in terms of the policy for the
agencies, and your guidance in that would be very helpful. That long-winded
explanation is really my statement to assure you that we will continue, and of
course the department can speak for the department, but we will continue. We
value our relationship to this committee and see it continuing in the future. I
thank Dr. Green for giving me the time this morning.
DR. GREEN: Thank you very much, Ed. I hear you are changing your life.
DR. SONDIK: I’m reprioritizing. That was the word I came up with this
morning. I was at Stanford for six years. I came to NIH for two years, and then
we stayed for 20. My wife has not forgiven me for that, that’s true, for
leaving California. Then I’ve been at NCHS for a prime number of years, almost,
17. Only you all would get that. I really felt that my grandchildren and
children were getting older, and it was time to reprioritize really. I love
what I do, but there’s a cost to that. There’s an opportunity cost to that. I
just decided to make that—there’s nothing underneath it. There’s no hidden
agenda. It really is that. I promised my family that I would do it. Every day
that I didn’t do it was a lost opportunity. I don’t want to go away from the
professional side at all, and I don’t expect to do that, but my family, my
wife, and I needed more flexibility.
I’m going to be on the rolls until May, at least that’s the plan. That’s
flexible, I suppose. An acting director is going to take over on April
1st. That acting director is Charlie Rothwell, who is the head of
our Vital Statistics Division. Dr. Frieden picked him. They have a really great
relationship. Charlie knows what I thought would be good in an acting director.
I said it needs to be someone who knows the programs at NCHS. You never know
where someone might be picked from. He needs to know that. He needs to know the
department’s programs, needs to know CDC, which is extremely important, and
especially needs to know the federal statistical system. Charlie knows all that
plus he knows intimately the state side of things, which I think is crucial,
absolutely crucial. I think he’s an excellent choice to carry on the mission. I
gather there will be a search for a new director. I don’t know how long that
MS. GREENBERG: I wanted to comment on something in particular that you said
about reaching out more in the department, and I will in a minute, but like
you, I now have the floor. I just wanted to say that I think Ed has been a
great supporter, obviously, of health statistics and using data for health
policy and working both within CDC and the department but also with the federal
statistical agencies. He takes all of those roles — I may have four roles,
he has about 20, probably—takes them very seriously. He actually was an
acting director for the National Center for Public Health Informatics at one
point. He’s too modest to say it, but he was the acting director of NCI. To say
that you’ve had an illustrious career is an understatement.
On a more personal note, he has been a tremendous supporter of this
committee and actually all of the activities that I care so passionately about,
the World Health Organization activities that I have been fortunate to lead and
be involved with, the informatics focus that I started a number of years ago in
my group and working with the convergence theme that all of us are talking
about here. We’re using administrative data, clinical data, public health data.
He’s even thinking ahead to social media and all of that. I’d say he’s been a
very forward thinker.
I have found Ed tremendously supportive of all of those activities, and I’m
very grateful to you for it. In particular, we have had so many conversations
about how important Ed thinks this committee is. He’s not just saying that to
you guys because he’s here and it’s the right thing to say. He really does have
that broader perspective of not only the day to day responsibilities of NCHS
and how important it is to NCHS, but of the need for NCHS to be part of this
broader activity of going on, whether it’s health reform.
Obviously NCHS has been the lead in collecting data for healthy people. I’d
say he’s always had a really broad view, and he really values this committee
and has tried to protect, even in slimmer times, the budget, the NCHS budget
for this committee, which of course is very generously supported also by ASPE,
and Jim has also tried to protect it. I don’t think any of you have to worry at
all, even though, I must say, it wasn’t part of my plan that Ed would retire
before I did.
On a very personal note, it wasn’t part of my plan that I would suffer this
other loss that I suffered this week, which is more permanent. Life is what
happens when you’re making other plans.
I am really confident, and I have worked with Charlie Rothwell for it seems
like my whole life. Charlie has also been very supportive of going back to the
public health data standards consortium. He put in the first piece of money for
that. All of the things that I feel are very important for NCHS and for public
health; Charlie has been a leader in all of that. I also would like to reassure
you in that regard and to publicly thank Ed. Since he’ll have already retired
by the time I retire, so let me do it now for all the support you’ve provided
me, and not just me personally and professionally because you have, but for all
the things that I’m so passionate about. I’m deeply grateful to you for that,
I did have a more practical thing to say. Going back to one of Ed’s
recommendations, and we did use to have more periodic reports from NCHS. We’ll
process that in an executive subcommittee, that’s a point well made. We’ve even
had Charlie report here, but more on vital records. We do have liaisons from
AHRQ. We don’t currently have one from CDC after Seth Foldy left, and before
that for a long time it was Steven Steindel, but we should reactivate that. We
have a liaison from NIH, and Larry just got an email from him, in fact, that
he’s going to bring up Bob Kaplan. We have a few other liaisons as well, from
ONC. We do hear regularly from them. I said AHRQ, obviously.
We have a mechanism to work with those liaisons to see which are the
committees that really advise on a broad way their agencies and how should we
best interact with them. I know that Justine used to participate in one of the
AHRQ groups on behalf of the national committee. Of course, we have some
members who are already on some of the ONC committees. I did want to let you
know, because I hadn’t mentioned this, that as of January 1 the ONC reached out
to me as the executive secretary of the committee, and we’ve had our first what
is going to be a quarterly conference call between the staff for the ONC
standards and policy committee and the NCVHS, in which we’re sharing work
We’re trying to collaborate more on joint projects, et cetera. I think
that’s a very nice model possibly for some of the other committees. I was very
pleased that they reached out to us in that way. Debbie Jackson and Katherine
and I look forward to that collaboration. I just wanted to let you know about
DR. GREEN: Ed, I’d like to ask you to do four things right now. First is to
stay in your seat just for a minute. Second, would you provide us with a list
of not the agencies, but the actual contact persons that we should have that
communication link with? A third thing is I want your post-April contact
information. The last thing I wanted to do is just give you my personal thanks
for a prime number of years of being effective. If you’d just sit there for a
minute, I’d like to ask everyone in the room to stand up for a round of
applause for Ed Sondik.
DR. SONDIK: Thank you very much.
DR. GREEN: Justine, while you are set up, here’s the plan, we will see how
it goes. I’m betting that Justine, because of her skill set and experience,
will be done in 20 minutes. Then we are going to spend about ten minutes where
we will run through the sub-committee co-chairs where we are going to summarize
our current predictions of the work and the key points of work and engagement
between now and September, to continue the work list that we’ve been doing for
I’ll just ask the sub-committee chairs to speak in brief about what it is.
Debbie is poised to capture it. We’re not going to argue here today about the
adjudication of this, that, or the other. We’re going to have the aspirational
work list for now to September in about ten minutes. That will set us up to be
on time at 10:15 to hear the presentation from the population sub-committee
folks. Rumor has it that we will have a break shortly after that. Justine,
DR. CARR: I just want to say for the new members, you’re at a moment in time
in history– Marjorie and Ed are just pillars, infrastructure, and bedrock of
all that we’re working on today and we work on so much. I think this is a
momentous occasion. We’re casual because we see you every day, but for the
country, around the world, you guys have made a tremendous difference. It’s
been an honor to be with you. Today is a special day like no other, so thank
Workgroup on Data, HHS data quality, access, and use, just to review the
charge, we had three points. One is to monitor and identify issues and
opportunities to make recommendations to HHS on improving data access and
innovative use, including content, technology, media, and audiences. The second
is to advise HHS on promoting and facilitating communication to the public
about HHS data. Three is facilitate HHS access to expert opinion and public
input regarding policies, procedures, infrastructure to improve data use.
I put this together to summarize where we are. We have two streams. We have
on the top the supply side, which is the work that Todd Park began, which was
first to inventory the data available then configure it in a way that would be
usable to the public and then post it on healthdata.gov. Then we have, as we’ve
heard so much this meeting about the demand side, what are the needs of the
communities, how do they translate their clinical or public health needs into a
data story that can help them articulate their issues, and then where do they
find it, and when they get it, how do they use it? I think these are the two
themes that are ongoing, and I think the challenge for all of us, but the
workgroup in particular, is to focus on what are the gaps for these two groups
to get together.
Our current state is the following. The Work Group has reviewed the selected
set of data was presented to the workgroup over the summer. In addition, Susan
has done some wonderful work, Susan Queen, in doing an inventory of the data
sets and their uses. So not just what they have but the specifics of the
timeframe they cover, whether they have a cost, whether they have privacy
things, et cetera, and we’ll be talking about that today. We’ve spent a fair
amount of time identifying the supply side issues.
We’ll hear today some interesting statistics on how often the site is
actually accessed and for how long. We’ve had some very robust discussions of
usability, understanding, what are the needs of a developer, if they want to do
that, things that may be obvious to clinicians about what’s a discharge, what’s
an ED visit, what’s home care, or not obvious to developers, and where is that
Also, usability related to unsophisticated users, how do you access this
data not to build an app for iPad use, but simply to look at your community and
understand what are the incidents of various public health issues.
Thirdly, we’re in the process now of learning what can we learn from current
users, and the landscape that Susan Kanaan as done and we’ll hear about today,
is also very important, so what is already happening.
Moving to the demand side, that’s our focus for discussion today. What are
the needs and then how to navigate resources and how to assess the
appropriateness of all available data sets, and it gets a bit back to taxonomy,
whether it’s a very granular definition of something or an aggregate
definition. Actually Bill Stead and I, had a very valuable conversation
yesterday, and I am very much lobbying all of you, but especially Bill, for
input on the workgroup.
Next steps, three things, one is we’re working on defining this gap between
supply and demand. This is our ongoing work. We will be participating in
whatever we’re calling that learning session on April 30, May 1 and 2. Second
is defining strategies to bridge the gap, and we’ll be hearing today from Bill
Davenhall who’s taken the great work that Susan has done and given it a
Healthgrades five-star rating kind of thing. I think just having that out there
if you are a person who needs x, y, and z, go to this group. We’ll talk more
In addition, stimulating innovative uses, and this is a little bit broader,
but the more we understand this, our goal, we want to get this out into the
domain of innovators.
Thirdly, broad diffusion of available resources, and how to do that. There
is healthdata.gov, but it’s cumbersome, and it’s hard. I really like Bill’s
idea that we’ll talk about today, of something that you can go to, and at least
you know when you’re in the right direction and know what you have. We will be
participating in the Datapalooza. That’s something else we were talking about
today. Our goal really is to be either part of a demonstration, kind of
education, about the data availability or how to use the data. That’s for
discussion today. I think that’s it. Any questions? I invite the members to
enhance, expand, redact, or otherwise contribute to what we’ve said so far.
Lily Bradley is joining us as our liaison.
DR. GREEN: Let’s please introduce everyone.
(Intro Work Group members)
DR. SONDIK: I just wanted to mention something about the Datapalooza. I’ve
been charged with organizing a community health data session. I think
integrating these together would really be ideal. I’ve been given what amounts
to the afternoon on one day, I’m not sure which it is, which is essentially two
sessions together for any sort of format. I do have an outline of what I’ve
been thinking about, and I’d be happy to share that later. I think this would
modify it, but I think it would fit. I think it would be a good thing to do
unless you’re interested in something separate from that.
DR CARR: It would be helpful to see what you had in mind today. Josh
Rosenthal is not here today, but he also is doing a lot of things,
demonstrating things. I think it would be a very nice fit. Your session last
year was tremendous, very well attended, great discussion. It would be really
valuable to be in that part. Josh has a lot of demos, which I think excite
people and show them the potential and how to do it, but I think there’s
another part of the story, which is how do we get from here to there.
DR. SONDIK: I really wanted to see an emphasis, at least one of the
emphasis, on need and hear from a set of people who could really speak to that.
Another portion would be showing whiz-bang things that really could be done,
and that’s kind of exciting. I think what’s crucial in this is what difference
does it make. We got at that a little bit, but it’s the healthy people issue.
You can set targets and so forth, but do they really make a difference, what
kind of progress is there, and is that aspect of it making a difference. I
thought, given the level of sophistication that we’re rapidly building in all
this and the work that this committee has done on community needs, community
use of data and so forth, I really thought we were at a point where we could
really start chewing on some hard questions. I can retrieve my outline, which
is just my outline, so I’m happy to discuss it whenever you say.
DR. GREEN: We’ll have an immediately formed sub-workgroup comprised of
Justine, Linda Kloss, and you to find out what’s going on in which places and
MR. SCANLON: Remember the origin of the work group was the leadership in HHS
and other places wanted to have an advisory committee to advise us on exactly
these kinds of issues. They thought that it wasn’t not the NCVHS, but we
actually told them that we could create — it’s the obvious place to start
because healthdata.gov, as wonderful as it is, is probably the tip of the
iceberg in terms of the data we make available. So we managed to convince
everyone that what we needed was a workgroup of folks who do this for a living
and to advise us as a workgroup here.
That’s exactly what we’re starting here, and I think it’s moving along very
nicely. The only thing to keep in mind again is that we’re focusing on
healthdata.gov as a place where we can put a lot of the data, but all that we
do on healthdata.gov is eclipsed tremendously by the other ways we make data
available. I don’t think most folks know what those are unless you’re an
aficionado in the researcher/public health community.
I think maybe at a later stage we’d have to think of how we can improve all
of that. We have a long history of creating products that nobody uses at all of
our agencies because it’s what we thought was useful, and nobody every visits
those websites or uses them. It’s a report. I think here we’re trying to do
business differently. We’re trying to start from the need.
DR. CARR: Exactly, and it’s been interesting. Even at our last session we
had some data on how often you go to the primary site and then data on these
secondary sites that have imported the data and much higher use. For us to
discuss is why is that, how do we make that better. I would like to invite
other workgroup members– Bill, did you want to add anything?
MR. DAVENHALL: I just want to make a comment about the Health Datapalooza
and just update everybody. There’s been formed now the health data consortium,
which is going to be a 501(c)3 organization. The purpose of that is to
intentionally permanently give a home to the Datapalooza and the activities
that the Datapalooza has created. That will be announced at the Datapalooza.
The only conflict I would have here that I have to mention is I’m part of the
sponsorship committee of the Health Datapalooza for the last several years.
This has been a great effort of the Department and the IOM to make this
successful. I would say it is way beyond successful. It’s now likely our
problem is too many people.
It clearly gives us hope that there needs to be a more permanent addressing
of the problems with accessibility and usability of data. The work of this
workgroup will feed right into another organization who stood up like NADO or
other organizations who will tap into the cold face of all those folks who need
that data to provide data accuracy. I think that’s a very positive thing to
look forward in June.
DR. CARR: The creation of that speech to the fact that although it’s fun to
have the prizes, we’re really looking for sustainability models that will make
this part of the fabric of what we do.
MR. DAVENHALL: I think things we learned at the datapalooza was we did a
very good job of exciting people about the innovation. Then what we found is no
one helped anybody build a business out of it. It’s questionable about its
sustainability. Part of the effort is to beef up that part of it so that not
only will we get the ideas, but we’ll get the health that people need to create
essentially sustainable businesses. It won’t work unless that occurs.
DR. TANG: This is actually sort of a question to Ed. We didn’t get a chance
to ask Ed, but it ties into what he just presented. He made a couple of
comments in passing. One is he just said surveys are a bit passé, and I
don’t think you mean exactly passé, but there are other ways to
The other thing is you said that measures are a bit of measuring the past,
and what other options are there, taking advantage of some of the electronic
infrastructure we have now. In your current state, you have a little bit more
freedom to answer questions. We talked a bit about trying to leverage some of
these other electronic media to get access to both the docs and the consumers
and patients, which would require a lot of retooling on the part of the
Is that something that the department’s receptive to, do you think? How much
would you purchase to pursue those areas and that both address the supply and
most importantly the need? An example, a lot of reporting is retrospective and
done infrequently. If we could get more towards near real word time reporting,
it would influence the actual decisions made both at the individual level and
at the local population management, community management level. Could those be
something that are used both locally but also nationally?
DR. SONDIK: In terms of surveys, getting an adequate response is more and
more difficult. The standard response this rate these days from telephone
surveys is well below what OMB would accept as an acceptable response rate. It
means you really have to do some considerable amount of study as to the
non-response bias. Even after you do that, you’re still not really sure where
you are. When I say that it is passé, it’s that there are other sources
of data, but we don’t really understand them at this point.
I think we somehow need to be able to calibrate the new with respect to the
old and with respect to the truth, is the way I like to think about it. There
is truth, and we related that to surveys and so forth. I think you can argue
that surveys—there haven’t always been surveys. These things came in the
40s and then really developed in the 50s, the big national surveys. The
technology and the theory supporting all of it is there. It tends to hinge on
whether or not people are going to respond.
You have to consider this versus panels, which are not the same as surveys,
and understand the characteristics of panels. This may be a very good way to go
in the future. We do it all the time; all of us sitting around this table are
making decisions on the basis of people. We have no idea who they are, but they
gave that restaurant four stars. We need to understand. In terms of the
response of the department, we from an NCHS point of view, we’re very
interested in other sources of information, changing the paradigm as to when
we’re getting that information.
What we need, and it gets to the charge of this committee, is we have to
have resources to be able to experiment and to evaluate. If all we get are
resources to do what we’re doing in a really good way, then it’s hard for us to
do that unless we say we’re going to take time off, we won’t do the HIS for
awhile. That will free up several tens of millions of dollars for us to do
this. I’m not throwing out a Washington monument thing. It’s conceivable to do
I think what you’re bringing up is important for this committee, which
really is the prime committee in the department to debate something like this
together with our board of scientific counselors and perhaps a few others and
give some advice on direction. I’d say we’re very receptive to it. Tom Frieden,
who heads the CDC, wants the most current data that can be produced. There’s no
question about it, and he wants it for the right reasons, the kinds of things
that you said. He’s been very supportive of the move to electronic death
records. Birth, we’re doing pretty well on, but death is another story. I can
give you a very positive response to that. I think this committee could be very
helpful in laying out some direction.
DR. GREEN: We have time for one more comment.
DR. CARR: I would like to let the member’s just say a word or two because I
spoke for them.
DR. VAUGHAN: Dr. Sondik, one of the things I really enjoyed about your
session last year at the Datapalooza was that it seemed to be the place where
all the public health people came to gather and say what is this about. The
notion of bringing the community data piece into that as an explicit part of
the conversation I think would be very welcome and well received. We touched on
it a little bit yesterday, but I think along with entrepreneurial business side
there is the civic entrepreneur, the social entrepreneur, the entrepreneurs
within local government, let alone federal government. It would be great to
think about the new ways to bring them forward.
DR. KAUSHAL: We divided our work in the advisory group on the supply side
and the demand side. I would view the health data initiative as a real focal
point around the demand side. They’ve done a great job of coalescing the
eco-system where it’s public health professionals, but also innovators and
investors. I think my macro point is that we should be very coordinated and
tight with them. Otherwise we can do some great work on structure and this
data, releasing it, but nothing will come of it.
MR. CROWLEY: I would just add the themes of building this community, sharing
this information, connecting the supply and the demand, bridging that gap
between all these research resources and other public health resources in the
community can be facilitated in this day and age by a more socially enabled
network and more socially enabled web assets and infrastructure, so that
whether it’s the entrepreneurs and the data providers or the ones who have
developed the applications so they have a mechanism that’s facilitated to
communicate, to connect, to learn, and share so that some of the lessons that
we’ve seen in such places as the open source committee, which has done a good
job in being able to reuse assets and build new and innovative things, we could
take those lessons and apply them potentially within the health data community
as well. Of the overarching theme, just about making that data more social is
DR. COHEN: My comment, I’m a member of the committee and of the workgroup,
when we look at what Ed and Paul were talking about, figuring out new data
sources, a new way to collect information, provide information, I think
government, the feds and state government, is very thoughtful in considering
supply side. We’re moving forward in that direction, but we’ve never really
systematically evaluated the demand side. The nature of who uses our data has
changed enormously. The targets for our data used to be researchers and very
sophisticated folks who understood the complexity in the nature of our data. I
think one of the huge gaps at all levels from the government’s point of view is
ongoing assessment of who uses our data and how to identify the appropriate
channels to provide the information that people use. That’s the connection we
need to make. That’s the sweet spot for the workgroup, advising the federal
government of how to repackage, reposition our data to meet the new community
of users who we need to get this information to.
DR. FRANCIS: I am Leslie Francis, and I’m also a member of the Full
Committee and a member of the Working Group. It seems to me just from my
perspective on the Privacy Subcommittee, first of all, I hope that the
Workgroup will be very actively involved in the April workshop, suggesting
people coming and participating in the workshop discussion. Two themes that I
think are going to be really important as we go forward, with the research and
the public health community, there are long traditions of trust in data use.
Those are known quantities often. We’re venturing into a world of new users
where those trust relationships are going to need to be built. That’s something
to be aware of.
A second thing is we are used to data that when they’ve been made available,
they’ve been either public use data sets or they’ve been, as Sally Milam was
saying yesterday, research data sets that have been made available to
researchers under special kinds of understandings. I’m not entirely sure—
I think it’s going to be really interesting to look at the extent to which the
demand side is moving beyond those types of data that’s interactive with type
of user. That’s going to be fascinating.
DR. SUAREZ: I am also member of the committee and the Workgroup. I have two
quick comments. One is I think we have a unique opportunity given the session
that the committee’s going to have in April to perhaps feature some of those
findings and some of those lessons learned from the hearing that will be held
by the committee in April in a session during this datapalooza. That’s one
suggestion. I think truly coming out of that meeting we’re going to have a lot
of information to share about the findings and perspectives on data collection,
use, and dissemination, and the privacy considerations.
That’s number one, the other one being the co-chair of the standards
sub-committee, it would be helpful to perhaps consider having some discussion
about what I call the meaningful standardization of information and methods and
structures and mechanisms to collect and disseminate and analyze the data.
Meaningful standardization is a new perspective. We always say standards are
good, and we continue to say they are good, but there are always questions
about to what extent and up to where standards are valuable and then become
more of a barrier in some cases. The third quick question, more a question than
a comment, for members of the workgroup, are we going to be able to attend
datapalooza not as representatives of NCVHS but really members of—this
workgroup was really created to advise Datapalooza. I’m not lobbying for a free
ride. I’m ready to register. I’m not sure my organization is going to be
present there heavily, but my suggestion really is we should have the full
membership of the workgroup attending if they are available.
MR. SCANLON: In term of the potential directions for the future, the ideas
that came up today about new technologies and new data sources, we actually are
getting started along these lines with HHS. Actually part of our data strategy
is to move out—and this is part of the group as well– to look at new
technology and new data sources. I would say that—and we’re looking at the
surveys, but other means as well. We’re not just looking at them, and we should
probably report on these, we’ve actually got pilot studies going for the
surveys and for claims data and others things to actually move them forward.
There’s a little bit of funding. I think Ed is right. NCHS doesn’t have money
for methodology. We often help ASPE and others often provide that funding to
NCHS and AHRQ and others to move things along. As I said, we’re looking at
surveys, Skype-type data collections, telephone surveys, other means, and
social media where it’s not really a quantitative analysis, it’s more
qualitative in terms of looking at the web and Google and other things in terms
of searches. We would be very open to ideas from the committee and from others
about—again subject to resources—new directions to look for.
MS. GREENBERG: Following up on what Walter was saying, this year the
registration is considerably higher than last year, federal employees could
attend for $50. Now the early bird is something like $300, and it keeps going
up. They do keep extending the early bird, but when we saw that early bird, we
did want to make sure that at least a few members of the Full Committee were
attending. I think Linda Kloss and Ob we’ve already registered. Obviously,
Justine, you’re on the planning committee. How many other people are actually
planning to attend at this point and are registered or will be attending
through their organizations? Clearly, if you’re putting on a session, we need
to certainly travel the people who are going to be on that session, and we’re
prepared to. I would imagine we’d need to pay the registration as well. We need
to know that because I’d rather pay the early bird fee than something higher
than that. When did they extend the early bird to? March 15?
DR. CARR: It’s our plan to discuss that this afternoon.
MS. GREENBERG: I don’t think we have the budget to send the entire workgroup
and register the entire workgroup, but some are already going. We would work
DR. GREEN: Speaking of this afternoon, everyone’s invited to attend. We hope
that as many members can stay for as long as you can if you don’t have other
obligations or travel. This was an important conversation. It’s sort of like
the opening meeting of the workgroup today, so I appreciate all of your
participation in that. Here’s what we’re going to do. We’re going to deputize
Debbie Jackson and Marjorie to create the work plan out for the rest of the
year in light of the deliberations we’ve heard here today. We’ll vet it with
the Executive Subcommittee at our next meeting, and you’ll see it show up in
the minutes and that sort of stuff– unless we have enough time at the end to
come back to it. Thanks very much Justine. It’s the Population Subcommittee
DR. COHEN: Susan, this half hour session is going to be broken into two
parts. The first part will be Susan giving a detailed description and
discussion of her environmental scan, and then we have 15 minutes left for
discussion and conversation that I’ll lead. Maybe everyone should get up and
stretch for a minute.
MS. KANAAN: Since I know that time is of the essence, while they’re looking
for slides, I’m going to begin. There are a few things I want to emphasize,
perhaps to correct some mistaken assumptions about what the purpose of this
scan and the nature of this scan was. This scan was not a scan of data,
suppliers, packagers, or users. It was a scan of open source resources and
supports for communities who use data.
The reason for doing that—I should add that some of the entities that I
looked at, do also package and supply secondary data, but that was not the
major focus because NCVHS is looking for the best way that it can contribute.
In order to do that, it really needs to know who else is out there and what
other people are doing. This was an attempt to identify what kinds of
intermediary resources are out there.
The scan involved telephone interviews with three major organizations:
community commons, and I talked with three people there; Jim Keddy from
California Endowment, to talk about building health communities; and Pat
Remington from County Health Rankings, and we also talked about roadmaps. I
also had email exchanges with Deborah Pontiny(?). I’m afraid I don’t know the
correct pronunciation of her name because I haven’t talked with her yet. The
ABCD institute or Asset-Based Community Development Institute at Northwestern,
does some very interesting work with communities, including at least one of the
projects in Rochester.
I also did a web-based scan of 13 programs, resources, and organizations,
all of which are open sourced. There are a number of other programs and
resources that are not open source that are in this same space. They fall into
a number of categories. There are several very broad-based national platforms
and resources. I would highlight particularly Community Commons and the County
Health Rankings, which is connected to the RWJ roadmap program. In addition,
every state does some degree of support for communities using data.
I highlighted the work in Massachusetts that Bruce is very involved with as
kind of a model. There’s a degree of support available in every state. Pat
Remington told me they typically refer the localities in their network to state
resources. There are several in that category, then there are federal programs,
which I really didn’t gather too much information about, but it’s important to
mention them. There are also caucuses, which are smaller entities but are also
trying to figure out how to help communities use local data. Then there are a
couple of other organizations that fall in other categories.
There are several important takeaways that I want to share with you. The
telephone conversations were very helpful in creating a context and really
understanding how some of these key intermediary organizations think about
data, community data use, and what they’re finding out about it, too. Some of
them, like Community Commons, are also gathering information on local data
users. It’s a resource that we should be aware of for the future.
I think a general point here that is not in the slides but that I want to
emphasize is that all of this very cursory almost impressionistic scan that I
did in a fairly limited period of time– I do want to emphasize this is not
exhaustive, but what I became aware of is a kind of embryonic infrastructure
that is not unlike or has aspects of the national infrastructure that we
envisioned in our community as a learning system report. I’ve done some
preliminary mapping to see what are the major things that are already existing
or are in formation and those that aren’t. I don’t have time to go into that
today, but I think that’s one of the important things for us to do, to do that
mapping in the future.
There is an infrastructure that is beginning to develop to support
communities and to build this connection between the supply and demand sides.
We need to collaborate with it. We need to be aware of it. We need to see where
the committee’s guidance is appropriate and the extent to which these
intermediaries like Community Commons and the County Health Rankings can be
direct recipients of the committee’s guidance, when they can be a conduit and
dissemination network to communities, and when we can collaborate and learn
Three points that I wanted to emphasize that came out in the phone
interviews are three really important themes about the “why”, how to
think about why communities need to use data and how that process happens. One
is what Jim Keddy called, data for social action, and what we’re calling
community driven change, and the really critical role of policy in
understanding what needs to happen at the community level. This is important
because of the social determinants of health and so on. It really broadens the
whole understanding of community data.
Second is something that the ABCD Institute has a major emphasis on, and
that is data on community assets as well as community problems. They’ve done
some very interesting work. They’re some of the intellectual leaders in that,
helping communities assess what their assets are. The third, which the
Community Commons is very active in, is the idea of fostering a national
movement, which is something that they’re investing a great deal in.
These are important things for us to be aware of. Some of these programs are
really at a very early stage. There’s an opportunity to collaborate with them
and to really co-create as their work evolves and as our work here evolves.
We’ve identified some gaps, which you see on this slide, hands on technical
assistance, all of the things related to stewardship, and then this big set of
technical questions having to do with the continuum from very granular local
data to the larger types of data, and combinations between health status and
social determinant data. There’s plenty of work for everybody.
I collected some preliminary data. This just gives us an idea of the kinds
of resources that are already starting to be available for communities. Don’t
put too much stock in these numbers, but you see that things are happening.
Things are becoming available. I think I probably just about used up my time
here. The priority of widening and strengthening the bridge between federal
data liberation and these existing and evolving resources was really reinforced
as I looked at what’s going on out there. This seems the obvious and natural
role for the committee. Clearly, communities do still need guidance on
stewardship. That’s one of the major findings. I’m going to stop there and turn
it over to Bruce now.
DR. COHEN: Just a couple quick comments. Why are we doing this? My bottom
line is the federal government can through the national committee play an
enormous role in this change that’s going on every day in America. Communities
now are much more proactive in making decisions that effect the health of their
residents. I think over the last 20 years, decision-making and planning has
devolved from a more centralized approach to a very decentralized approach,
from the feds to the states, states experimenting with healthcare reform, from
the states to local areas, creating the space that allows local decision making
to happen that really ends up being more effective. I don’t believe in
trickle-down economics, but I do believe in the role of government providing
resources to help stimulate community-driven change. I think that’s why we want
to be in this space. A couple of quick observations, there’s a huge gap in the
feds in terms of thinking about community assets. We’ve always operated from a
more traditional public health, disease model. We’re really good at collecting
vital statistics and doing surveys although there are limitations. We’ve very
rarely thought about what are in the expansive definition of public health,
what are the measures of well-being in communities. There are data out there.
There might be opportunities that I really want to see us focusing on what role
can the federal government play in stimulating more work on understanding and
standardizing community resources and assets.
I think this is a huge new frontier where the demand is building up. There
are local organizations that have begun this. There are some themes, but this
is a wide-open area. Secondly, we’re very proud when we can generate state
estimates, or states are very proud when they can generate county estimates.
Communities, counties are much too diverse and too big to be actionable to
improve quality of life. The next frontier for us is figuring out how to get
down to what I think the actionable level of geography is, which is
neighborhoods. People think about their neighborhoods or their towns, if their
towns are small enough. There are lots of opportunities that I see we have in
helping support, through our resources and through our initiatives, movements
that will really have an incredible impact on communities who are focusing on
improving their health, health defined broadly. Any questions for Susan? Again,
the purpose of this scan—part of it was already achieved, identifying
individuals for our workshop in April, helping focus on some of the issues and
identifying some of the organizations that have something to say in this space.
DR. FRANCIS: I think it was a great start, but I actually have a
methodological question that goes to thinking about the workshop. Sometimes
when people talk about an environmental scan, there are implications of some
sort of a methodology in determining the scope of the environment to be looked
at. There’s some discussion of where are you going to look, how are you going
to do the looking, and what—this is really by way of getting at what’s
left out? I’d love to have you tell us a little bit more about how you thought
through– I know there was no contact with any of us on the– at least there
wasn’t to my sense with me before the environmental scan got set up– of how we
were going to think through the methodology of it. Whatever you can do to help
us at this juncture to understand that, that would be fantastic.
MS. KANAAN: We did have a number of suggestions– I did have a number of
suggestions from different members of the committee, particularly members of
the population sub-committee talked very early on with Vicki, and she was
significant in helping, contributing an additional dimension. Some of the same
names continued to pop up from a number of different sources. There was a
snowball process. Whenever I talked to people I would also ask for suggestions
of other organizations and entities. The large national organizations I’m
fairly satisfied are the existing ones, particularly Community Commons and–
DR. FRANCIS: I was actually going to ask you to tell us a little bit more
about Community Commons. I thought they charge money, don’t they?
MS. KANAAN: No, it is open source. They also have something called CHNA.org,
which is the data piece. They’ve combined– this is through the University of
Missouri– there’s a summary of each of these programs. There are three
products. There are the summaries. There are some findings and observations,
and there’s a summary table. I don’t want to suggest that this was a
scientific, rigorous scan, so if the terminology is wrong then perhaps we
should change it. This is really an initial reconnaissance to see what’s out
there. I ruled out a few– I didn’t bring all my notes up here with me, but the
prevention institute– there were a few organizations I identified, but they
didn’t seem to meet the criteria of being active, of having resources that
they’re directly offering to communities and so on. I looked at maybe 20
organizations and then narrowed it down to the 13. I’m not aware of any major
organizations that were overlooked.
MR. SCANLON: I think from other areas, other disciplines, population and
economic community involvement, there are some models to use at the workshop
and other ways– that’s not necessarily the way we would go in health, but they
have had to deal with these issues already, and in terms of– census is always
very popular since they release data for very low levels of geography.
Economics has already an infrastructure in terms of economic analysis that
specifically gear towards regional and metropolitan area economics. Just as a
model, maybe we want to have those areas represented at our– what data did
they use– it’s not just the chamber of commerce or the planning agency, but
there’s a lot of folks who use that data. It’s just another model and
distribution channel to look at.
MS. KANAAN: Again we are talking about resources for communities rather than
data per se. If you look at the function columns of the summary table that I
did, we’re talking about training, resource materials, networking
opportunities. I can’t remember, there were seven or so types of functions that
we emphasized, research, facilitating research.
DR. COHEN: But I think for this scan, that was the intention, to identify
those organizations, but I think the point that Jim makes is really good. I
wouldn’t be very surprised of the issue of getting smaller area data emerges as
one of the themes. We should try to get folks who successfully operated into
that space. I think public health can learn a lot from them. We should try to
incorporate that and those folks into helping advise us.
MR. SOONTHORNSIMA: Just a quick comment to reflect on Bruce’s point. The
focus here is more community centric and very specific. Just a suggestion or
maybe consideration for the future, I think this thing will evolve after this
spring workshop. That’s what you hope to see this thing snowball at some point.
I wonder in the future whether you might consider having these venues done at
the local level in the communities. Let’s say you pick a handful of
communities, because that’s where you’re going to get real– instead of being
Washington-centric– be more community-centric. That will go well with the
themes, just a thought.
DR. COHEN: New Orleans is centrally located. We’d love to meet there, but
you’re right. When we talk about dissemination of our findings, they’re going
to be– we’re going to be targeting the Secretary and the Department at one
level, but I really think the focus here is adding value to communities. Maybe
we can take this show on the road, and we need to think of other dissemination
channels that would reach communities more effectively than the traditional
focus of reports from this committee.
DR. MAYS: I just want to first support what Ob said. I know once before
populations had a set of hearings, like in San Francisco, we were able to get a
lot of community groups, and then I forgot where the other one was where we
brought in a lot of Native Americans, then we had Philadelphia. It actually
does make a difference, and it gets around some of the cost issues in some
ways. What I wanted to say is that in terms of the scan, and it may be useful
just for our own purposes to call it something different, so as we push outside
of this group to give it a different name so that we’re all on the same page. I
think that in terms of looking at issues of community, it was really to move
from the report into what’s different.
I think it’s very important, if you listen to what Susan was saying, we’re
in a different place. Quite often, well let’s, in terms of the report, stay
with a one to one building on the report. The landscape is a little different.
I think it’s important for us to remember that much has been developed since
then and that there’s a different direction that communities are going in, in
the sense of where they are around their data issues and what they see as the
primary needs versus what may have come up at the point of the early hearing. I
think that’s really, Susan, from my perspective, what would be the most
helpful, to have you go through this and say what is different than what we saw
in the report so that in the hearing we’re making sure that we’re going in the
direction the community is asking as opposed to the things that we heard the
first time if there’s been a change to the priorities of those things. I think
Justine asked a question earlier about how to use the information that will
come in the surveys. That would be my second suggestion, to make sure that
we’re going in those directions as opposed to– I guess the hearings were maybe
two or three years ago.
Foundations have put significant money into community data. There has been a
lot of changes in terms of communities’ use of data. I just want us to make
sure that we’re going where the new issues are as opposed to building one to
one on what we heard before.
DR. FRANCIS: Just a quick comment, somebody we haven’t talked about inviting
here is some of the foundations that have been putting money– we should be
sure to have that on now.
MS. KANAAN: I think RWJF or even California Endowment or others, I agree
with that Leslie– my guess is that the workshop is going to be the context in
which we discuss the very question. I’m not sure that it’s possible to say too
much about what’s different now than two years ago. Certainly we know some of
what’s different. I think a lot of that finding will come out of the workshop
DR. MAYS: I think that the feedback information that we’re working on will
help us enormously. I think we want to be a little flexible. If I want to
understand correctly, usually after we identify the people we give them a set
of questions. We don’t want to lock the questions in and the sense of it’s
narrow. We want to give them a bit of a wider berth in terms of where it’s
going in the future.
MS. GREENBERG: I had a more specific comment, but Larry also asked me to
help wrap this up. I think building on what you just said, Vicki, it was two
years ago that we had the hearings. I still do see them as a starting point.
Part of what we saw then was that many of these communities were just getting
started. Having a more longitudinal look as to what’s happened– their progress
is unlikely to be linear given that life is not linear and certainly in a
complicated area like this.
I see this as kind of the next point in time for the committee, although
you’ll have to decide on your own, you’ll have to decide this on your own, but
I see this continuing for a number of years just as it started. I think Larry
very correctly is product oriented. We’re not just going to talk about this for
the next five years, are we? Let’s see what intermediate products we’re going
to have, particularly what the needs seem to be for either recommendations or
privacy has indicated, models, or that type of thing. It will be a point in
time and then lead you to the next stage.
My sense is that– it may prove wrong– but the report and what you heard a
few years ago and then more recently through the stewardship thing, is going to
still be relevant. Obviously, if we were just sitting there, then acting on
that would be a problem. Instead, you’re going forward with your next level of
information gathering. I think that some balance between having some kind of
short-term goals but recognizing this is a process that will continue, and
hopefully resources will remain available on all fronts to continue it, but a
more specific point that came up about the small area data.
I know we had some discussion in the teleconferences, at least of the
Population Health– and it’s very relevant to the Privacy, too, but I don’t
know that we’ve continued it and thought about it both from the point of view
of who’s going to participate and the people you’re reaching out to for this
additional feedback. That is this whole long standing area which went into
negative, with less well-received in recent years, but I think going back to
the 70s and maybe before, of small area estimates and estimates that
supplement, at least, the data you can directly collect, because of the
problems with small numbers, problems with confidentiality, et cetera. I do
know that our office of research and methodology has reinvigorated that small
area data estimation process. Is that correct Ed? They’re doing research.
They’re very interested in that.
I think there are two things that it’s important for the committee to look
at. One is what is going on in that area, and it tends to be pretty
methodologically dense, so I would start with the National Center of Health
Statistics. Others are– and certainly at the state level you may be working on
that, too, but the other is do the communities want this data? Would they use
it? No matter how sophisticated it is, and even if it’s better than their
direct data collection, do they want it? How would they use it? I think that’s
very important. It’s going to be necessary. We can’t collect all the
information that they need or they want, and particularly to do comparative
stuff. I hope we can work that in some way, one question or either in the
feedback tool or in the discussion itself.
DR. COHEN: I think it would certainly emerge in the small group discussions.
We can have some of the speakers talk about the pros and cons, particularly
community folks. I’ve sensed somewhat of a change in attitude. Obviously
collecting the data directly is preferred, and at the same time if you want to
know the smoking rating from your town or your neighborhood, you might settle
for some synthetic estimate.
DR. CORNELIUS: As it turns out I am sure this April activity will be a
success beyond our wildest dreams. Part of that success may be– we’re talking
about a bunch of themes at the same time. While we talk about community as a
learning system, should we really succeed it’s a two way street. We’re reaching
out to the community, and they’re going to reach out to us.
First a comment about a task, thinking not just about the April activity but
opening up the door to what comes next. I wrote two words, one on the issue of
auditing, and the other phrase, transparency, as it relates to either a subset
of community stakeholders or whatever mechanism you create. The issue we run
into is we have these activities in April. We stimulate folks and create these
ideas, yet when we engage in communities it goes back to these situations of
trust. Part of building the trust is the feedback mechanism. They want to know
that now that we did all this, how do their ideas reflect some kind of
semblance of what the community participants feel is coming out of that?
The whole point about saying that it’s going to change our dynamic; it’s
both a product and a process. We’ve been nailing down the products. How do we
do this? How do we do that? Really, as we’re moving forward what we’re saying
to communities that we’re building relationships between the communities and
the federal, statistical system. It’s also going to be, like as Amanda was
saying, that it’s going to be a paradigm shift. We may, of all of us here, may
end up thinking about this differently when we talk in June about it than we
are here today. We have to allow ourselves to be open to the fact that this is
going to be a moving process
DR. GREEN: I want to thank you, Susan for another great piece of work. It
obviously stimulated a terrific discussion. It seems to me the workshop has
begun. There’s a certain irrepressibility about it. We’re on to something here,
what Ed said earlier in the morning, NCHS needs most from this committee is
guidance on priorities and focus of the coming issues. We’re on the center.
Thank you very much. We will recommence very promptly at 11:00. Dr. Queen will
be ready at that point and this is going to be another very interesting
DR. GREEN: Dr. Queen let’s get going about the health system measurement
DR. QUEEN: Good morning, I am here to introduce to you and discuss with you
the health system measurement project, which was designed as a web-based tool
by the department to display data on key health indicators compiled from across
a variety of federal sources, along with descriptions of all the measures, and
links to the data sources. It was developed to bring together in one place a
very broad array of data and measures on these key health topical areas.
At the same time that the project was designed to bring together this broad
array of measures and data, the aim was also to limit the scope of the site so
that it would be both manageable and easy to use and transparent, and not have
too much in one place. It’s an ideal starting point for a very high level
overview of the U.S. healthcare system.
It focuses on these ten critical dimensions of these ten critical dimensions
of the healthcare system that were determined by an internal workgroup,
including areas, and we’ll get to all of them later, but availability, cost of
care, prevention, et cetera. Within these ten dimensions the site has
information on about 50 measures. I think we actually only have 44 at the
moment, but we have the capacity to have up to 50 measures from HHS and other
federal agencies. All of the measures are standardized and have been assessed
for reliability and validity.
Multiple years of data are provided. Most of the measures have ten years of
data. Some have four or fewer, but most of them have ten. Data are also
uploaded annually, and in some cases more frequently, but most of it’s annual.
Future years will be added. The visualizations are easy and depending on the
measure, users can choose from among various forms to display the data,
including aside from graphs and tables where state data are available, you can
have a map to visualize the data and then have additional data on that state.
The background of the system, it’s been several years in progress. Although
it was launched last year, May 15, 2012, there was a measurement and evaluation
workgroup made up of experts across the department that numerous subgroups with
broad representation from the various data collection agencies and programs.
They were to select measures that would provide a broad overview of the system,
focus on these key areas that they determined were the most ten critical areas,
have established reliability and validity and would also align with other HHS
Things like the HHS strategic plan, the national quality strategy– you’ll
see references on the site if you use the site to HealthyPeople. Aside from the
major key areas that were determined, the workgroup also wanted to include some
measures in areas that are not often included on these types of website, such
as oral health, service use, substance use, screening, follow up, innovation in
the pharmaceutical and medical manufacturing sectors.
The data sources, these actually are the agency data sources; this is the
list of the federal agencies that were providing systems for the site. There’s
also some limited information from a couple of non-federal sources such as the
American Hospital Association annual survey, their IT supplement, and I think
there’s some from Truven Health Analytics, which I think used to be Thomas
Reuter’s market scan. The range of data systems just from within the federal
government is considerable.
For those of you familiar or not familiar, you’ve got the National Health
Interview Survey, the National Health and Nutrition Examination Survey, AHRQ’s
Medical Expenditure Panel Survey, and their healthcare utilization project. You
also have some less frequently– the big payers, like the National Ambulatory
and Medical Care Survey, CMS’s minimum data set, vital records, and some data
from FDA’s Center for Drug Evaluation Research, also data from the Bureau of
Economic Analysis and the Census Bureau’s Current Population Survey, the annual
social and economic supplement. It’s an impressive assembly of information and
The data are supplied by the agencies themselves. The programs have
identified what they called a data owner. The data owners actually submit the
information into the system. Nothing that’s submitted is raw survey data or raw
numbers. It’s information that’s been run or executed in some way applicable to
the survey instrument from which the data are coming.
Prior to submitting the data, and this is an important feature, there’s an
assurance that all appropriate privacy and confidentiality requirements have
been followed and met, along with the information quality guidelines that the
federal government has to comply with when disseminating data to the public. I
think for users, there’s a certain assurance that when you’re using this site,
you are accessing data that has met some of the highest standards for quality
I’m not doing a national demo of the site because one bug is that it’s
really slow on Internet Explorer, so we wouldn’t get through the screens, so
I’ve done screenshots. This is the home screen. It gives a nice description and
a couple of sentences as to what the site’s all about. It provides the topical
areas. They’re listed there in the center for users. Across the top of the
screen, you have topical areas, measures, help, and about. It’s fairly
straightforward, not too many bells and whistles, but it does have a lot of
data in it.
I would click to get the ten topical areas, and I realized this morning that
when I did the screen shot I cut off two of the topical areas, but here are
eight of the topical areas. Those areas are important, but you can see each one
has a nice description of what the topical area is. A user would just click on
whatever area they’re interested in now. For today there isn’t time to go
through all of them, so I just thought I would take access to care and look at
a couple of the measures that are available under access to care.
This isn’t probably readable, but for each of the topical areas, when the
user clicks on it, they get a description of what that topical area is. They
get a list on the left of all the charts that are immediately generated or run
in real time for that topical area. You’ve got the usual sorts of care on the
left, and you can see there’s usual sorts of care, specialty care, delays in
access to care. There’s a whole set of slides or graphs that are immediately
displayed. You can see it says “see this measure in detail”. You can
get more information about that particular measure.
I wanted to look at difficulty seeing a specialist. You can hover over with
your cursor, hover over the actual number that’s being displayed in the bar.
You can select among whether you want to see this as a table or as a graph. You
can also click on additional information for see this measure in detail, which
then gives you this whole header which has the data source listed, and it has
the metric calculations, so it will describe the numerator and the denominator
of the data. On the right hand side the user will see what other topical areas
this exact same measure is going to be listed under, and where it says,
“see this measure by” you have a list of all the charts that are
available to be generated for that measure. All this is in the cloud, so when
you run it, the charts immediately appear. The user just has to toggle down the
screen to see additional information.
DR. FRANCIS: Can you see it by a whole bunch together? Could you do
Medicaid, race, age, disability status, et cetera, all together?
DR. QUEEN: You can go into– users can export the data. This site won’t do
that for you. You can export the data into Excel spreadsheets and other
different files. The extent to which you can manipulate it in that way is going
to be hindered by the fact that the data have been generated for the percentage
of people who reported difficulty seeing a specialist by a race– you can’t
rerun the data because you’re not getting back to the raw data.
If I wanted to look at the data source, I click on medical expenditure panel
survey. It takes you not to the agency, but it takes you directly to the survey
or instrument site so that you can either get additional– you can download the
data from here, you can get their publications, their data documentation, et
cetera, whatever tools that AHRQ has available, which they do to query the data
further in depth. Their public use file for the MEPS, a user if they’re
interested in seeing this, they can go into a query tool that MEPS provides
that will allow them to do that.
DR. COHEN: And the level of geography?
DR. QUEEN: It depends on the measure. When they go into see this measure in
detail, that information that was listed on the right of the screen for this
one– percentage of people who reported difficulty seeing a specialist, MEPS
doesn’t have the level of geography to permit state data, but you’ve got
The site also will inform you– here is the same measure broken out by
coverage. You’ll see there’s a red line. The red line is to alert the user that
something’s different for this year. You can take the cursor and move it over
that red line, and it will explain what the break in the data– why it’s there.
For some years, we have data that could not be collected. In this case, the
survey instrument changed. The question that was being used to ask that
particular measure changed. It resulted in a change in the numbers. Users have
to be aware, is it a real change in the actual number of people have difficulty
seeing a specialist? They’d have to look at that, consider that.
MS. KLOSS: Are the guidelines that you referred to for data quality and
privacy, are they on the site? Can you look at that by type of data?
DR. QUEEN: They are not on the site, but we’re looking forward at
improvements and enhancements to the site. I think we could make this more
informative for users even about what we provide right now as to the data
sources. There are bugs that are being worked out and things that we want to
add to it. The information quality guidelines is literally a link in HHS that
will give users a lot of detail. It’s not the privacy side of things, but each
agency has a whole set of guidelines and procedures that they follow. Some of
them are very similar if you look at what NCHS does in terms of its protection
prior to releasing data to the public. In SAMHSA, the major surveys have very
similar procedures that they follow either with disclosure review boards or
MS. KLOSS: I ask people I think there is such a dearth of those kinds of
guidelines, particularly on data quality. It may be something that we could
look at as something that could be turned into a very useful tool.
DR. QUEEN: It would be overwhelming, I think, for the users of this site to
take them to some of the detailed information on what’s done, but I think we
could generate something that gives them a more easy to understand overview of
how the data are protected.
For each chart, you see how it says on the right “the coverage type
is” and there are checkboxes. Users can select or de-select. They can
filter the data by year, et cetera. They can choose to– next to the title, you
can choose to display it right there either as a table or as a graph. Then you
have options to either download the data or explore the data. If users select
to explore the data, they’re going to see a display like this– I actually
don’t think it’s too helpful to have the measure value bar on the left at this
point, but you can see the percentage that’s there, the calendar year, the
coverage types, so the users can manipulate these things if they want to
display it differently.
What’s more interesting for each measure is across the right and the top
there are a set of boxes that– there are options for users. There’s managing
the data, which can deselect columns and rearrange columns. There’s filtering
so you can group rows of data together. You can sort one or more columns.
There’s actually a short tutorial video that appears on this site that will
help users figure out how to do this. Visualize is another box that allows the
user to change the display of the data either in terms of a line or a bar graph
or a map if the state data are available.
There’s export, which allows people to export as either an XML or PDF or
CDSV, and Supprata (15:38?) which is the contractor for this site also has an
open data API which allows software developers to access data that’s all hosted
at other supprata sites. They can create applications that use these APIs to
visualize and mash up the data from here with other data sets. It’s possible to
develop an iPhone application or a web application. The sites points those who
are interested in this to Supprata’s developer documentation, which I haven’t
looked at, but maybe the working group can help us with.
About, there’s a nice box called “about” for the measure, and for
those of us who are trying to look at the site and its utility and usefulness
and has been using it, “about” tells you for that particular measure
how many users have accessed it and how many have downloaded it. That’s a nice
I wanted to just give another example under access to care that has some
different options from the previous ones. This is percentage of people who did
not receive or delay care. If you click on the data source here, you’re going
to go straight to the National Center for Health Statistics’ HIS survey. If you
select the box that says “click here for additional information” it
will show again the complete set of charts that were listed on the right on the
I wanted to show an example from the glossary, percentage of people who
delayed care due to cost by ethnicity. In this example, the user, because
certain measures have OMB standard characteristics, the glossary holds them
to– you can click on the glossary and you can see the standardized measure
that was determined for OMB for metropolitan area, for race/ethnicity, in this
case, and the glossary will then link to the user, if they want to go to have
more detailed information on the OMB federal standards, it will give them that.
A map of the data, in this case, the HIS for a number of states had enough
information to display the data according to state. If you click on the
individual states, click on a data to see data from that state, and it will
show you a graph for that state. You can select the number of years you want.
This is a measure list for all measures that are in this system. We are now in
the process of looking at how to improve the HSMP. Some of the names that are
described here, you don’t find what you think you’re going to find. We’re going
to be making some modifications to the site to make it more user friendly and
more easily understood.
Next steps, we just recently conducted some usability testing here in HHS on
the site for functionality. I realize it sounds a little odd. It was launched
in March of 2012 and here we’re doing usability testing now, but it was quite
informative. We come up with some, from the people that participated– it
helped to identify some easy fixes that we think will enhance the site going
forward that we can do now.
We’re going to be doing some focus group testing next week and over the
coming weeks with different sets of users, identifying audiences who use the
site. It’s not a site for academic researchers or people who are going to want
to do logistic regressions, et cetera.
They can help us all identify how we can make the site more useful for the
audiences that it was designed to serve. We’re looking at potential new
measures and sources of data. We’re trying to keep to that limit of no more
than 50, but there may be the possibility of taking some of them, archiving
ones that may not seem to be as relevant.
I’m also very interested in working with the Work Group. This site has been
one site, separate from the Health Indicators Warehouse, there are a number of
sites, dashboards that do these kinds of things. This one I think is rather
unusual in that it has data from such a variety of sources, more than two or
three. It’s a large number of sources, mostly from the federal government.
I’d like to know how does this site fit in or relate to the other sites.
This site doesn’t link to the other sites. I don’t know if other sites could or
should link to it. We’re definitely working with the Health Indicators
Warehouse; because I see very different roles that these two sites play, and
how do they compliment each other versus compete with each other.
Someone mentioned yesterday the ONC Dashboard. CMS has yet another
dashboard. Would it be possible for us at this site to say if you want more on
this measure, you might want to consider looking at the –.
And then there are external sites like SHADAC, their data assistance center
has state data, and they use the American Community Survey, and the CPS survey.
Then RWJ has their data hub, and they’re using different sites. They’re using
the YRBS, and we may be using the VRFSS. They’re also using data from USDA,
which we aren’t. How do we make this complement and fit with all these other
systems? That’s the thing I’m hoping the data access and use workgroup can help
us with, and also making it more known. If you want to find this site and you
don’t type in health system measurement project, I don’t know how you’re going
to find it.
It’s a really handy site. For people in state, local, or federal governments
that need to have data quickly for a presentation that they want to make, and
anyone that’s worried about having data that are clear or approved or knowing
it’s valid, it’s excellent. Making it more known is something we could use your
help with as well.
DR. COHEN: It is a very pretty, useful data source. I really like it. You
brought up an issue that I think would be a really nice topic for the workshop
as well. We’ve got this data site. We’ve got a proliferation of data sites,
some of which have information that cover the same domains, which may or may
not be the same. I’m sitting in my community meeting and wondering what is the
estimate for insurance coverage in my state or whatever the measure happens to
be. How do I evaluate these sites, and how do I compare them, how do I make a
decision what to use? It’s almost like we’ve thrown the tools out there too
quickly without any guidance about how you use this for that, how these
compliment each other and what the differences are. I think we really need
collectively to do more work in making it easier to decide how to use all these
DR. GREEN: Other questions or comments?
DR. CHANDERRAJ: Susan, two questions. One is a kind of a comment towards a
question to. As regards the cost that prevents access to care, I find that I
see a few patients every clinic day in a free county hospital– people don’t
necessarily see a doctor because of cost, because they don’t want to be
bothered with their health. They’re in kind of a death wish mode saying what’s
the doctor going to do after all? He’s going to prescribe medications and
tests, and that’s why they don’t want to access care. The second question was
most of the data that you collect are from surveys that are collected by
computer methods. Most of the people, the poor people don’t even have a
computer access to data. Are we missing a lot of these people in trying to
collect the data for access?
DR. QUEEN: Let me answer the second question first, which is the surveys
employ a variety of methods. The CPS is a telephone survey, but it also has a
follow-up interview. They’ll follow up. The HIS survey, you have interviewers
going out into people’s homes. Even though they’re using computer methods,
they’re bringing a computer with them; they are meeting people face to face.
The NHANES survey is again, you’re going out to people’s homes, they’re doing a
representative sample. They do extraordinary efforts to try to obtain high
response rates from the home exam. Then they’re managing to recruit these same
people to come into the trailers for an almost four hour exam to do all these
physical objective tests. The MEPS survey is a personal interview as well.
The attempt to get information, the aim is to reach people in neighborhoods
across the US that are representative of the population. Homeless people are
not going to be– people that are migratory and are homeless are certainly not
going to be represented in these samples, but lower income, neighborhoods that
are poor, there will be people represented in these surveys. As to the question
on the cost, that is a particular survey question. Along with your delay in
getting care, the starter question is something like did you in the past 12
months delay getting care? Then they’ll have a series of options that are asked
of the respondents. In this case, cost was one of them, but there are a number
of other options for the respondent to select from.
MR. SOONTHORNSIMA: A couple of points, one I think to your point earlier,
two points you made, I think one point you made said how do we tie it all
together, because you have so many different agencies and you alluded to that
earlier on, where you get the data from for the surveys. One thing I observed
right from the get-go is I wonder if there’s a way to have an über site
that ties them all together.
It’s difficult, to your point. Every agency has its own set of data and
sites and surveys. That’s one thing, to make it a little bit easier to
navigate. It’s probably not panacea but enough to begin to have people and
perhaps tie it, wiki, some sort of self-navigate– help people navigate across
all these sites so you don’t have to tie them all together, physically or
logically. You can tie them all together through links and navigationally.
That’s one thought.
The second one, as I went through this, wanting to make this data even more
useful is trying to understand who the audience might be. As I go through this
I start thinking a lot of people would be using this, policy-makers for sure at
the state level, insurers, even providers, but what about consumers?
Understanding about what the potential audience might be and then ask the
question right out of the gate, why is this important? Unless you know what
you’re looking for– I started going to the different areas, for example, of
measures of insurance cost. That’s where I spend my time. There might be a
different way to navigate to all these different statistics that you have, the
reports that you have based on the audience, just a thought.
DR. QUEEN: One of the parts of the site that I think we need to work on has
“data for you”, and it has these different audiences. I am not sure
that those are the audiences or that the measures that are listed under the
audiences are the right ones. It’s something we want to look at.
MR. SOONTHORNSIMA: Start with that, and you can move on. Some could overlap
DR. BLEWETT: It is so visually appealing and easy to navigate. I was doing
the same thing, going through this. My group actually populated the RWHA data
hub. We use a lot of the same sources, but one of our criteria was it had to be
state level. Here, of course, I’m looking for the state. You have a lot of
national data that doesn’t drill down. If you could navigate in through what
state data do you have, like I’m a state official, I would want to know the
indicators that you have for the state, rather than having to go and click and
see each time whether there’s– I did the workforce one, wondering if they had
state workforce, which would be great, but you don’t.
DR. QUEEN: That’s something that we got out of the usability testing.
MR. CROWLEY: One framework to think about when trying to reach out to your
audience and trying to connect with them is what e-commerce sites do. When
people go and are trying to access a product and assume data is the product for
this audience we’re trying to reach, they embed a number of tools, which say
understanding your needs, people with similar needs also found value in this
site, in this data set, and this data set. As we start looking across the
different assets in HHS, you may say these are the ones for your audience, for
what you’re interested in. Other people are also interested in these sites and
found value for that and have a way to serve that up, in an easily digestible
MR. SCANLON: This was intended as a dashboard for health reform. Everyone
agreed we should not use the word “reform” anymore, so health systems
change and measures. Otherwise it’s a lightning rod for everything bad or good.
The process was very interesting. We had senior level folks and the data
producers across HHS on our measurement and evaluation group. The first thing
was to decide what are the counted areas that you would want to have on a
dashboard for health reform?
That’s what you see here, not that there couldn’t be others, but if these
are the things you would expect health reform to improve, to move the needle up
or to move the needle down, if it was cost, for example, these were the ten
areas. Within each, you could have had– like HealthyPeople, you could have had
600 objectives or measures, but the idea here was to pick five to ten in each
of these areas. It wasn’t meant to tell the whole story, but if you were
driving, like it’s analogy to a dashboard, you wanted to get a quick look. Is
health insurance coverage increasing? Is it not increasing? Is it access to a
care specialist, primary care– is the workforce getting better or not? Are
these things moving in the right direction? It was not to answer the question
where you would want more information. That’s where it would lead you to other
data sources to find more detail.
It turns out that, right now, it’s basically a baseline measure, other than
some health reform changes that have already taken place. Beginning in October
when health insurance coverage expansions and Medicaid expansions take place,
then we would hope to see changes in a positive direction and how fast they
might take place as well. This was meant to be a fairly– the method here was
expert judgment on what the content would be. There’s a lot more data, as you
can imagine, to back this up, but the idea was a dashboard. I know dashboards
have fallen out of favor, understandably, but this was meant to be– this is
not a new site in the sense that when NCHS releases or Census releases the new
insurance estimates. This will not be the first place they appear. There’s a
whole set of communications efforts that go along with that. They’ll be updated
at six-month intervals. Here’s where you can go to see the whole picture.
DR. CHANDERRAJ: Can one measure also be included in patients’ access to the
shopping mall clinics, and of the access of patients to these clinics, how many
of them are going there versus seeing an actual physician provider?
DR. QUEEN: The trick would be we would have to have a survey that has a
distinction between a shopping mall clinic versus– a number of them will ask
for usual source of care, and they’ll have clinic, community health center, and
maybe as we move forward the shopping mall clinic might be something that’s
added as an alternative. We’re dependent upon what the surveys capture.
DR. COHEN: Some states actually have added to their BRFSS question about
where you receive care, minute clinics or walk-in clinics in drug stores, but I
don’t think there’s any ubiquitously available data at any level that
identifies emerging source of care. It’s something certainly to consider as we
MR. SCANLON: If there’s a way to– what the HIS asks now is “was it a
clinic?” It could be any kind of a clinic. We don’t get the name or
anything like that, but we might be able to get a couple of sub categories. We
should look at that for the future. MEPS actually goes back to the doctors or
the clinic. There we would know, but it’s a much smaller sample. We would know
exactly what it was and what services were provided.
DR. GREEN: Susan, thank you.
DR. SONDIK: I just want to give a quick kudos to ASPE, who pulled all this
together. Maybe you said this in the beginning, but pulled all this together
from multiple sub-agencies here that have all of this, and it went through a
lot of good thought about how to display and definitions and all of that, and
the glossary business, which I’m glad you highlighted. I actually think that
aspect of the sources can be, in a way, the most valuable. This is where people
can go in, and they can really start to learn something and plumb the depths of
the data sources.
DR. QUEEN: We actually want to add to that. Right now the data source takes
you to the site of the data. We want to add some more on the site itself about
each of the data sources and add to the glossary.
DR. COHEN: The Health US Glossary and Appendix is my favorite source of
information. That’s the model for anybody who’s looking for information.
DR. SONDIK: I was going to say that. I felt, too, self-serving. It is really
terrific. It’s this volume that is this thick. Those pages are terrific.
MR. SCANLON: In term of the gateway, which we’re all alluding to as a place
where people could look for whatever data we have and take you there; we’ve
tried different models. To be honest, others have tried different models. It’s
hard to do better than Google, to be honest. We actually expired our gateway
data finder because it couldn’t really do better than Google. I think what we
would like is a mediated expert-based display system, which is a little more
than Google for the data things. If any of you have ideas about models, it’s
hard to do better than some of the commercials. They have recency, they have
relevance. It’s a little harder, but a little more expert mediated display
capacity I think would help a bit, but we just retired our gateway because we
couldn’t do better than Google.
MR. DAVENHALL: A comment I wanted to make. I always tell people you want to
make sure you’re solving the right problem. Doctors know this. You do some
diagnoses first. This is a marketing problem you’re talking about. Most of what
Susan has talked about– this is excellent material. People just can’t find it.
In business, we do things like buy ad words. I just did “health
utilization data” for the US, and you have to go down seven listings
before you hit upon– it’s not Google. It’s just how it’s coming up and being
hit on. Before that, there are three or four commercial companies, somebody
called Tetrad and the Census Bureau comes before healthdata.gov. This is an ad
word problem, a marketing issue that’s saying there’s a way you can fix this
where you can actually begin to push to the top of your searches. The way to do
this successfully is go home and put in your favorite healthcare terms, and
then just log on. This would be useful to provide that data back to the
committee. Here are the terms I put in, and here’s how far I had to go down in
my listing before I started to see what I was looking for. Some of these are
Adobe files. You know you’re not going to get data if you get those.
Part of it I would say we struggle with this thing of being all dressed up
and no place to go. It’s a marketing issue. As the committee, I would like to
encourage my fellow committee people to help me out on this. I don’t want to
MS. BRADLEY: I would say that we’ve gone back around to we want to, and
leveraging Google, figure out why our results aren’t coming up and then work on
the search engine optimization. One of the biggest things we can actually do is
link across– we need to cross-reference each other because they do recognize–
people recognize government websites as high authority.
DR. STEAD: I’ve talked to the people at MedlinePlus. They live in this space
every day at the National Library of Medicine.
DR. GREEN: Susan, again thank you, again thanks to ASPE. We are only one
hour and 17 minutes away from the next most exciting meeting of the workgroup
ever. We need to get out of here on time to let people have an hour for lunch
or so. As I’ve said before, we’re going to work with Marjorie and Debbie to
summarize the work plan for a big chunk of the rest of 2013. We’ll get that
organized a little better with the Executive Committee and get it back in to
your hands. We’re not going to do that right now. We’re not going to summarize
However, as I have warned you for the last two meetings, the way we’re going
to end up is going to run the table and each of you take a moment to think
about one or two sentences that you want to say about this meeting, something
that has just so annoyed you that you have to get it off your chest to
something that you found so exciting that you can’t believe you’re still
sitting here, you should be back home working on it, to suggestions, reactions
to something you heard, something you just realized that has really grabbed
you, has gotten your attention, anything you’re particularly concerned about.
It’s open mic time for each of you. Before we start this, I want to remind
everyone that we are persuading the lady to my right regardless of her
retirement plans, we are going to do whatever’s necessary to have an
opportunity to be with her again at the next meeting. No matter how much she
resists or what she says, we’re going to do something celebratory around
Marjorie Greenberg. This is not her last opportunity to annoy you or say
MS. GREENBERG: I am a little overwhelmed, myself, after these two and a half
days of meetings, and I haven’t even gone to the best working group meeting
yet. I want to thank you all for being so engaged, as you always are. I
remember the old days when there was an unwritten protocol that the new members
didn’t say anything until at least the second meeting, and I’m sure glad we got
rid of that. It wasn’t written down anywhere, but that seemed to be the culture
in the old days before Twitter and tweet and people saying everything all the
I’m thrilled with your new membership and joining us you’ve already brought
new energy and ideas to the committee, so that’s great. Some of you don’t want
to think about my retirement; I don’t want to think about Paul’s retirement or
people who are going off the committee. I thought it was really good. It worked
out that we didn’t have any action items at this meeting. We really had a lot
of conversation among the committee members. I want you to know nobody takes
this more seriously than Debbie Jackson, sometimes she says she’s channeling
me, but I’m channeling her. She’s very sensitive to setting up the agenda to
make it as productive as possible, but also to make sure that you really have
the time to process things.
Walter and I were talking about how the culture of this committee, and he’s
been on a lot of other committees, too, than a lot of the other ones. We kind
of let it hang out to some degree. We don’t package it all up behind the scenes
and then present. We’re open, very open. I’m not saying that others don’t
function appropriately, but that’s kind of the culture here. We do discuss
things; we raise problems; we debate things, et cetera. I thought that this was
very good. I hope it was helpful to the new members to get a really good idea
of what’s on the agenda and what we’re thinking about. I appreciated that, and
DR. SUAREZ: I think this was an extremely important meeting across the board
of the committee. I think first having all the new members, I agree, I think
it’s been terrific to bring them all to where we are and see how much the train
is moving at such a high speed and try to make sure that everyone catches up
with all the things that are going on.
I think the most significant message for me of the entire meeting has been
that we are at a time where we can seize the moment for a transformative series
of activities that we will be doing. When we started talking about basic things
like attachments, they’re not as significant in terms of the word itself as
they are in the context of all the things that are possible to be transformed.
Yesterday afternoon we heard about smart cards, and we’re hearing again
about the importance of a unique national health identifier. All these very
significant and transformative elements of our healthcare system, we as a
national committee more so than most of the other committees and other groups
have the opportunity to seize the moment and charge on with these
transformative initiatives. That’s the charge that I see as important.
MR. SOONTHORNSIMA: Ditto to both comments. I think ever since I have been
part of the committee, almost two years now, I’ve seen increasing alignment not
only among the sub-committees but also in the industry and experts that come to
the table. The past three days, we see further alignment. Furthermore, there’s
been a lot of depth and breadth in our discussion. It’s been quite a learning
experience, but also it’s a challenge for us how to harmonize all these points,
and create that more strategic view, as Walter suggested. There’s a lot to talk
about on the things that we need to improve. To paraphrase Marjorie’s point
yesterday, people, stick to the breaks.
MR. BURKE: I also thought this was a rather strong session. I was
particularly taken with seeing the early results from two committees working
together more closely, population and privacy. I’m very excited about the
upcoming workshop. Something Lee said earlier today struck me as particularly
significant, which is that as we get out there in this workshop and hear from
the people we’re trying to help, we’re going to be getting a lot of requests
coming back. We need to be ready to respond.
As we ask people to tell us what they need, we have an obligation to be
prepared to respond with something other than “thanks very much”. I’m
looking forward to the next chapter in that process.
DR. SCANLON: I actually found this meeting different than all the others
that I’ve attended, in the sense that it really did start to open the door to
taking advantage of things that were happening in a very positive way. While
we’ve hinted at that before, what we had at this meeting were both reports of
activities within the department that have really made progress in these areas,
and then our discussions got into a broader and higher level. The real issue
and troubling issue is how do we best take advantage of this. There’s been the
phrase drinking from a fire hose, we need a roadmap, and those kinds of things.
Those are nice phrases, but how do you essentially operationalize it?
Now, I’m going to echo what I’ve said other times in the past. We have a
certain bandwidth that we can deal with. When we try to exceed that we are
actually not being helpful to the department. I think of this a bit like a
triage situation. It’s not that people showing up in the ED do not need help
and can’t be helped, it’s just that sometimes there are other people that need
it more in the immediate term. That’s what we need to be thinking about, and
that takes discipline, to ignore the people who are over there in pain and
moaning while we deal with something else. We need to do that. I have to say
I’d rather go with Marjorie’s phrase yesterday about the breaks than her phrase
today about letting it all hang out. We’ve got to hold some of it in so we can
focus on the task we need to most. That was my takeaway.
DR. CORNELIUS: I was particularly blown away by a quote from Marjorie. I
wrote it down. Life is what happens when you make other plans. I’m going to
carry that with me and your energy. I think that’s a charge to us to be
flexible. It’s nice to be structural and analytical, yet we have to be fluid.
DR. BLEWETT: I want to thank everybody for being so welcoming. It’s really a
great committee, and I’m really honored to be on it. I have one more
subcommittee to go to before I decide which one I want.
DR. BLEWETT: I was a little bit overwhelmed yesterday. Today I had a little
more handle on what was being talked about. It’s amazing, the breadth of this
committee. I think Bill Scanlon’s idea about setting priorities, it seems like
everything is important so I don’t know how that’s going to happen, but I’ve
learned a lot so thank you again.
DR. SONDIK: I thought that the discussion around the datapalooza, not of it
per se but around it, and trying to balance needs, I think the supply/demand
framework is really important. It’s clear that it’s not all in supply and it’s
not all in demand. It’s in some weaving of the two together with a lot of art
and innovation. I would love to crank into that the idea of– I hate to use the
word “evaluation”, but somehow tracking what it is we’re getting for
this. What difference it actually makes, I think is extremely important and
crank that into this loop so we have a better idea. It was exemplified, in a
way, by Susan’s presentation of the dashboard and the role that something like
that actually plays. I think it’s very complex. It’s not a simple answer by any
DR. CARR: I think I always say this at every meeting, but it is really an
honor and a pleasure to be part of this esteemed group. The brilliance and the
collegiality and the light of being part of this is just wonderful. That’s
number one. I have two things to say. One actually is a suggestion, Jim. As I
also look around the room and think about the number of hours away from our day
jobs, I think it would be nice if the Secretary sent a letter to our employers
to acknowledge the amount of volunteer service that we give. I don’t know if
everybody would– it’s an opt-in or opt-out, but it’s true.
I could speak for my boss who’s incredibly generous with my time. I actually
think that it would be hugely impressive. Some folks struggle with being able
to get time away. If their boss understood that the Secretary counted on this
committee in the way that we’ve heard today, I think it would go a long way.
The second thing is I agree we really were able to enjoy the richness of
ideas today. I think we need work on structure and process around our new way
of working. The subcommittees, we had a new way of working, but now we have two
of the old subcommittees so we don’t have that structure. While we have a
plethora of brilliant ideas, we need to state where we’re going to get to and
really evaluate each thing that we’re saying and doing to say is this going to
get us there or is this going to distract us? We can’t get everywhere, to
Bill’s point. We have to triage. I think if we begin to think about having that
discipline, of here’s my endgame and these are the steps, and when I take this
step it brings me this much closer.
MR. CROWLEY: To echo what Justine said, as a member of the working group on
access and use and not a member of the full committee, but being a part of this
group really is an honor and a pleasure. It’s amazing the depth of expertise.
You can see how these different ideas and concepts come together in this group.
It’s something that Larry Green had said to me last night; these ideas do have
the ability to diffuse to hundreds of millions of people to help this nation. I
think it’s something that’s useful to keep in mind.
You can see from the existing resources and assets that have been stood up
around data throughout the HHS and the wider community, it’s an amazingly
valuable asset, but you also see that there are best practices we can take
throughout other industries, the internet, economy, and others, there are ways
to leverage that to really accelerate the dissemination and use and create even
more value from that. I’m really excited and looking forward to working with
everyone here to help make that a reality, also understanding, too, how the
ideas and concepts for that can be translated into action and happen.
DR. CARR: One other suggestion, at the dinners bring cash.
DR. NICHOLS: I would say first it’s really nice to no longer be the newest
member of the committee. What I’m impressed with is how we got better when
these people came on board. I can’t wait to see who replaces me. It will be
impressive then even more. I would say it is wonderful to have this energy and
intelligence in the room.
I do think while we all would agree that drinking from a fire hose is
probably the right way of describing a lot of what went on, especially
yesterday morning, I think we might need a better metaphor. In the spirit of
the academy awards, I’d like you to remember the theme song of the Sound of
Music, “The Hills are Alive”. That really describes in my view health
data policy, health reform implementation. There are so many things going on
not always in tune, but it is, nevertheless, alive. It is a great time to be in
this space and a privilege to be in this room.
Every time it’s on the calendar and I block it off, try to, and I get to two
days before and say I don’t have time, I’m overwhelmed, but I’m so glad I came.
The thing that summarized it for me was how we’re moving to a focus on
community-driven data conversations. We won’t be able to predict exactly, and
that’s the point.
DR. KAUSHAL: I, too, am a member of the workgroup, so it was a privilege to
sit in on your meeting today. The most interesting and exciting thing I heard
about was the coordination with the health data initiative. I’m looking forward
to looking on that piece in particular. I spend 60 percent of my time with
young, commercial companies who are all crying out and saying, where can I get
a hold of this data. I’m excited to get it to them and excited to sit around
this table next year and talk about all the great products and services that
have been released and all the value we’ve been able to create out there.
DR. TANG: I’m feeling a bit bipolar. I really was elated by some of the
demonstrations of the data that we could had, or what could be known, then I
think of the testimony from a couple of years ago about the people with the
problem. In a sense, this was a discussion I was part of before about the
underserved. I think we have a big data underserved community. This might be
the privileged few that know or even start to know about it. I’m trying to
figure out how do we get there, how do we solve that problem?
I’m a little bit– some other analogies– sometimes you need the Meals on
Wheels, in other words, you need to deliver it to the folks prepared. In
electronic health record systems, for example, it’s one of those things where
we really have to get some core knowledge and practice into the hands of people
who will use it. You can try the health stream kinds of things, but it doesn’t
work. You can try to bring them into class and feed them, and it sort of works.
If you go to the one-to-one and fix their problem, it works for a long time and
they’re really appreciative.
The experience for hi-tech and regional extension centers, the people who
were served by them, which is basically Feet On The Street, were twice as
likely to fulfill meaningful use. There’s a lot to– I wonder if there’s
another approach. We talk about tools and things like that. I wonder if we
actually need feet on the street. The problems are so real. We’ve even said it
here. It’s so unfathomable just to be aware that this was there in that format
that could be usable, but how do we connect the dots. Is there a way to connect
the dots without feet on the street?
If you look at the total cost of care, is it better if we have home care
than just making sure they don’t come back to the hospital in 30 days? It may
be worth something to know how do we really get– the resources about doing
something, about something you discover through the data is probably there. We
just need to make known– it’s the difference between effectiveness and
efficacy– what’s in front of their house, their face, and it could be tracked
over time. There’s so much good available, it’s just how to deploy it.
DR. MAYS: Being here makes me feel like I’m at the best of continuing
education because I always learn something. It’s kind of like it’s not just
about what I do in my day job, it’s elevated. I always feel that in terms of my
ability to work across teams and things like that also gets elevated because
you have to learn another language here. It’s not a foreign one, but it feels
foreign sometimes. You learn about other priorities and activities and things
like that, and you start to raise your own work by elevating it in terms of
taking the inputs. It’s always exciting to be here from the learning
For me, this meeting has been a mixed one. I just can’t imagine NCVHS
without Marjorie. My thing has always been if I can think about it, Marjorie
can solve it. She may not like sometimes the things that I bring to her, but it
really allows me to think about how can we really do this the best that we can.
It may not be what we usually do, but Marjorie will try and solve it and fix it
and think about it, so that part makes me very anxious. I like thinking about
how can we reach a different group or do things differently. That’s one of the
things that having you and Ed– it’s like when I have data problems or anything
and I go to Ed with an issue or someone at NCVHS, it’s always this willingness
I like the expansion, but Bill has made me so nervous about this expansion.
I want to go back to a suggestion that I made. I haven’t started counting the
meetings yet, but whatever meeting it was, I made a suggestion about maybe we
need to consider if there’s a way– Marjorie before you go, I’m going to put it
on your plate– to consider integrating a community member into this group.
If we’re seriously going to continue to work like this, if you look at how
much time we’re spending trying to generate community, how much time we have
with Susan going and doing this, it might be time for us to think about– and I
don’t know if you want to do it permanently, but in some kind of way– to
integrate a person that that’s their role, that’s their expertise, and that I
think they can help us to work a little more efficiently and to focus on what
we do the best.
Some of us do know the community stuff, but I think it might help, and
they’d sit as a resident expert in doing that. We sit here and debate with each
other, but I think having that person that is that in between for us, maybe
we’ve reached that point if we’re very serious that our agenda is now
community-driven, we need to not be the arm chair experts at doing it. One more
thing before you go, if you could think about a solution, it would be great.
DR. CHANDERRAJ: I am also very glad to be here. I’ve learned a lot, and I’m
really impressed not only with the expertise of the committee members, but also
wisdom of some of the leaders, Marjorie, Jim, Justine, and Larry. I’m so
impressed. Some of the comments, segueing into Justine’s comment about a letter
to the employer, not only should emphasize the importance of what are the
developments of this committee in shaping the policy of healthcare delivery in
the country, that will be very important, especially to the new members. We
don’t know what impact we’re making by these deliberations in coming here.
Over the course of events, I learned a lot about the savings accrued by
going electronic and gathering the data. Several speakers have alluded to the
savings. I think it will be a good thing if we can distribute the savings among
the participants, encouraging them to be partners and gathering this
information, especially the physician community. There is– I don’t know the
academic physician standpoint, but most of the community physicians feel that
the government is out to get them. That’s why the data is not coming out from
them, and they’re reluctant to provide this data. They see meaningful use as a
Trojan horse where the federal government is trying to get data on them to
impact their practice. If we can ease that feeling, then the data gathering
will become very useful and enroll them as partners rather than defenders, or
whatever we’d call it. The last thing is also transparency. That will go to my
previous point, that transparency from CMS and the insurers, private insurers
to enable the physicians about healthcare providers, providing care, so that
they’re not in fear of providing care at the same time, worried about the
DR. STEAD: I would second Lynn’s comment about how welcoming the group has
been. It does seem to me like it’s a moment where there’s a potential for
transformation in all the discussions I’ve heard, at the intersections, the
intersection of population health with the care for individuals as a member of
a sub-population, the intersection of Bruce as Paul’s conversation yesterday.
This business of the intersection of big data strategies with more
traditional survey and analytic strategies that Justine was mentioning, the
opportunity to really build privacy and security in from the bottom up so it’s
not a defense, it’s actually an inherent feature. I that is the opportunity,
modulo Justine’s comment that we will need process, because I think to do that
we’ve got to work as a committee of a whole, and that will take a new process.
MS. GOSS: There have been a lot of wonderful comments here today that I’d
like to echo quickly. I do echo Lynn’s comments about being welcomed. I do feel
like I’ve joined the family of NCVHS, so thank you for making this a very
wonderful first meeting. I am very honored to be here with you on this side of
the table. As I noted in my opening remarks, I’ve testified to this committee
in years gone by, so it was interesting to me as I sat on the first day of the
standards subcommittee and we were talking about claims attachments, I worked
on that in 2003 and trying to pull that together as a convener of the HL7 and
X12 efforts related to that many years ago.
For me, the excitement about being here is that I believed for a long time
that data is collected at the point of the encounter between the provider and
the patient. We need to bring the patients, keep them in the focus, but let
that data flow for all the necessary purposes and not complicate the process.
I’m very much a businessperson. I like to be efficient, and I like to get stuff
done, and I want to be proud of what we’re doing. It’s going to take a lot of
process for us to be truly effective thought leaders in this arena. Our ability
to be productive– I saw– when was that hearing, Tuesday or Wednesday, that
hearing was so productive and the feedback we got because we had very strong
staff support, work that had been done by OESS and others in preparing the
questions and preparing the industry to come to the table.
We got the bang for the buck out of that, and then we can start to think
about the key aspects and the hard topics we need to solve. Guess what? These
aren’t new topics; it’s just a new flavor of it that we’ve been evolving for a
long time. I think our staff support, our process, and our mindfulness about
being efficient is going to help us achieve some of Bill’s remarks about how do
we push away the noise and focus on what’s going to make the difference for the
people we love and the rest of the citizens of the United States.
DR. COHEN: I am constantly reminded that the whole is greater than the sum
of the parts when I’m with this group. The model of putting all these
incredible people from different experiences and different perspectives in the
same place to deal with one issue is incredibly awesome and overwhelming. I
wish I had more of these opportunities in my day job. That’s what makes this
place so unique, the expertise from so many different perspectives.
I’m constantly reminded that we have an awesome responsibility. We hold data
and trust, whether it’s for our patients or clients or the residents in our
communities. We collect these data for important, incredible uses. It’s our
responsibility to make wise and good decisions about how to use this
information. This is a place where we’re fortunate enough to consider how to do
Finally, Lee said something earlier, and I have to constantly remind myself
that this is a journey, it’s important to focus on products, but in some ways,
they’re just markers of our commitment and the sustainable and long-term
journey that we’re engaged in and that we need to pass down to the next
generation of national committee members, to maintain their vigilance on this
MR. DAVENHALL: I want to thank the committee for even allowing somebody like
me to be part of the family. I would have to say that I admire most of the
people that I’ve known for some time but I never really knew what you did.
While I know that there are no simple answers, I came here thinking that we’re
all in the same boat together. What I discovered very quickly was that I found
out what ocean I was floating in, and that’s really quite helpful and quite
revealing. On one hand, I’m really humbled by the fact that I’m even here and
that you allow someone like me to come into your fold and be assimilated.
There’s no distinction anymore, I just feel like I’m in the right place at the
right time and I have the freedom to express myself. That’s not a feeling you
get from very many opportunities to serve.
I want to compliment the committee and their ability to do that. I feel like
I really got to know most of you. Lastly, where else could you get to learn how
to cook salmon with George Salmon? It’s the epitome of what the committee has
all been about. We got together in the forms of friendships and bonds to carry
forth. I’ve been in this business for 40 years and created a career on using
the data that this committee has shepherded for all its life. As an end user, I
never knew this existed. I knew it existed in name, but I didn’t really know
what they did. I would say I’m humbled by the fact that it wasn’t by accident
that I was able to use this incredible legacy of well thought out data. With
all its pimples and warts that we want to talk about here, there’s nothing
close to it anywhere else in the world.
MS. KLOSS: I know the next great meeting is now going to start in 40
minutes, so I’ll be very brief. I have many observations, so I’ll hold those. I
had some thoughts to go back to Justine’s comment, about how we’re working in
two worlds right now at subcommittee and as the whole committee. We haven’t
quite figured out– we’re not smooth yet.
A couple of observations, we learned so much at the discussion of claims
attachment. As I was thinking about that, wouldn’t it be great to have some
high level discussion of that by the full committee before the subcommittee
starts drilling down into recommendations? It’s going to come back in some
letter, but then really it’s almost too late to explore how we innovate. I
think we need to look at ways to get the inside of the committee upfront on
some of these issues and allow the subcommittees then to actually go faster on
work. I thought that was illustrated in this meeting because so many of you
participated in the combined work on the hearing.
I think we have a much better sense of all of the opportunities that will
come out of that because we had so much cross-fertilization on that. I think
that was kind of a lesson. I think there probably are some ways that we need to
look at restructuring the work of the subcommittees. We’ve got a lot of heavy
lifting, and you can’t do it in an hour with the next group waiting to occupy
the room. I think that piece needs some thinking differently. We’ll figure it
out. Thank you all very much.
DR. FRANCIS: As somebody who is a year into her second term, I was struck by
two things that were wonderfully different about this meeting. The first was
how much more I learned about what’s going on in the department than I had
learned at many other meetings and at NCHS in a coherent way. I think that is
enormously useful as a way to help us set priorities, the more we know about
what’s going on in the department, the more we can see where we can be helpful
ahead of the curve, so to speak.
The second observation is that more so than at any other time in a meeting,
and this came from the department, too, I had a sense of how our work makes a
difference. I think of all the hard pieces that I’ve been a part of at this
committee, the sensitive information letter was the hardest.
It was a letter that we really struggled with. Look at the influence it’s
had, as we heard yesterday. My hope is that the work that we’re going to be
doing over the next year will be similarly influential, although one hopes it
will be birthed in a somewhat– maybe I’ll put it this way. Families struggle
with having different views, and when families work well, the way they handle
that struggle is with mutual respect. I think that was true of that process.
The family analogy to me is that mutual respect, which I’m sure will continue
as we work through what are not easy issues again.
MR. SCANLON: To all the baby boomers, and I count myself in that category
and we have some younger as well, the rule about retirement should be that if
you’re younger than Mick Jagger or Paul McCartney, you shouldn’t be thinking of
retirement. That’s my take.
MR. SCANLON: A couple of things before we close tonight. I want to reiterate
a couple of points. I want to thank all of the regular committee members for
all the work that they’ve done, and I want to thank the new committee members
that we’ve named for joining the committee and being willing to devote their
time. That includes the committee members that we reappointed as well. I think
what we are– every now and then in your work, you see something come together
the way you hoped it would. That’s kind of the way I think of the NCVHS.
When we started the reinvention of the committee, expanding the role to
include HIPAA, electronic health records, and privacy, and now we’re looking
more at population health, community health, and really bringing all of these
dimensions together– every now and then at HHS, and I’ve had the experience,
where something I planned or something we did actually worked the way it was
supposed to or even better, and that’s happened with the committee. I just keep
my fingers crossed that we keep along these dimensions.
I think we do have to be careful, as Bill said. We really have to
prioritize. The temptation is to go off into every attractive area that appeals
to us. I think it requires some discipline and prioritization as to what will
make the biggest difference and what was worth spending our time on. I think we
do have to be careful of that. The other thing, I’m happy to see us all coming
back as we do. While there’s tremendous depth on the committee in individual
areas, ranging from standards to privacy and statistics, electronic health
records and so on, and there have been tremendous accomplishments and
contributions in those areas, and I just think when I look at how far the
department’s getting how much a difference it made because the committee made
those recommendations. I think in standards for example, just think of how
different it would be if we didn’t have the committee to advise us on privacy
or electronic health records.
I wanted to assure the committee two things. One, if there’s any way we can
make your work easier from the point of view of staff. If there’s a way to work
smarter or faster or more efficiently to respect your time, please let us know.
Secondly, I want to reiterate what Ed said and what I think I said yesterday,
that despite the “s” word and changes in personnel, the department is
certainly committed to the committee, is very supportive of the committee. It’s
really one of our most productive committees. I just want to assure the
committee that there should be little or no interruption in terms of support or
services. In fact, we’re finding even additional staff members for the
DR. GREEN: Well, you guys sure follow instructions well. Remember we started
saying if you could just take one or two sentences and go around the table. At
the end of the next meeting I’m going to say pick one word that captures your
experience for this meeting, and we’ll see what happens with that. One very
mundane thing, our liaison from NIH, Bob Kaplan, emailed me yesterday pointing
out the latest IOM study about public health that Steve Wolf chaired.
It has a recommendation in it that NIH work with NCHS to figure out what we
need to do to have better alignment with our data standards internationally. If
he had been able to be here, he would have brought that message himself. I told
him that I would relay that, and what I’ve done is I’ve told Marjorie, so
that’s all we have to do about that for right now. I don’t have anything to
add. I appreciate the comments very much. I just want to reinforce one word
that wasn’t said quite as loudly as I think we need to say it, and the word is
“transition”. Ed Sondik after 17 years is retiring. There will be a
search for another director. Marjorie is going to retire. We’ll need a new
We have a fully populated committee with new members. We have the potential
for some additional staff. We are agreed: we simply must transition our
structure and function. We are agreed that we have to get better at
transitioning from what we might do to what we must do. This is going to be a
lift. This is not going to be easy. Many of you picked that up. Between now and
the next meeting, and also for the September meeting, change is inevitable. It
may be relatively substantial. I urge all of you to welcome it like an old
friend. It’s okay.
I really like Linda’s positive framing of it. We need work. We need to
figure this out better. It’s not working well enough. I think part of the
secret sauce is we have not figured out how to take advantage of the terrific
staff. We behave more like staff than the NCVHS. We like to get down in the
weeds and start talking about the details of this word and that word, but we’ve
got some great people around, and if we said the direction is that way and the
critical concepts are A and B and we’re going to watch this like a hawk and if
you steer off of A and B we’re going to get on your case but then had
confidence in our staffing to bring stuff back– we just haven’t figured out
how to do that yet.
I think that’s on our horizon. I ask for your help in figuring out how to do
that. That will help to deal with Bill’s bandwidth thing. We’re going to have
to work that from both sides: narrow the scope, elevate the support that pushes
us forward and try to accelerate. We talked last night and actually during the
meeting about Carl White. His name comes up over and over again. He was very
fond of teaching with adages, as I’ve remarked to you before. Your last
discussion there reminds me of one of his that I’ll conclude the meeting with.
He was very fond of saying, Larry, everything takes longer.
DR. CARR: Could I ask that as we adjourn we give Marjorie her standing
DR. GREEN: While everyone heads for lunch, could I ask Ed and Marjorie and
Jim to stay here just for a minute? I’d like to get a picture of you standing
(Whereupon, the Full Committee adjourned at 12:30 p.m.)