[This Transcript is Unedited]
Department of Health and Human Services
Working Group on Data Access and Use
June 12, 2014
Hubert H. Humphrey Building
200 Independence Avenue, SW, Room 705-A
Washington, D.C. 20024
TABLE OF CONTENTS
- Welcome – Overview Recent and Planned Activities
- HHS CTO’s Office, and the Health Data Initiative – Damon Davis, HHS
- Agenda Item: Live Demo and WG Feedback Discussion of Some of the Updates to HHS Programs – Kate Brett, NCHS
- Follow Up
- Comments by Karen DeSalvo from ONC
P R O C E E D I N G S (1:03 p.m.)
DR. CARR: Welcome everybody to the Working Group on Data Access and Use. I
am Justine Carr, chair of the workgroup and Steward Healthcare. To my left, the
incoming chair of the workgroup.
(Introductions around table)
DR. MAYS: Welcome, everyone. I know many of us were just here last week for
the Datapalooza so I really appreciate the trip back. As you know, today is my
last meeting. I am delighted that Vickie is taking over. Not only is this a new
beginning with Vickie in the leadership role, but it is really a new beginning
for the configuration of this group and how we work.
I think we have learned a lot over the last couple of years about what is
the way to optimize the input of folks. I think you will see in the
configuration going forward that we will be tapping your expertise in a
coordinated and very valuable way. It will also be a great opportunity for you
to hear what is going on in the Department and for us to have a robust,
real-time exchange about what we know and how we can help.
For starters today, I guess Damon is going to give us an overview. After
that – the one thing we talked about and maybe we do it now – I want
to make sure that everyone in the room recognizes the depth of expertise of the
folks in the room. Maybe, Damon, that would be helpful for us to go a little
bit deeper. I know we just did introductions, but go around a little bit more
and maybe give a paragraph about what you do and how it relates to the working
group. With everyone’s permission, maybe I will start again.
I am Justine Carr. I am Chief Medical Officer for Steward Healthcare. We are
a community-based healthcare delivery system that comprises ten hospitals and
2,700 physicians in eastern Mass. We were one of the first pioneers and also
part of the BlueCross AQC program. We are a program that is very much focused
on total medical expense, as opposed to fee for service.
I have lived experience on many aspects, meaningful use being my most
consuming lately, Paul. I am leaving so it doesn’t really matter.
DR. MAYS: I am a professor at UCLA in the Department of Psychology and also
in health policy and management. I direct a NIH health disparities study. We do
a lot of technology-driven activities to try to get interventions to
populations that are greatly in need. We design text programs for women who
have uncontrolled diabetes. We are working on another totally virtual smoking
cessation intervention in Korean youth.
We do a tech collaborative with the Federally Qualified Health Centers in
our area, where what we are really trying to do is to get some of our
physicians to kind of think about when you give somebody some advice you want
to exercise, you diagnose them as depressed, can we get you then to also have
your care manager or some of the volunteers that we use to actually set you up
with some technology that both gets information back into the care system, but,
as well, empowers the person to be able to do work. Part of what we are really
working on with those physicians is kind of what are all the collaborative
tools that you can use? How can they help you improve what you are doing?
We have a number of other types of things that we do, just in the community,
that also revolve around the use of technology for self-help, as well as kind
of monitoring their health.
DR. TANG: Paul Tang, Chief Innovation Technology Officer for Palo Alto
Medical Foundation, which is essentially a large medical group practice. I have
a couple portfolios in the Innovation Center. One is on helping seniors age in
community, looking at the social determinants and the non-medical side of
humans as they grow older. The other piece is in using online tools to help
people with chronic health conditions change their personal behavior, basically
getting feedback from their own health and us being able to support them as a
DR. ROSENTHAL: Joshua Rosenthal. My day job is with a company called
Roadmap. Kind of background, PhD, Fulbright to the Sorbonne’s Institute for
Advanced Study. I do some academicky things – guest lecturing at Harvard,
Hopkins, MIT from time to time, around having people – healthcare
entrepreneurs actually building businesses that have market validity and
sustainability. I have founded or co-founded multiple analytic companies,
specifically taken each one to successful exit either through private equity or
through strategic investment, and done this largely commercializing public
data, Dartmouth Atlas, things like that. Work has won awards from clinical
stuff to non-clinical stuff, Entrepreneur Magazine, Business Week, Innovation,
stuff like that.
What we do in the day job is actually a business management platform. I have
showed some of this from time to time. We basically take every public data
source which has been released in any meaningful way and synthesize it and put
it in a nice browser and sell it to the payers and integrated networks, as they
actually, for the first time, don’t just want to underwrite risk, but want to
manage members and improve patient satisfaction. Literally every dataset that
we are talking about have pretty extensive technical expertise around using and
business expertise, as well.
I guess if you need a thesis like in a nutshell, look, data is great. Up
until now, it has been about apps. Doing these healthcare/entrepreneurial boot
camps, most startups fail. Most of the folks at HHS, I think, will admit we
can’t do it just in a non-profit world. You need a robust public/private
partnership. When they thought about releasing the data that was really
interesting. It worked well. Geolocation, weather, those are real markets.
Healthcare, obviously not a real market in a lot of ways – supply driving
demand, for instance. Stuff hasn’t worked as we might have hoped.
Part of that has been having the people producing the data actually
recognize what market drivers can actually craft policy in a meaningful sort of
way. In terms of entrepreneurs actually using the stuff, it is really tough. It
is really easy to build an app, but if no one wants to use it, there is no
meaningful exit around that. Having meaningful expertise, using the data and
building a market has been something pretty tough. I am very, very passionate
about helping other folks be able to do that.
On the status quo side, there is a study I am just sending to you using
behavioral risk factor surveillance system data where a couple of our friends,
good friends from Dartmouth Institute, showed how that out-predicts risk points
for Medicare Advantage instead of claims-based systems. Public data, high level
granularity just smoking risk models. That can be very threatening to people
who have system or expertise or even kind of investments around kind of archaic
– I don’t want to say archaic systems, but fee-for-service model systems.
I am very passionate about using public data for very specific uses in the
market and having a thousand flowers flourish, which hasn’t quite happened yet,
but I think we are at the cusp of it in a lot of ways. I organize the tech
track at Health Datapalooza. I do a lot of stuff like that.
DR. COHEN: You are always a tough act to follow. I am glad BRFSS is working,
since that is one of the –
DR. ROSENTHAL: I know. I thought you would love that. I am going to send you
– I can send that out right now.
DR. COHEN: I am a public health data geek at all levels. I started out at
NCHS. I worked on HANES and NHIS. I helped author a series on statistical notes
for health planners. I moved up the food chain to work at the State Health
Department. I collect survey data. I have been chair of the IRB. I have
developed a web-based data query system. I also work extensively in communities
in the metro Boston area doing health needs assessment activities and working
in a small village in Nicaragua to generate health data to address an epidemic
of chronic kidney disease there.
Research interests are in redefining how we measure disparities,
particularly with respect to race and ethnicity and also in reproductive
technology, looking at assisted reproductive technology and the impact that has
on population health.
With respect to this committee, my focus is on what I call data liberation
2.0, which is making all of this information available and useful to
MR. DAVIS: So my name is Damon Davis. I work in the HHS IDEA Lab, formerly
known as the Chief Technology Officer’s Office. There we are focused on the
broad innovations that will help government basically do government better, but
also be better engaged with the communities who are utilizing the data that the
I always like to remind folks how broad the department actually is. We are
multiple, multiple operating divisions of massive agencies – National
Institutes of Health, Food and Drug Administration, Centers for Disease Control
and Prevention just to name a few of the big brands. Part of my job is to act
as a liaison to those various operating divisions, who are all engaged in
multiple levels of innovative work in terms of getting data to be openly
available from some of the data and from the reports and project and grants
that they administer, as well as sort of refining the data into something that
can be more useable for the entrepreneurial community, more useable for the
localized communities who are trying to make public health and access to health
better for the people that they serve.
My job is, again, to serve as a data evangelist, both inside the Department
and outside the Department, as well as with groups like you all, who can offer
some insights into how the data can be utilized better, can be made available
more quickly, more efficiently.
Just as a bit of additional background, I was previously in the Office of
the National Coordinator for Health Information Technology, where I focused on
communications and a little bit of our consumer e-health program where we
focused on other data elements, namely, the blue button program where we were
trying to make sure that individuals, one, know that they have the ability or
the right to gain electronic access to their personal health information, but
then not only getting access to it, but actually trying to obtain it in order
to be a meaningful participant in the health care and social services system,
not just have health care done to you, but be participatory in your own care. A
little bit of background as to my HHS experiences.
DR. FULCHER: Chris Fulcher. I serve as a co-director of a center at the
University of Missouri called CARES or Center for Applied Research in
Environmental Systems. We have been around for over 20 years focusing on GIS or
Geographic Information Systems, data visualization, and, as it has involved
over the years, really much more in the area of community and community
process, not just from a health perspective, but economic development,
education, et cetera.
We have built a number of custom learning systems over the years for
different foundations, government agencies, and non-profits. What we realized
very quickly was that it wasn’t serving communities as well as we wanted to
because we were driving people away to all of these different systems. The
outgrowth of that is partnering with our non-profit, called IP3 or Institute
for People, Place, and Possibility, that helps manage community commons, which
is a national public good utility.
Really, the conversation now is really around alignment in investments,
alignment of how we collect data. As I mentioned to Damon earlier, we stand on
the shoulders of what you all have created at the federal level, not just at
HHS, but USDA, Census Bureau, et cetera. It is fantastic, in terms of getting
the conversation started at that community level. Invariably, they want to go
beyond what is available nationally or at the federal level. What local and
regional data can they lay on top of that to tell that better story about
people and place? Without that federal context, the federal datasets, they are
lost in so many ways not having that underpinning there.
It is an exciting journey. It is evolving with Community Commons and how we
are working across these different funders and making more of this data
available in this public good utility.
DR. VAUGHAN: Leah Vaughan, I am a physician/epidemiologist/public health
researcher, particularly focusing on spatial/temporal analysis of geospatial
data. Broadly cast, I would say my areas of interest are particularly around
data and technology for social good and public health impact. I am most
especially recently inspired by work with young people.
DR. GREEN: I am Larry Green. I am a physician in Denver, Colorado, Chair of
the NCVHS. I do projects. They are all about redesign. If Damon is a data
evangelist, I am a redesign evangelist. I work for my grandkids.
DR. FRANCIS: I am Leslie Francis. I am both a philosopher and a lawyer. I am
a distinguished professor of philosophy and of law at the University of Utah.
My PhD is in philosophy.
I am the president elect of the Pacific division of the APA. A lot of what I
work on in philosophy is privacy and a wide variety of applied ethics issues,
including environment, actually. I also – on the law side, I mostly work
on, these days, various areas of health law and disability rights. I have an
NSF grant looking at the legal needs of adults with diagnoses of autism. They
tend to get thought of as children, not as adults.
I guess if you were going to bring it all together, my interest over my
career has been in using what I have been privileged to have academically to
try to further various issues of social justice. I see public health as very
much one of those.
MS. JACKSON: Debbie Jackson, I am with the National Center for Health
Statistics, CDC. I have been able to parlay a Master’s degree in English to
catapult into public health through the American Medical Association, some time
there at the accreditation council on graduate medical education in Chicago. I
am a master convener.
DR. SUAREZ: Good afternoon. I am Walter Suarez. I am the Executive Director
of Health IT Strategy and Policy for Kaiser Permanente. In that role, I
actually work significantly on various areas of health IT policy and health IT
standards, specifically areas related to electronic health record standards,
administrative standards, and privacy and security, health information
exchange, other things.
I have a special interest in two areas that I work on. One is privacy and
security. The other one is public health and population health management. In
this last area, in this population health management area is where Kaiser has
invested significant resources and funding in developing population health
management tools that use a variety of data, not just our internal enrollment
an member and patient data, but also community-based data to understand better
the community where our members and our patients live.
We have, of course, many, many different initiatives around health data,
health data analysis, big data analytics, and all of those kinds of things, so
all the way from the clinical space, clinical research as well as applied
health research, and then all the way to population health areas and
evidence-based medicine and comparative effectiveness research and other things
We are a large integrated delivery system, as our nature has been always. We
operate in about 10 different states, about 38 different hospitals, mostly in
the West Coast, 600 medical facilities with over 20,000 – almost 20,000
employee physicians and covering almost 10 million members at this point.
DR. CARR: On the phone, Kenyon, are you there?
MR. CROWLEY: Yes, I am here. Good morning, everyone.
DR. CARR: Good morning. Just introduce yourself and give us a paragraph or
so about what you are doing in your day job.
MR. CROWLEY: Sure. This is Kenyon Crowley. I am with the University of
Maryland Center for Health Information and Decision Systems, where I am deputy
director and run a research portfolio covering a range of health information
technology topics focused on the adoption, effective use, and evaluation of
information systems at different clinical contexts, stretching from
patient-facing to provider-facing. So I have a group that does a lot of work in
healthcare analytics, then we also focus on quality and transparency of data.
DR. CARR: Is there anyone else on the line.
MS. NILSEN: I am Wendy Nilsen. I am from the National
Institute of Health and the National Science Foundation. I run the Smart and
Connected Health Program at NSF, which is a joint NIH-NSF venture to really
kind of bring the smart technology to health, broadly defined. At NIH, I am in
the Office of Behavioral and Social Sciences research. I lead many of our
initiatives in mobile health in ways to thinking of it from the clinic all the
way out to empowering health in the community, as well as data colletion and
MR. DAVIS: Thank you. That was helpful. Some of the things that I forget are
you guys are here participating as volunteers. You are volunteering your time
away from a day job. I really do appreciate the fact that I can offer some
insights as to what we are trying to accomplish, but then hear your varied
perspectives so I know where you are coming from. Thank you. That was a great
idea, Justine. Over to me?
DR. CARR: Yes, sir.
MR. DAVIS: Fair enough. So this is Damon Davis. I am going to walk us
through a little bit of an update as to where our work is at the CTO level with
regard to our open health data initiative. I commonly refer to it as the HDI,
the Health Data Initiative, which you will see on this first slide here.
I will just remind everybody quickly that one of the things that happened
back in November of 2013 was we released a strategy for our HDI execution plan.
It is an HDI strategy and execution plan. In that strategy and execution plan,
we had basically five overarching goals that we wanted to make sure to focus
on. What I am going to talk through today is a little bit of a sort of progress
report as to where I feel we are with some of these initiatives.
To remind everybody what those five goals are, the first of which is
advancing the healthdata.gov platform. We want to make sure the site is both
user friendly, but user functional. We want to go ahead and try to highlight
department assets that support achieving HHS strategic initiatives. In
recognition of the fact that the Department has its own strategic agenda and a
publicly stated strategic plan, we want to make sure that the data liberation
activities that we are focused on are actually very much aligned with that
We want to educate new and existing and internal and external participants
in the health data ecosystem. It is important to recognize the distinctions
between each of these audiences because they are varied, but there are some
similarities across each. Recognizing that there will be newcomers to the
platform healthdata.gov, as well as existing or veteran users who are
constantly looking for additional resources or data.
There are also internal and external users of HHS data. One of the fun
projects that I have been lucky to be engaged in is sort of building out the
healthdata.gov blog. One of the stories that I very much like to tell is not
just about how the innovative entrepreneurial community might be utilizing the
data, but how the Department, itself, is using its own data. I find that it is
really, really interesting when you start to talk to people across the
Department about the activities that they are engaged in when they start to
say, oh, well, the Agency for Health Research and Quality uses this CMS dataset
for the Monarch System, for example. It is really nice to start to see those
cross-collaborations. That is another – that is the internal side. The
external side probably speaks for itself given our close proximity to the
recent Health Datapalooza.
We want to enable and incentivize the health data ecosystem. There is
certainly a portion of what we are trying to accomplish that is making the data
available. We also want to enable people and incentivize them to come to the
platform, look for additional datasets that are now available, but also make
suggestions about what could be made available, how it could be made available
in a different format, et cetera.
We want to also implement administration and departmental policies that
foster data openness. Clearly, the White House has made some decisions that are
affecting all departments of the government, in terms of initiating their own
open data journeys. We were very fortunate to have the dynamic Todd Park as our
chief technology officer prior to his arrival at the White House. What that
afforded us was an opportunity to get out in front of many of these efforts. We
are lucky to be somewhat heralded as one of the leading edge agencies in terms
of our open data agenda. However, that doesn’t mean that there aren’t other
policy considerations that we have to make.
Then there are also the internal policy considerations that we need to make.
For example, how is it that we make sure that open data is, in fact, something
that is going to be institutionalized and not some flash in the pan fad that
will go away for some reason that we are not able to predict.
Those are some overarching ideas as to what it is that we are going to try
to accomplish with our strategic and execution plan. Again, it is available
online at healthdata.gov on the blog. You can search for, morbidly, execution.
You will find that blog quite quickly. It does come up. It is pretty rich, in
terms of the different activities that we would like to be engaged in.
I will just drive into each one of these different goals quickly and give
you a little bit of a progress report on each. You will forgive me for reading
from some notes as I try to make sure to hit some of the points.
One of the things we are trying to accomplish under advancing the
healthdata.gov website is improving the quality of metadata for better
usability. I will just maybe read through each of these kind of quickly to help
you understand, especially for those on the phone who may not be able to see
The areas under advancing the platform are improving metadata quality. An
area of the platform for non-HHS datasets, creating some user feedback loops
and developing and publicizing the ability to store and host HHS data, as well
as improving sort and search.
Within each of these domains, I will just highlight some of the activities
that have happened very quickly. In terms of metadata quality, this is in
progress, but, unfortunately, I feel like we could be doing a little bit
better. One of the challenges, I think, is just the recognition that this is a
very, very large agency and metadata quality is just going to have to start
somewhere and build out from there.
I think it is also very important to recognize that there are a small cadre
of people working on this. It is a very, very large behemoth of a task.
Metadata is by no means an easy thing to tackle. We are going to need a little
bit of time. One of the things we are trying to do is build in some of the
template language, the Common Core metadata fields that will at least drive new
datasets towards a little bit better metadata going forward.
An area of the platform for non-HHS datasets is in progress and we have been
making some significant progress there. My colleagues Serena Parks in the back
of the room and I have been working diligently to reach out to various states
and localized entities with the desire to make healthdata.gov not just a
discovery zone for HHS datasets, but a discovery zone for health datasets in
the public domain writ large and predominantly from .gov sites, but there are
opportunities for us to take data from other resources. We will explore that at
a later date.
Just to sort of recap what we have done so far, we have nine states who are
federating data into healthdata.gov right now. Illinois, New York, Missouri,
Hawaii, Maryland, Colorado, Oklahoma, Washington, and Oregon are currently
sharing data or have federated their data catalogue into healthdata.gov. Part
of the strategy has been to think through not only just where can we find
states, but let’s think strategically about where we can have an impact. One of
the things I decided to do was in recognition of my inability to reach out to
50 states, I decided to focus on the top 10 most populous states. Hopefully, in
the wake of that activity, we would bring along other smaller states, who would
be interested in open data, too. Currently, Serena and I are focused on Texas,
Florida, and California. Obviously, there are other states in the mix, but that
is just a brief update on those.
Soon, we will have some United Kingdom datasets, as well. A quick tidbit,
the HHS has a memorandum of understanding with NHS England. We have been
working through a couple of different work streams as a part of that MOU. I
would say more about that at a later time, but just know that one of the work
streams is open data. We have decided to focus on obesity data as one domain of
data sharing so that we can start to have some collaborations in terms of
understanding obesity in each of the two nations, as well as encouraging some
entrepreneurial endeavors in those areas.
I spoke a little bit about other federal entities having their data be made
available, as well. I think it is important to recognize that health data does
not just reside at HHS. Health data is available out of the Department of
Education, the USDA, and many, many other places where health is a
consideration as part of the mission of that department. We have been trying to
figure out the different avenues through which we can get some health data from
other agencies. Just recognize that some other federal agencies are going to
appear on healthdata.gov, as well.
In considering the user feedback loops, we have been thinking through, just
generally, how is it that we can start to more efficiently interact with
communities? How do we engage with communities that already exist? When I say
community, I mean communities not so much in the localized, geographical term
as I do in the entrepreneurial space, in the clinical use of data space.
Communities who are utilizing health data is how I am using that term.
There are some feedback loops internally where, with the operating divisions
that I previously mentioned, we have what is called a Health Data Leads Group.
That Health Data Leads Group are my direct liaisons into each one of those
operating divisions. What they have done in many cases is set up their own
Health Data Leads Group in recognition of the breadth of their own agency. What
I am alluding to is a feedback loop that basically suggests that the internal
operators are actually trying to actively and proactively seek out data
resources from across their various operating divisions. We also want to
utilize the platform in order to create the ability to ask questions about data
you already see, offer up ideas for data you would like to see, and generally
just creates the ability to have a conversation about the data in whatever way
the public sees fit.
We want to also develop and publicize the ability to store HHS data and host
it for other entities. For example, there are some small elements of various
operating divisions that don’t necessarily have a budget for data storage, but
they have an awesome project that they are really, really interested in sharing
the data from. If we can create some level of efficiency for them by offering
an IT solution that will allow them to share their data, create an automatic
API into that data, but not necessarily have to go through the contract
mechanisms that would require them to set up, say, a centralized data store or
what have you, we can potentially offer some efficiencies there. The platform
is going to be built out in those varied ways.
One last thing, improving sort and search. We have not yet started the
improvement of sort and search, but I think it coincides with the metadata
piece where we are building out better keywords and things along those lines
associated with the datasets.
Moving onto the next slide, highlighting departmental assets that support
achieving HHS strategic goals. We have been working internally on sort of
drafting a definition of what a strategically relevant dataset is. We have sort
of coined the acronym SRDA. You are not a good fit if you do not have an
acronym attributed to your work here. I feel like part of my work here is done
now that I have this acronym.
I think it is really important to recognize this is something I would love
to talk with this group about more fully. Please put an asterisk by it or
whatever. At some point, on a future date, I would love to figure out if, in
fact, this is a worthy action for us to undertake.
Just to build it out a little bit, what I am thinking through is we have
tons and tons of data. People are going to evangelize for the utilization of
all kinds of different datasets. I am of the opinion, at least currently, that
it might be very, very good for at least a communications and marketing and
sort of outreach campaign type thing to say here are the datasets that are
relevant to our strategic mission. Here is where you can find them. And give
people additional education about that group of data, in order to create a
larger conversation about other and alternative data uses.
Currently, I am engaged with my Health Data Leads to a small degree with the
idea of defining what a strategically relevant dataset is. Then we are going to
march down the pathway toward making those strategically relevant data assets
more widely known. Mark that in your minds as something we should come back to.
I would love to hear whether you think it is a worthwhile action and some of
the different ways you think this could be a valuable activity.
We want to further drive greater data openness. I would argue that while we
have set the default for some of our data to open, there are a lot of people on
the line level, who are administering the grants and doing the reports and all
kinds of things, who sort of have heard of open data, but may not necessarily
understand their direct space in HHS as a proponent for open data. I think we
want to further drive the openness. We are still releasing reports with PDFs. I
have an issue that I am working right now that we got from Twitter where
someone said this is great that you have this report, but, hello, can I have
the data? I think we still have a little ways to go in actually, truly having
the default setting be openly available data.
We want to work with each HHS division on external outreach to broad as well
as targeted, specific ecosystem participants. The Health Datapalooza is a
perfect example of the broad, I think. It is everybody from clinical users to
researchers to the media, entrepreneurs, et cetera, et cetera. I think there
are going to be times across our work where we are going to want to reach out
specifically to some targeted audiences to help them understand what data is
available, how it can be utilized for their various needs, and then get
feedback from them as to how we can advance that cause.
There have been some great educational resources coming out of various
entities across HHS. I would categorize that both in this highlighting our
departmental assets, as well as in our next section, which is education and
Let’s see the next thing I want to do is increase traffic to healthdata.gov
as a discovery zone. I think that one is pretty obvious. It is a great
resource. If we are not talking about the fact that the resource is there and I
mean we as in, we, the collective in this room and on the phone, we are doing
the data a little bit of a disservice by making it freely and openly available.
We want to increase the richness of the blog content that is coming out, talk
about not just HHS data activities, but data activities that are transpiring
across the nation. If people are using the data for a challenge or if people
are just setting up a challenge that is going to meet some public health need,
we want to help amplify that message.
One of the things that I have been trying to encourage folks to do is not
just post a dataset, but actually post a dataset and post a blog about the
availability of that dataset. We now have something to tweet about that then
creates this broader communications channel that directs people to the
utilization of the data. Again, the broad outreach about how we can drive
people to be engaged with healthdata.gov is going to be another valuable
component of what I think this group can offer opinions on.
I am going to go ahead and advance to the next slide, which is educating new
and existing internal and external participants. I should just say I am happy
to take questions. You guys should not just let me run, run, run. If you see
something you want to ask about, please jump in. Let me just pause for a quick
DR. FULCHER: With SRDA that is fantastic in terms of from the HHS
standpoint. Do you see that happening in a similar vein across the other
federal agencies, in terms of elevating those strategic datasets? How are you
quantifying which ones are the strategic?
MR. DAVIS: It is a good question for us to think through. One, this is an
experiment. We are going to see if it is even worthwhile. When we do it, does
it yield anything? It may. It may not. I may be that there is increased traffic
to those data assets on the various catalogue platforms, such that we can give
some measure of whether this was a worthwhile activity.
Again, I don’t know, but if this is successful and we can tout it as a
success, it may be that this is then cookie cuttered into other departments to
help them with their own open data communications and marketing
“strategy”. It is going to be really interesting to see what the
feedback from this group is, what we decide internally, and how we mash it all
up into an approach that is going to be meaningful for defining and actually
highlighting the strategically relevant data assets. I am kind of – I am
really excited to see what is going to happen.
If nothing else, if it fails, we will have learned that that is not –
and that is sort of a tenant of the IDEA Lab. Let’s have some experimentation
and figure out what is going to work and doesn’t work and learn something from
it. I am excited to get some learning from this.
DR. FULCHER: Would that be something that we could be kept up to date on?
MR. DAVIS: Absolutely. I would be happy to update the group through any
proper channel that is deemed appropriate. I am happy to come back to this
meeting on a recurring basis. I would love to have that feedback loop. This
goes back to what I was talking about previously.
DR. CARR: You already penciled in for every meeting just like this because
this is fantastic. Secondly, it is also true that if you have something that is
time-sensitive, we can pull a call together and just kind of give you some
quick feedback. This group is brilliantly potent.
MR. DAVIS: I appreciate it. Yes. This group has been fantastic for me
before, in terms of the sort of feedback that we had on – it wasn’t the
strategic plan, but there was another document that was sent around. Very
quickly, we got some really good feedback. I appreciate that.
DR. COHEN: How have you built in a mechanism for ongoing community feedback
for these efforts?
MR. DAVIS: So that is something that is still under development. I will also
need the consideration of this group in order to do so. Frequently, when we go
out we will talk about healthdata.gov and the ideas tab and the Q&A tab and
encourage feedback to come through some of those formal, online feedback loops.
We do not have, necessarily, in my opinion, some of the strong recurring
community feedback that we probably would like.
That is as much a function of sort of realizing where we need to prioritize
outreach and feedback to having the resources to even handle it. Currently,
Greg, Serena, and myself are the predominant people who are engaged with
healthdata.gov from this level. I think it is challenging from a resource
allocation perspective. That is not to say that it has to necessarily come
through us. If you have a group, say Open FDA, if you have people who are very
interested in the FDA datasets, that community doesn’t need to come to us. They
can go straight to those guys. I would love to know what is happening because I
think there could be some fantastic stories and interesting information coming
out of there, but that is one of many challenges that I think we face is the
continual community engagement.
DR. SUAREZ: Three quick questions, one on metadata, one on usefulness versus
usability, and one on structure versus non-structure.
Metadata, a lot of important things go into metadata. One of them is
certainly data provenance. If you could speak a little more about the work that
is being done to pursue standardization on that. I know ONC, for example, has
an initiative that applies to clinical health information and other information
that focuses on data provenance and codifying data provenance as part of
metadata. It would be interesting to know how that element on the metadata is
being captured and what you can tell us about that.
The second question is about – you mentioned usability. In the first
slide, you actually mentioned it like three times. Has there been work done to
evaluate and perhaps identify a way to document the usefulness of the data
rather than the usability of the data?
The third question about structure versus unstructured, the sense, of
course, is that most if not all of this data is somehow structured data, in
other words, data that is coded and has some coding that can be analyzed
through computable mechanisms. How much of the data exists in unstructured
form, unstructured meaning, basically, non-coded form like notations or free
text or other things that are not really coded information?
MR. DAVIS: I will start at the top and work my way down. Metadata and data
provenance. I immediately think of the example of sort of research data out of
NIH. One of our – we will get to the policy slide at a later time, but one
of the things that we are going to need to do is more strongly consider how we
are looking at federally funded research data. There is the idea of sort of the
publication in PubMed, but there are the datasets that go behind the actual
utility of that report and, basically, the proof of the hypothesis and the
research. One of the challenges, then, is as that data becomes available for
broader use, how you attribute that data back to that research based on what
that individual grant and researcher did.
Therein lies one of our challenges that we are currently considering. How do
we get to a place where we can better align the data that has been produced
with the activity that produced it so that you can have more of a direct
lineage from some of its multiple alternative after uses or subsequent uses
back to its origination? I don’t have a specific answer as much as that is
something that we are working on.
In terms of documenting the usefulness of the data, it made me think of an
issue that I brought up with Greg recently. We were trying to think through
what it would mean to actually put, say, a rating system, literally, on
healthdata.gov that would say data openness, for example. If this is a PDF, it
gets a one. If this is XML, CSV, or an API, you are moving into the threes,
fours, and fives.
I would liken that to a data usefulness type rating with the assumption that
anything that makes it into an API or CSV type format has gone through some
level of additional work that would make it useable in that type of format. I
am hopeful that we will figure out a way to do something along those lines.
DR. SUAREZ: Before you go to the next one, on that point, I think that still
is talking about the usability, how usable is the data. In other words, if it
is really a rated one level data where the data is PDF and is not computable,
but that still is usability, not so much usefulness as in the other end of the
spectrum, which is has the data demonstrated that it is valuable and it helps
them achieve the goals of using the data?
MR. DAVIS: How would you get to that? Can you think of an example?
DR. SUAREZ: There are actually, in some ways, examples of how to grade, if
you will, or at least associate the degree to which some data is more useful to
achieve certain goals than other data. I think that is what I was getting to.
Sometimes putting out data there just for the interest of putting it out there
doesn’t mean the data is useful, but it could be as usable as anybody can do
MR. DAVIS: You could make an API of something nobody wants.
DR. SUAREZ: Creating some level of expectation or understanding that, while
this data is out there and is usable, its utility, its usefulness to achieve a
particular goal might be limited.
MR. DAVIS: If you could, would you send me an example of what you mean? I
understand conceptually, but I would just like to see an example. I am having a
DR. CARR: I am taking the chair’s prerogative to jump the line for a second.
I think one example – we have talked a lot about this. We think that we
will get to the number by having a construct of usefulness. I actually think
that let the data speak for itself and how many times was this data used in a
dataset could be one example of the usefulness. The frequency of using it, I
think, would be a starting point. That would be very helpful.
MR. DAVIS: I guess the thing that I am thinking of is don’t forget that
there is no – anything on healthdata.gov – predominantly, the things on
healthdata.gov are free and openly available. Part of our challenge, I think,
is an inability to even know where the data went and how it was utilized until
something like the Health Datapalooza comes up where people say, hey, talk
about me on stage because I am using this data in the following way.
My point is I think we have a lack of information about how usable it is
because we have a lack of information about where it is being used. That is
kind of by design. Take it. Do great stuff with it. I think, then – that
is kind of why I am pushing you for a specific example. I am not sure how we
could get to that given our current model of freely and open available and not
Structure versus unstructured, I don’t have a good answer for you on that. I
would love to talk with you more.
DR. ROSENTHAL: This is basically just fantastic. I am thinking when we
started this like two or three years ago. Now, when we say metadata we are not
doing kind of one on one what we mean. Now, we are getting into structured
versus unstructured. This is brilliant. This is brilliant progress. Health
Datapalooza is an example of that on a number of different levels, just from
doing something cool with an app, meaningful usefulness in addition to utility.
A couple different things, we can dig out some of the work – we did
this – you and I did this around kind of nine different ways you can
consider very quickly and inexpensively using kind of open source mechanisms on
.gov to basically generate kind of user feedback or user loops with typology.
You see it in direct to consumer stuff all the time, top down and bottom up.
Some of it is by counts, some of it by breadcrumbs, some of it by basic
learning center commentary around it. There are ways — the– are like
mechanisms that you can implement pretty quickly and easily for capturing some
of that stuff. I won’t bore you with the details. We have that document that we
can dig back up for you.
MR. DAVIS: I would appreciate that.
DR. ROSENTHAL: Kind of thinking about like utility versus usefulness. From
HHS, it falls into that kind of ranking by strategy for policy implementation
given the strategic directions and different functional vectors of what I call
top down of what you are doing and then also some other ways to capture some of
the bottom up and not just kind of breadcrumbs, but user clicks, user feedback,
basic rating. Ask your user base how useful was it, not just how easy was it to
utilize, but how useful was that.
If you think about doing basic learning center-type stuff and blog is a good
cut at that, but instead of just kind of going into Twitter where it disappears
in a stream some place where it is like archived with a little bit of metadata,
actually having the content metadata match some of this other top down or
bottom up metadata for the data elements, themselves, would be a really
interesting way to kind of connect that. We can go through the details of how
to do that if you want to see like examples of that.
The only other thing I was going to say was metadata, when we started this
two or three years ago we were saying, hey, what is metadata? Is it important?
Yes, it is for these reasons. Okay, can we retroactively kind of go through
current datasets that are important? Well, we should prioritize which datasets
are important so we know where to focus metadata efforts and then come up with
some sort of template standard typology. It doesn’t have to be UDS, but
something reasonable for new datasets – like bake that into API-type
machine readable language.
The other thing I think is kind of worth considering is how regular will
this stuff be? Now that we are kind of – we have moved past a bunch of
kids typing out an app at the Apple store, now that you are actually asking
entities to make investments, whether it is public/private or social good or
even just like market generation, if we had a set – like is it going to
come out again? What frequency? I know some of that is dependent upon
allocation and things of that nature so you don’t have clear answers into that.
After metadata comes what is the life cycle or repeatability of a data element.
I am not speaking for me. This is one thing that comes up very, very
frequently. A hospital compared, great, everybody goes nuts for it. Is it
coming out next year? Maybe it is. Physician costs. It makes headlines. You
want to move past outlier reporting. Do we invest in it? Is it going to come
MR. DAVIS: That is an interesting point. Yes, it does. It is one of the
things that we have also been considering is, quite literally, when you go to
healthdata.gov, you know, it will say here is this great data from 2010. Okay,
well, that is really awesome for back then. If there is a little bit more data,
I would be very, very interested in it.
One of the things that we have actually been trying to do is go around to
our various operating divisions and say, hey, listen, I see this dataset out
here and it has six years of data, but it seems to fall off in 2011. Can we get
the next three years of data? They have been very responsive. They recognize,
wow, we look kind of silly having a stream of data that is not actually up to
date and current. They appreciate us sort of proactively helping them along. I
like what you said about it is sort of inspiring confidence for you guys to
say, oh, I see this will be coming out again. This is a resource.
DR. ROSENTHAL: It doesn’t have to be the whole ocean, but particularly
saying what datasets are strategically important. Maybe starting with those. We
can go through the mechanisms we did, but they sort of answer Bruce’s question
and Walter’s question. There are pretty easy ways that you can do kind of
bottom up utilization and cut it by community or cut it by developer or cut it
by business or whatever you want to do. It doesn’t require a lot of curation on
your guys’ part. We can go through that at a later time.
MR. DAVIS: That is an interesting point. You have basically created the
cross-section between the strategic relevance of a dataset and its periodicity
such that you can create a more meaningful description of the data, its
reliability, et cetera.
DR. ROSENTHAL: You can tie it into metadata in that way. If you do any posts
– if you basically said for every challenge we do, if we are going to give
away some money for entities doing this, that, or the other thing, maybe part
of the requirement is you have to post what happened, what did you learn, how
you were using that data. How do you build a learning asset over time? You have
mechanisms at your disposal to be able to do that.
MR. DAVIS: That makes sense.
DR. MAYS: One is a comment and then I want to pick up on the NIH. One of the
things that is very exciting is to hear that you are doing data agreements with
other countries. One of the things I love to recommend is to consider doing a
data agreement with a country that has cradle to grave data. We don’t. The
ability for us to be able to look at someone else’s data on obesity or whatever
it is – and cradle to grave has unique identifiers so that they follow
them over time. It would be very useful, particularly for like the NIH science
because then you really have a much better sense of what it is that you should
zero in on. It would be very cost effective to be able to use another dataset
and then to do what we need to do in the U.S.
MR. DAVIS: One of the challenges that I think we already foresee in
utilizing some international data, though, is that there are different
thresholds for various things. In obesity, even if you just started at
childhood obesity, a child in the U.S. could be – I don’t know – 0-13
or whatever it is and then in the UK it could be 0-14 or 15. There are
different thresholds for what obesity is actually considered literally from
your BMI and body weight perspective.
It is important for us to consider. Obviously, there is the awesome and fun
consideration of what it would be like to have alternative datasets from
international agencies on healthdata.gov. We also have to think about the
actual utility of the data in terms of making it meaningfully understandable.
DR. MAYS: My other is about NIH. That is there is this issue in terms of
when you think about your SRDA. At NIH, they have – because it is
organized by disease, they have a very specific – what? It is like each
unit or – I’m sorry – each institute is kind of disease-driven. Each
of us that get funding from each institute, if it is at a certain level, we
have to develop our own datasets to make public. I am sure you have heard that.
We each have to spend money or find a way.
What NIH is starting to do, which I think is absolutely great, is that when
you put results out now, in terms of their big studies, they want the
infographics to go with it. It is like, okay, you publish this data in a
journal. That is great. They want people at a very different level to be able
to use that. Again, it is like where are they going to get to posit it besides
just on the NIH website? There is a lot of richness that is there, in terms of
what one can build on. Without any instructions, we each throw up our datasets.
We each do it our own way. We deposit them probably at the University of
Michigan or other places. I think there is a lost opportunity there.
MR. DAVIS: I would imagine – there is clearly going to be some
duplicative efforts towards data generation. This goes back to Walter’s comment
about data provenance. If you can figure out where data originated from, what
it was collected and curated for and then figure out if you have an alternative
use that fits within the confines of that data and its creation, you basically
could eliminate a significant portion of your upfront cost and effort in order
to simply reuse the data that someone else has produced.
I often like to say data can be – people look at data through a
different lens. You and I are going to see the same dataset from two completely
different perspectives. I think it is important for us to start to expand the
ability of the research or entrepreneurial community to say it is really great
you used that data for this, but it would really be awesome if I could
repurpose it over here for that.
DR. ROSENTHAL: On that line, has NIH even outside of .gov, given any thought
around kind of requirement, not just making data available, but even metadata
models, right? If you are working on a dataset, you have an ERD somewhere. That
is not PHI. In a direct consumer model, the first thing you look for is ERD,
right? It may not solve all your problems, but it is going to solve 80 percent
of them. It is not a top-down standard. It is like Walter’s working on where we
are going to like hammer this out. If you basically say this is a best practice
of an ERD for this type of data, that has tended to work really, really
efficiently in other markets. Is there any – back to the learning center
thing, that might be some of the stuff, not as metadata, but even ERDs. In
terms of kind of controlling it from finances, is there any thought of that
kind of strategically on the NIH side?
MS. NILSEN: Not that I have heard. It creates huge datasets.
DR. ROSENTHAL: Yes. If you are going to put out a set – you have done
it through legislation. Anything getting funding has to be machine readable. To
Bruce’s point, open data, data liberation 2.0, 2.0 means not just machine
readable, but like an ERD, right? It is on some analyst’s desktop somewhere. I
or someone else or a researcher has to literally recreate it from scratch. You
are wondering why it is like chicken wire. It is because literally I can’t
– any other industry I can go in and find standard ERDs all over the place
and not like top down, but what people tend to be using. It is so simple. The
work is already there. You can put it on a PDF, even.
DR. CROWLEY: I love this conversation. I think the learning center aspect is
actually right on in trying to put these assets together. I just have one quick
point for the ERD discussion. In addition, you need a data dictionary so you
know what the fields are, know what the fields mean in the context they are
being used. An ERD along with the data dictionary would be, I think, a lot of
DR. ROSENTHAL: Tie it onto funding. One way to affect that is by tying it to
funding. Another one is just policy. Literally, we made it machine readable as
a requirement, legislation. That is great for healthcare, but, I mean, that is
80 percent of the work.
MR. DAVIS: One of the considerations, especially for the new datasets that
are being produced, but less for those that are – so we have existing
contract vehicles and grants and things along those lines. Those things are
going to take a heavy lift to go back and say from this point forward, this has
to be machine readable.
Clearly, there is an opportunity for everything from this point forward to
say we need to have open data as a byproduct of this activity. It is also going
to require that there is some actual, specific language in the contract or
grant or whatever the mechanism is to say here is what we mean. It can’t be
open-ended or else ex-contractors are going to do the minimum viable product in
order to produce a dataset that needs to be further produced into something
usable. Thank you.
DR. CROWLEY: I would just add to that it also has to do with how they will
communicate, how they are making it usable for the community, and then they are
starting to compete on how usable they make the data usable for the community.
MR. DAVIS: That is a good point, as well. Okay, let me get through the last
few slides here. How much time do I have left? I am probably over.
DR. CARR: We tend to be unstructured. I know we have two presentations
today. I want to be respectful of that. We also are going to have a visit from
Karen Desalvo, I think, at 3:30.
MR. DAVIS: I will make just a slight adjustment. We have one – I’m
sorry. I didn’t see your comments. We have one presentation today. We were
going to have two. The schedule is further valuable than you might have
DR. VAUGHAN: Really briefly, to loop back into I think an earlier
conversation we have had, to point you to the really excellent work of the
Federal Geographic Data Committee. It has a great deep dive of public/private,
multi-agency commitment to metadata that has been – was a very long and
excruciating process, but has come out the other end with something that is
quite elegant and robust, to see what might be taken from that, rather than to
completely look at reinventing the wheel.
The other is to share with you a comment at a civic data meet-up that I
participated in locally where one of the technologists was very interested to
hear about – had comments about data.gov and healthdata.gov and had
apparently had tried to contact the data steward that was listed and found that
those were a lot of dead links. He has made it his personal project to
summarize all of them. He is going to share that with me. I will pass it along.
MR. DAVIS: Thank you.
DR. VAUGHAN: It points to, again, kind of what can we build into the
infrastructure of, sure, people are going to move on, but is there a more
general way of assigning how people in the future will follow on on those
MR. DAVIS: It is an interesting point. That goes back to a prior point we
were making. I don’t recall what the subject was, but the bottom line is sort
of reducing the confidence in the platform. If you know you are going to go
there, have an issue, suspect to put in a comment and nobody is going to
respond, why would you ever go back to that platform if you don’t think anybody
is paying attention? We do have to make sure and try to ensure that the public
email addresses and the contact folks are actually going to an email box that
is actively sort of taken care of.
DR. CROWLEY: Even with the response, having a structure, a certain level of
customer service for the community, maybe it is all responses are in 48 hours
or 24 hours, but enforcing standards of —
PARTICIPANT: Kenyon, it is really difficult to understand you.
DR. CROWLEY: — with the overflow models. Within the Open FDA link, they
have created this community of conversations about how to use – the
community has already started to create discussions around that. That is an HHS
agency being used in order to keep efficiency across the department.
MR. DAVIS: I heard about 75 percent of what you said, Kenyon. I am sorry.
DR. CARR: It is a little bit broken up. If you are on a speakerphone, can
you go on a handset? If you are on a handset mobile, can you go on a land line?
Turn the volume control down on your side.
DR. CROWLEY: Briefly, in order to facilitate this community discussion the
way that Open FDA seems to be doing and I have used this already is the stack
overflow model. Directly from the FDA site, if you have questions about the use
and functionability, everyone is asking questions about it. There are already
discussion starting about how to improve the experience, what data is
available. Since that is within HHS already, we want to keep consistency in how
different departments are creating the community discussions. For the data
assets, you might consider something similar.
MR. DAVIS: That is a good idea. One of the things that I took away from my
experience in ONC was we were trying to figure out how to build communities
around Blue Button and other things. We kept going round and round about how
are we doing to get people to healthdata.gov – excuse me, healthIT.gov at
the time. We realized it is not necessarily about bringing folks to us. It is
about going where people are so you can have the conversation there. If stack
overflow is a commonly sort of referenced or utilized platform through which we
can get some robust feedback, conversations, et cetera, then that, perhaps, is
what we need to do. I appreciate the comment, Kenyon. Thank you.
I am going to proceed with slides, otherwise, we may never get through this.
To continue with status, I am moving now to the educating new and existing
internal and external participants slide. Spotlighting the value of openly
available health data to health care transformation is in progress. This is
something that we have been working on in terms of thinking through how data is
being utilized in, say, the ACO models, for example, or how data is being
utilized, once again, through those entrepreneurs and innovators who made
themselves present at the Health Datapalooza.
The idea being really shining a light on the fact that some of these
innovations are, in fact, driven by open data. We need to tout that as a
We want to increase the percentage of machine readable datasets on the
platform. We have already somewhat alluded to the fact that we have tons of
PDFs and others formats, but we really do want to focus on the fact that these
datasets need to be made available in machine readable formats. People are
spending time, money, and other resources to try to take data that they cannot
just get. They are literally doing manual or some level of plain language,
automated processing to get data into some machine readable format. The fact is
that that is a waste of time because the data came from somewhere. Making that
data source openly available after appropriate privacy and security measures
are taken is incredibly important.
All of my Health Data Leads are being repeatedly asked to consider at least,
at the high level, what are you annual reports? What are your semi-annual
reports? What are the things that you already know on the calendar are coming?
Let us consider the process for making that output machine readable from this
point forward. That is a recurring refrain.
You can imagine it is great for me to say and for them to hear, but there
are resources that have to go into making that something that is going to
actually happen. However, that is a set of targets that we can obviously have.
We know that that annual report is coming out. There is no reason we shouldn’t
be considering the availability of that data down the line.
We want to continue to expand our external outreach. That is also in
progress. A great example of that is actually something that Jim Craver did at
the NCHS. He put together a Codecademy learning session on the data assets that
are available out of the health indicators warehouse. That is a great example
of going where the community is, providing an educational resource, and letting
people run with it. It is my understanding that we are going to sort of
evaluate how well that has been received. I would be interested in having other
operating divisions do something along those lines. We do have goals in the
strategic and execution plan for producing educational resources that people
can utilize in the same way.
We have also been trying a lot to be out on the speaking circuit. It may not
sound like that is a sort of relevant approach, but you would be kind of
surprised how many people don’t actually know that healthdata.gov even exists
or the fact that there are data resources there and available. When we start to
talk through this strategy that I am presenting you today more at a high level,
less at the granular progress report level, they are really interested. You get
a lot of questions coming back.
People sort of underestimate how important it is for the speaking circuit to
be worked, but it is a valuable resource because people are coming to those
conferences to walk away with information like what we have to provide. I would
encourage you, as you are out on the speaking circuit, to also refer to
healthdata.gov as an opportunity for folks to find resources that they may be
able to utilize.
We talked a little bit in the strategy about developing use cases and
internal marketing to enhance HHS as workforce engagement and to continue this
culture shift towards open data. That is not something I have had any time to
really focus on. One thing that I will tell you is that there is an easy sort
of “win”. HHS has an internal sort of – for lack of a better
word, social network, called Yammer, where, basically, there are thousands and
thousands of HHS employees who are engaged in this online platform. You can
literally say, hey, I need a PowerPoint expert for blah, blah, blah. Someone
from across the country or in another part of the division or right down the
hall will say I can help you, give me a call.
You get this really interesting mash-up of all kinds of people coming
together that you wouldn’t even meet at the cooler let alone in the cafeteria.
These are people that are geographically diversely located. My point in
bringing that up is it is an opportunity to point folks to healthdata.gov and
basically say here is one of the activities we are engaged in, here is a recent
blog about some things that are happening at CMS, blah, blah, blah. It is an
opportunity to sort of spread the word about a public platform on an internal
network. I think that it presents a real opportunity to improve communications
and awareness about what it is that we are trying to accomplish.
On the last bullet, I will just say quickly that the Idea Lab, as I said at
the beginning, has been thinking through sort of the different ways that we can
do the business of government better. There are a multitude of different
innovation pathways that the Idea Lab has created in order to foster innovation
– celebrate it, foster it from its inception, and then expound upon it
once we’ve decided that something is worthwhile to expound upon.
I won’t go into the various pathways. They are really cool and creative. I
would encourage you to go to the Idea Lab site and just read quickly through
them. One of the things that I realized was important is each one of these
might have an opportunity to produce some level of data. That data may be
valuable on the outside for people who have had similar ideas for innovative
projects. Introducing the tenets of the Idea Lab innovation pathways is
something that we have also considered.
Incentivizing and enabling the health data ecosystem, we want to continue to
publicize the availability of the data. I think this strategic goal probably
could be merged into some other ones, but it did fall into sort of
incentivizing the marketplace, helping folks to understand that data is
actually one of the main fuels for the changes and transformation that are
going to happen across healthcare and the delivery of social services.
We want to seek new ways to engage entrepreneurs, who may use the data for
fuel for their businesses and innovations. I would look to this group and your
networks to help us to understand some of these new and innovative ways that we
could be engaged with some alternative communities than the ones that we are
already engaged with. I won’t go into that any further, but just put on your
thinking caps as to how we could try to do some outreach and get some people to
understand a little bit better what the possibilities are.
We want to develop relationships and support the needs of federal and
non-federal data projects, data enclaves, data repositories, and innovative
test beds for more powerful analytic capabilities. It is often said that things
like CMS claims data are incredibly valuable. People have – you have to go
over hurdles to try to obtain access to the data. If you could actually obtain
that data in some way and get it into your own data enclave where you have
personally generated data from a cadre of people with FitBits and clinical data
from a core set of say Kaiser Permanente-type practices, you could actually
start to mash up some of those datasets and get some real meaningful analytic
information out about the population, about the way healthcare is delivered,
about all kinds of quality metrics, et cetera.
We are engaged in a process of trying to figure out the different
methodologies for how we could even begin to broach the topic of getting data
from HHS into external enclaves. Just know that this is something that we are
currently thinking through in recognition of the fact that it is not just about
some of the data that is openly, publicly available, but it is some of the data
that you almost feel you could probably never get ahold of that we want to sort
of find the pathways to yes as opposed to having the answer constantly be no.
Just recognize that we are interested in finding the ways to put data into some
of these external repositories in a way that makes sense for us and for those
I will close with the implementation of the administration and departmental
policies. One of the things that we set out to do was create a charter for the
health data initiative. That task is thankfully complete. I was very proud to
get Secretary Sibelius to sign off on a charter for the Health Data Initiative
before she departed the department. That was pretty exciting. The charter was
signed in May 2014. We now have a health data initiative that will go on into
the future. Thank you.
I am on the next slide, implementing administration and departmental
policies. Another thing is the implementation of the open data policy, M1313.
As I said before, we were lucky to be led out front by Todd and Greg and so
many other sort of innovative minds who were instrumental in creating the first
Datapalooza. We find ourselves in a pretty strong position in terms of the
implementation of M1313.
That is not to say we are perfect. We still have things we need to
accomplish. Generally, we have done pretty well in terms of creating structured
data output of the catalogue to federate to data.gov, for example. We have been
pretty at the discovery and management of datasets. We are expounding on how we
can make that activity even better. We are drafting plans to address the
impacts of the Holdren Memo on increasing access to the results of federally
funded scientific research. We want to understand the Holdren Memo in and of
itself, but we also want to understand the Holdren Memo in the context of the
M1313 Open Data Memo.
What we really are engaged in now is sort of a process of understand and
clarifying the relationships between all of these various memos and actions and
all of these things coming out of OSTP so that we can create some meaningful
and strategic approaches to how we are going to implement any one of these
along and how they sort of fold into and mash up with each other.
I think that is about it. I guess I will close with a final slide with other
considerations. I will just say the timeline for the strategy and execution
plan was tight. This was the first time I had ever writer a strategy and
execution plan of this breadth and depth. I felt like I could do everything by
1Q 2014. That is not entirely accurate. Needless to say, the schedule was
pretty tight. There is some slippage in some of the dates that you would see in
the strategy and execution plan.
However, some of the goals are further along than we planned them to be. The
data federation that I talked about with multiple other federal data resources
from the states and from other entities has gone gangbusters. We are well ahead
of the goals that we set. There is some positive news there. Needless to say,
though, we have some data quality and metadata issues that we have to hammer
out with those folks. We are in conversation with them. We have their data
federated, et cetera.
The time and resources of other participants are a factor. If we are going
to enable and incentivize the marketplace, well, it is not just going to be me,
Serena, and Greg doing it. It is going to be a cadre of other people. I need
the Jim Cravers and the Steve Cohens and the other folks of the world to be
engaged in this process of external outreach, et cetera. They also have day
jobs not unlike the folks in this room. We have to figure out the different
ways that we can make this happen.
There are lots of shiny objects. I will be honest with you, there is some
really cool stuff happening. Every time one comes up, I’m like, oh, we should
do that. As soon as we do that, the last shiny object that came up is now being
neglected. We have to be really, really conscious of how many things we are
actually accepting on our plate and just be okay with saying I would love to,
but I can’t right now.
Our Office of Business Management – I forget what OBMT stands for
– basically, brought us a data visualization project. We have been
thinking about a dashboard for publicly reporting the progress of the strategy
and execution plan. Those were two things that were not in the strategy and
execution plan, but they clearly need love and attention. You can just see
there are lots of interesting things coming along that are going to divert our
attention for various reasons.
Finally, we need to accumulate some of the methodologies for some of these
objectives. We don’t necessarily have the expertise for some of the things that
we would like to accomplish. They have to be done. We just – I am not a
metadata expert, for example. I am not necessarily an expert in getting the
metadata experts together in order to create some lessons learned and best
practices that we can quickly implement across the Department. It needs to be
done. We need to figure out how to develop those methodologies to achieve some
of those common objectives.
I just wanted to pour it out, some of the considerations when you look at
the strategy and execution plan, especially in consideration of what we have or
have not achieved so far, just recognize what the climate and landscape look
like for our small cadre of people. I will close there, take more questions if
you have them, but I really appreciate your time to offer an update as to what
has transpired so far. Thank you guys very much. I appreciate your
DR. CARR: We thank you very much. This is extraordinary, what the progress
and the escalation in the progress that has been made. It is very – it is
great. It is great to be working together. I guess we will just go around the
room, starting with Chris.
DR. FULCHER: Damon, that is great. I wanted to get to the point where –
we are consumers of data from the warehouse. I am wondering about creating a
feedback loop around – because we are collecting metrics, user-centric
metrics around every dataset that is being used to create a map, every dataset
that is being used to create a CHA report from HHS. I am proposing as part of
the experiment, why not provide those metrics back to you because you get a
one-time consumer hit on your site, but what we are getting is hits over and
over again? Instead of a BFF, we could be a DFF, like a data feedback friend.
MR. DAVIS: I like that.
DR. FULCHER: It is not just our organization, but Healthy Communities
Institute, Health Landscape. There are a lot of great groups out there that are
using your data and maybe setting up a protocol for that formalized, automated
feedback loop to you all would be something really worth exploring. Then you
can get a combination of your strategic relevant data assets and you can have
trending data assets. How close are they aligned between your strategic mark
versus the trending? Are they way out of whack? Are they close? I just wanted
to add to that experiment and offer maybe a test bed opportunity with what we
MR. DAVIS: I think that is a fantastic idea. I am so glad to have my first
DFF. I think that is a really cool idea because if you guys are seeing the ways
that you can be contributory back toward an understanding of how the data is
being utilized down the road, to go back to Walter’s comment, that is one of my
challenges is knowing even where it is going and how it is utilized. If you are
able to provide some level of feedback, then that is really fantastic.
Often, what we need is one example in order to get a whole bunch of examples
to pop up. Somebody else will say, hey, we can do that. I would be very
interested in pursuing that further with you, Chris. Thank you.
DR. VAUGHAN: I would also amplify that by suggesting follow on with some of
the API companies, the Masheries, the Keen IOs, who are actually kind of
pulling out some of the datasets, but also doing some pretty sophisticated
analytics on it and, again, give you a much better idea of what they are
seeing, in terms of how it is being disseminated.
MR. DAVIS: That is a great idea.
DR. VAUGHAN: Another opportunity for this public/private partnership.
MR. DAVIS: I would love to take some connections from you.
DR. VAUGHAN: I am delighted.
MR. DAVIS: Leah gave us a connection, to go back to the data federation, she
was in contact with Puerto Rico. We had a conversation with Puerto Rico earlier
this week. I mean it when I say I am interested in these kinds of – these
connections. We will follow up on them and make them meaningful parts of our
DR. GREEN: Damon, I want to add my appreciation for your time. I have two
questions. One goes way back to where you started quite a while ago. If I heard
you correctly, you said you could benefit from a working definition for
strategically relevant datasets?
MR. DAVIS: That is correct.
DR. GREEN: I’m sorry – data assets.
MR. DAVIS: Data assets. That is right.
DR. GREEN: Just say a few more words that further refines and defines what
it is you need.
MR. DAVIS: The strategic plan is publicly stated. It has multiple tenets to
it. To be honest, arguably, the data assets that we produce out of any one of
our activities, arguably, should be strategically relevant. Why are we doing
something if it doesn’t directly relate to our HHS strategy? That could be one
I guess what I am saying is the strategically relevant data asset is another
way of saying high valued datasets. It shouldn’t be star.star. It shouldn’t be
every single asset out of every single activity that we do. It should be at the
very least, if we could have folks focus on specific data related to strategic
plans steps one, two, and three, here are the two to three datasets that we
think would be most relevant in this domain.
It is a means of sort of streamlining or focusing folks on some data assets
versus, I guess, trying to boil the ocean, if you get my meaning. Was that
helpful at all?
DR. GREEN: Yes. One follow up on that is high value to whom?
MR. DAVIS: High value has a lot of different meanings. As I said before, you
and I are going to see the same dataset from two completely different angles. I
think there is a space for the public to indicate what is high value to
“them”. I think there is also a strong opportunity for those who
collect and curate the data from the inside to say this data is valuable in the
following ways – it is supportive of these initiatives or it can be
correlated with these other datasets, such that you start to see its strategic
relevance because it is potentially “core” to some other activities
or it is such a comprehensive listing of data points.
I think it could go a bunch of different ways. I have not necessarily
refined what that definition would look like, but that is something that I’m
looking for the Health Data Leads to be contributory in, at least from the
internal perspective, of here is what I think is strategically relevant for
these specific strategic goals and why. It may be that we turn it around and
ask the public what is your opinion of these various datasets for whatever
I don’t have a specific approach as to how we are going to accomplish this.
That is part of the reason I am interested in the feedback from you guys.
DR. GREEN: What you were just speaking about a moment ago, I was quite taken
when you said the Data Federation was going like gangbusters. What is your
thinking about what accounts for that?
MR. DAVIS: It is interesting. I think it is a lot of different factors. Part
of it is – so at the federal level, we were mandated – I mean, if we
weren’t standing here at HHS and we were at some other agency that did not have
a Todd Park or other type person there, they are now required to do this. They
are back in 2010, I guess, when this whole thing started here, in terms of
where they are.
I guess what I am saying is there is a forcing function, at least at the
federal level. Acknowledge the data, its collection and curation, and manage
the data across its entire life cycle. Think about open data as a byproduct of
the activity that you were about to procure an IT asset for or for the grant or
for the research or whatever. There is that.
I think that there is the open data community that has just grown. It is not
just health. It is all across the nation in a multiplicity of domains where
open data is actually fuel for all kinds of varied innovations. I saw a really
interesting example about how someone is using USDA data and historical weather
data to allow farmers to get crop insurance based on the conditions that are
predicted for the year. Don’t let me mess up the example, but the idea is that
open data has been transforming all kinds of different domains.
As that community has grown, I think so, too – there has been an
additional growth in those that are interested in making data available. There
is this growing demand, but there is this growing supply where – there are
governors across the nation that are saying our data will now and forever more
be openly available. You are not going to get a group of governors together
without a little bit of competition. I can guarantee you that when one governor
says one thing, another governor is going to say we are going to do that, too.
It is states and localized entities.
I think it is just becoming something that people are demanding more and
more. It has become more and more of something that can be accepted by the
public that you can put data out that is not necessarily something I would be
uncomfortable with having the public utilize, especially when you have the
examples of how it can be utilized for the greater good.
I think part of the reason the open data initiative has moved so positively
forward is because we have so many examples of the great stuff you can do once
you can get your hands on the data. You can capitalize on your ideas and make
things happen. Those are not official. Those are my opinions as to what has
DR. FRANCIS: This goes back to Walter’s question about the Datapalooza and
privacy this morning. We are all aware that, as data become increasingly
enriched and as analytics get more powerful, the possibility of identifying
individuals gets higher. You did mention – provenance is one way of trying
to trace what happens to data after it leaves your hands, but I am wondering
whether there is any thinking, any resources that might be helpful, about
follow up on the privacy risk side.
MR. DAVIS: When you say follow up, can you be more clear?
DR. FRANCIS: If you release a dataset and it is meant to be let’s say a
public use – it is a public use dataset, which was constructed in such a
way that you thought no one would be identifiable from it. It actually turns
out that when it is used as part of an enriched package, identification takes
place. Have you thought about how to figure out whether that is happening? What
to do to try to mitigate the possibility of that happening? What could be of
MR. DAVIS: I have learned a lot in this space. Opening caveat, I am by no
means an expert. I have had the opportunity to talk with a lot of different
folks about what our privacy and security measures look like across the
Department, especially from the open data perspective. I get the impression
there are about 18 different data fields or something like that that you can
remove from a dataset in order to de-identify it. That is the standard.
Off the bat, if you are not revealing sex, zip code, name, date, et cetera,
et cetera, et cetera, you have already started from a relatively significant
place of providing some level of data security because you have clearly sort of
taken away the immediate identifiers. It is my understanding, too, that people
are doing all kinds of things, in terms of perturbing the data and swapping
data fields and all kinds of other stuff in order to further de-identify. You
know the woman who appears in the age range 35-39, basically, they are changing
her age, for example, in order to make her a different age and, therefore, not
directly identifiable as plotted against other datasets.
It is my understanding that there are a myriad of different activities based
on how sensitive the data actually can be. Again, I think that the value of the
data from a public use perspective is so high and our data privacy activities,
I think, are pretty strong such that I don’t necessarily have a concern with
that. Part of that, I think, is because each individual operating division
actually has its own robust set of privacy and security measure that it goes
through before making the data publicly available.
There is an attestation in the back end of healthdata.gov that basically
says you have gone through the processes for your operating division and
compliant with HHS to say this data is private and secure based on the things
that the Department and our operating division have outlined. Every one of our
datasets has that privacy and security attestation in it. Each one of the
health data leads, before promoting the dataset to the Idea Lab for promotion
to healthdata.gov has to make that attestation. I guess that is about all I can
say from my non-expert perch.
DR. FRANCIS: Just as a follow up, some of the methodologies used to protect
individuals from identification actually make the data less usable, some of the
perturbations. My question really comes from the joint goals of having the data
and having it be appropriately protected. One way to think about that is to try
to actually figure out what the risks are of different kinds of datasets being
released in different types of contexts. That is why I ask about what kinds of
follow up. If the data just go out there in the big, bad world and you never
know what happens to it, you are not going to be able to think through how much
do we have to perturb, for example.
MR. DAVIS: It is a good question. It is a major challenge because you have
the desire to make the data as usable as possible and identifiable. You also
have the desire to keep people’s PII protected. You can’t do both. We have a
major challenge in that way.
DR. CARR: My apologies, but Kate has a hard stop. We can come back to this,
but I want to give Kate a chance to talk about Health Indicators Warehouse.
MR. DAVIS: Thank you, guys for your time. Over to Kate.
MS. BRETT: My other hat is being a team commander of a deployment team. We
are on call right now. I am sending people out. We have a call at three. I am
going to go forward. I am sorry.
So I just wanted to start out by saying thank you for inviting us to give
this presentation. It was something that Justine had seen Jim’s presentation at
the Datapalooza. She thought it would be a good idea to show it here. He is
unavailable. I am here. Hopefully, this will give you a little feedback on what
is going on with the Warehouse. I know you all have heard about it before. I am
not going to go into a lot of detail, but sort of talking about new features.
Just as a really high level overview, what the Warehouse is is a large
structured database of high quality curated data and metadata. I think that is
where we are putting our efforts. It is about population health. It is about
kind of the big picture information on health, healthcare utilization and
access, social determinants.
We have been pushing a user interface that allows people to actually see
what is in there, but it is not beautiful. It is here is what we have – an
ability to chart and map, data permitting because not all of the data are at
lower levels than national, so that you can, again, see what is there and then
a real push with our API to make the data open and machine readable.
What it isn’t is a data tool. It is pre-tabulated data. You cannot go in and
look for your particular interest point if it is not already there. It doesn’t
generate tables. It is not designed for specific target audience. In fact, what
we are trying to do is pull data from those specific target population
databases and put them all together and all in the same format. I am kind of
the data person in the background. I spend a lot of my time pulling data in and
reformatting it so that it is all the same. We are not trying to replace
anything else that is out there.
Right now – we keep saying we have over 1,200 indicators. We are
pushing – we are closer to 1,300 right now at different geographic levels.
That is where I will say I feel – I would love to push more local and not
put as much effort into our national datasets. I also work at the National
Center for Health Statistics. We do national data. It is a push/pull there from
a lot of different data sources, obviously, not all from us, federal, some
NGOs, some trade organizations like the AMA and the AHA. We are pulling it
where we can.
We, basically, are representing five initiatives or programs. This is how
the – I am sure many of you have heard this – this is kind of how
this warehouse started. We were providing data for different initiatives. We
thought there is no purpose of doing a data run for one particular initiative.
Let’s make it available for everybody, make it machine readable.
We have done a number of different things. We are getting data from CMS, the
community, and utilization and quality indicators, the Community Health Status
Indicators, which is a project that start out being very important, kind of
went into hiatus for a little while, and is coming back, county health
rankings. New to our list is the Healthy United States, which is an NCHS
product. We are pushing the machine readability of that publication, which,
currently or until now, was only available in PDFs and Excel spreadsheets, and
the Healthy People 2020 Initiative.
The day before Datapalooza – we did this last year so I don’t know why
we keep doing this. We released a new version. It is actually 2.0 so it is a
big change for us. It was a refreshed look and feel. It was a refreshed way to
use the data. We updated the homepage design by improving some of the pathways
to indicator selection. We realized what we started out with was not useful
– no helpful to most users so we got rid of some of the selection criteria
and pushed people into a certain way of finding indicators.
We introduced a space to highlight relevant sites and sites of interest and
sites – we are specifically looking for products that are using the data
and highlighting that. We started out – I think we have five or six now.
We are hoping to add some more.
Again, Healthy United States, I can’t tell you how painful that publication,
which is a required publication from NCHS – they came to us and said we
need to make this machine readable. Help us do that. It is not produced in a
way that makes any sense for anything other than PDF. It is a nightmare. We
have 38. I would say, yay for us, we have 38. We are going back in time and
getting those 38 indicators. We will keep moving forward, but it is slow.
We put down Septicemia Deaths for 2010. That sounds like it is ancient
history, but one of the things we are playing with is, okay, we have this
process for getting mortality data in. Can we now expand so that then maybe we
cannot just have the ten leading causes of death at the top level, but maybe we
can move down to the top leading causes of death for both men and women or the
ten leading causes of death for Whites, African Americans, et cetera, et
cetera, race, ethnicity. That is going to – that will help generate some
interest in the data. We are working on that. The Septicemia deaths were a
response to CHSI. It helped us realize we can do a lot more with this.
As a comment, yes, we are only at 2010. 2011 is about to be released any
second. We have already got the data in the pipeline. We have analyzed it
already, internally. We are waiting for them to release it.
Natality, we are up to 2012. We have put that data in. BRFSS, we have county
estimates. Actually, we made this arrangement with BRFSS folks that we would be
their county estimate provider to the public. They didn’t have, at the time
– I don’t know where they are going with that because they are creating
their own model data. We are going to use whatever it is that they create. They
might be producing it, as well, for themselves. We are, again, having this
machine-readable format that is the same for all the other things, all the
other data that is in the system. Hopefully, that makes it more useful. Some
other stuff from Census, American Community Survey, and Bureau of Labor
As we get the data, our push – we are a very small team. We lost a
person. We are about ready to hire somebody. As far as data goes, it is me and
as much data as I can create and push into it. The sooner we have more staff,
What is new with API? In February, we released version five. We now have a
registration key that is required. We did that so that we could go back to our
users. We now know who the users are. We can say what are you doing with it?
How is it helpful? Can we do something better for you to make this a more
useful system? There is no – we are not putting a burden on anyone as far
as the keys. It is just a way for us to hopefully do some evaluation somewhere
down the line.
There are three applications, currently, that are making up the Warehouse:
the indicator selection section, the developers section, and then the services,
the data, itself. Those are three applications that are running. As I said,
version 2.0 released – did a lot for the API. We have a new “Try
It” section, which I am going to – I don’t know if you can see that,
but it allows people to actually go in and kind of create code and on the fly,
do some testing of rest services. It gives a lot of information about how to
use the API.
We did a lot more – we did some work to help people with the workflow
and the documentation section is much more complete. We have included some
– we are going to include the Codecademy information. It is still not
released yet, unfortunately. There is still some work on it.
I just wanted to say a little bit about responses to some of the thoughts
about the Warehouse, in terms of usability and other factors. As far as
usability goes, we do allow – provide links from each indicator to other
indicators that have either the same keyword or same topic area. We do –
the charting is pathetic. The map makes Michigan look like some ridiculous
state that doesn’t exist on the planet. It is open-sourced code. We weren’t
trying to make this very nice. It was just here is the data. Is it what you are
looking for? Great. Go and take it and use it somewhere else.
Again, this new links to sites and projects where the data is being used, we
are hoping is going to help people go in and get – be creative with the
data and say, oh, if you can do that with the content, then maybe we can do
I have released an indicator report that is in both PDF and now in Excel. I
was really frustrated with the PDF where it is all in alphabetical order. It is
like, well, how am I supposed to find everything with the word STD in it, which
is at the end of the alphabet, but it is actually spread out all over? There is
now a much better way to find – to download the whole set of indicators
and play with them. It also allows – on the Excel spreadsheet, it also
gives information on the geography that is available. You can say, okay, what
are all the indicators that are to the county level? By the way, our CMS data,
by the end of the month, should be all at county as well as HR level.
We have put in an application for an evaluation staff member to work with us
and try to do some evaluation. That is key to increasing the usefulness of this
site. We have all been struggling with how to do evaluation. It is not on a
federal – it is hosted by our contractors. We can’t use the CDC
mechanisms. It is a little tricky. We will work with this fellow and hopefully
get something going with that.
Just quick Google analytics information. The average user goes to 4.4 pages.
We are keeping people at least a little bit. They spend an average of about
three minutes per session. Our bounce rate isn’t great, but there it is. We can
keep on getting better. Our numbers – our page hits are increasing slowly.
Again, we are going to keep looking at this and see how we can improve.
As far as data documentation, we have published the ERD for both metadata
and the data. It is now out there. The metadata, we spent a lot of time on. We
have been working really hard. Our initial – the initial use of metadata
on the download files was horrible. We spent a lot of time with the contractor
fixing that. I am still – I still struggle with better ways to improve
that metadata and, in fact, the first thing that the new staff is going to be
working on is really working hard on getting that cleaned up. It has been a big
work in progress.
Timeliness, now, that is where I will say is our weakness. We are a
warehouse. We publish only after the data producers have released data. We are
not producing the data, ourselves. We are taking it from elsewhere and putting
it into the system. What we have done is created a lot of programs for repeated
computations so that once we get the data, we can reformat it and get it into
the system as quickly as possible. We are at the waiting end of how – we
can only put the data out there as fast as we get it. We are not creating it
That is about it as far as this presentation goes. I really wanted it to be
short and sweet and hear your comments and questions, which is where I really
want to get some information from you all.
DR. COHEN: I am a former member of OAEP in my past. I will be kind.
Actually, it has to do with when we think of strategic relevance, the two
things that folks want from data are granularity and timeliness. The meeting I
just got back from, we were talking with NCHS folks and with state folks about
what we can do to make, starting with vital statistics, our data much more
Clearly, there is policy oriented folks who think the data that we generate
is too old to use. We were discussing provisional data, both with the mortality
folks at NCHS and a bunch of state folks. I am really optimistic that there are
ways that we can generate much more real-time data that is representative of
the geographic units for which we want the data that would really reduce the
time cycle significantly.
This is something NCHS had done in the past, the concept of 12 month ending
data. Now, the states and beginning as a whole and individually to focus on how
we can get data out, what is good enough. I really think this is the next
frontier that we should all be focusing on around releasing data more timely.
2010 is not a good year to be releasing data for, even though vitals – you
know, Dalton promises death data through 2013 will be available by the end of
the year, still, there are ways to get the data even more timely.
I think many of our data collection systems are fine-tuned enough to be able
to generate solid quality information in a more real-time way. We have just
never really thought about the need to do that strategically. I think that
should be an initiative that we all embrace.
MS. BRETT: I would definitely agree that we are way behind the times as far
as the data. I will – from one side, I can say, again, we will put out the
data when it is given to us. I can’t, from my seat, do much more than that.
There is provisional data that is released, but it is not released in a way
that we – it is released in a tabular way to everybody. While we could
probably release that data machine-readable, it is at a pretty high level. I am
not sure how useful it would be. That is a good place to start and think about
what we can do to push ourselves forward.
I was just at another meeting before this hearing that, yes, we are catching
up with the vitals, but starting then the following year, the money will dry up
and we will be going backwards in time.
DR. CARR: I want to be respectful of your hard stop. If you need to go
MS. BRETT: I can go another five minutes or so.
DR. CARR: Can we pull it up on the screen? I am guessing you can’t do it on
DR. ROSENTHAL: For what it is worth, I use your stuff all the time, very
regularly. The progress on metadata is just mind bogglingly good. It is like
best practice. Publishing the ERD and like Bruce said, like, yes, latency is
one issue, but kind of in the realm of open data, what is interesting about it
is that I know you are going to have next year’s data up there at some point.
If I am building a system or a product or service or making an investment, you
release hospital, you release physician compare. Does it come out next year?
Does it come out three years from now? I don’t have any visibility into it, but
I know your stuff at some point is going to come out there. That gets to the
reliability. Is it worth the market investing in that. That is why we started
with stuff out of the gate and seeing the evolution of that. It is pretty
MS. BRETT: Great. Thank you.
DR. CROWLEY: Quickly, I want to say in terms of the issue with trying to get
meta-tagging and other curation of the data and pushing the ball forward with
that, it might be interesting to look at different ways of using the community
to actually do that moderation. There are a lot of health students out there, a
lot of data students, data scientists, a lot of people interested in this, but
they are looking at some sort of model that sort of merges community-based
feedback through that curation with moderated data resources at the agency.
That is all.
DR. CARR: Did you change phones? We could hear you very clearly before and
less clearly now. If you have another option, it would be great.
MS. BRETT: So there are two ways to get to the data at this point. There is
the topics. This is where I would love to hire a librarian who could help me
with really categorizing data. That is one of my things that we are going to
fix. The other way to do it is to search.
DR. CROWLEY: Can you hear me a little better now?
DR. CARR: Yes. That is much better. Thanks.
DR. CROWLEY: I just had a quick comment on the meta-tagging, curating, and
cataloging of data. Looking at ways to leverage the community after the
platform to help with some of that curation could be a way to sort of enhance
the resource as applied in partnership with an agency resource. There are a lot
of health students, data students, other people in the community that would be
interested in doing that. There are sort of methods you can put in place that
could help push some of that curation need to the community.
DR. CARR: That is great. Do you have some of those students, Kenyon?
DR. CROWLEY: I have some. They are all over the place. I don’t know if
anyone has seen the CDC’s Micro-tasking promotional site, which created this
sort of gamified platform to have public health students help them with all
kinds of agency data tasks, other types of activities, both administrative and
operational. It has really caught on. There is a lot of activity. I would be
happy to forward a link to that at some point. It would be useful.
DR. CARR: That is great. Thank you. What you are showing us is septicemia
per 100,000 and then it breaks it out by predicted categories.
MS. BRETT: By predicted categories. This can be looked at – because
this is, as it says on the top, national, state, and county data, you can then
pick a state and it will show the state data estimates. If you go to the
counties, you can actually see the county estimates. I am not sure what I was
trying to show here, but they are there. This, right now, is the county —
DR. CARR: So this can be exported? How is this exported? To Excel, did you
MS. BRETT: Excel or CSV or machine-readable to the API.
DR. CARR: So the way that you decide – I think it gets to what Damon
was saying before of what would be – what would be topics people would
want to see? How do you decide what is worthy of indicatorship?
MS. BRETT: That is a good question. We have been continuing to struggle with
this. Our goal is we have somewhere on this – I will find the page later
– that says propose a topic. We are looking to hear from other people what
it is that people are interested in. We have a governance structure to whom we
will bring the proposed ideas to. There is some criteria that they have to be
publicly releasable. They have to somehow be representative of the geography
from which they come from.
We haven’t done anything that is not national, at least, or that doesn’t
cover all the nation. At some level, we are open to, say, the California Health
Interview Survey and including information from that because, certainly, there
are people for whom that would be useful. We are hoping to get people to –
frankly, we thought we were going to be overwhelmed with people telling us all
the data they want us to put in. We have been completely overwhelmed with
DR. CARR: So a way that might be helpful – Damon was saying earlier
that that concept of things would come to HHS, but now reaching out, we have a
population, a community workshop coming up in October. We ought to make that
connection of what we hear there that we can transmit to you of things that
people wish they had.
MS. BRETT: That actually would be very helpful.
DR. GREEN: Thank you, Justine, for making that connection. I like your
glossary under your sources. I learned there that DNA there means Data Not
MS. BRETT: It depends on where you live.
DR. GREEN: What I would like to ask you to just say more about has to do
with something I have heard for the last two hours about questions about the
urgency of data release and the timeliness and all that sort of stuff. I want
to hit the pause button on that because so much of the utility of these data
actually is related to the question compared to what. So often, the comparison
is, well, that is what it was in 2014, what was it in 2004? What was it in
Many of the Triple Aims issues require time series analytic-type work. The
big matter question is are we making any progress on lowering cost and
improving population health and changing healthcare. What is the thinking about
how the Health Indicators Warehouse allows connectivity back to the old data?
MS. BRETT: That is, I think, you could say that given our resources, we have
to choose to move forward or go backwards. We are kind of doing both. For
instance, the mortality data is a good example. We have individual years’ worth
of data only from 2008 through 2010. We actually started with 2007 data and
dropped it because at the end of a decade, you have to go back and recalculate
based on new population census data with the intercensal estimates. It is a
real pain in the neck.
Hopefully, by the end of the year, we will have 2011, 2012, and 2013 data.
Then we will have a trend. Is it further back? No. That is where I think
resources are going to – the question is what is of more value to –
or the most value to the most people, going back in time on the indicators we
have, moving forward and adding new indicators, or just staying afloat with
what we have got? Our budget, unfortunately, is only as big as it is. We are
constrained by that.
DR. GREEN: We all have our own opinions. My opinion is that makes me sad.
MS. BRETT: It makes me sad, too.
DR. GREEN: Seeing Paul stand up over there, another thing to consider is not
taking – pulling entire datasets into the data warehouse, but being very
intentional at a selected number of parsimonious measures that are stratified
by different users and potential audience. By parsimonious, I am talking about
ten – five, ten, that could be derived from datasets that are never going
to be available on here, but for which this health indicators warehouse could
provide some benchmark measures. That would take resources – that is not
– that should not be terribly expensive, it seems to me. That seems like
something that could be done.
I am not doing this just for intellectual interest. It just seems to me that
that would be a real pull to this Health Indicators Warehouse. If there is
someplace where you can go where critical measures are already listed about
what they were at several points in time, guess what, if you keep going through
our website, you can do work to find out subsets of those that interest you and
that sort of stuff.
DR. CARR: I think this ties back to the SRDA. I am hearing sort of a theme
that there is so much to do. We are trying to hone in on can we do one thing
before we do everything, to quote our chair. What is strategically relevant? Do
we try to hone that if we are trying to manage resources, depth versus breadth?
DR. ROSENTHAL: It doesn’t have to be a dichotomy. It can literally be a 2×2.
Whatever the most strategic are, you go deepest and then you expand out.
MS. BRETT: I think you could do both in a way.
DR. CARR: You mentioned this several times that you are resource
constrained. I think if we have a little bit – if we have a lot of things
that are a little bit useful or a smaller number of things that are powerful,
we want to think about how we prioritize.
MS. BRETT: I will add that some of these data, say the two-thirds that come
from Healthy People don’t require any resources on our end. We are going to
keep putting them in. I would say some of the indicators that we have on, say,
how many states have a smoking law about pool halls may not be very useful, but
it is a Healthy People objective. The data come to us freely and already
formatted so it is easy.
On the other hand, there are some – I think we could, in fact, identify
some strategically identified number of very important indicators and make sure
that they get more attention from us. That would be good.
DR. CARR: We are appreciative of you accommodating our time. I am sorry that
we ran over. We appreciate this orientation. I think we have seen it before,
but seeing the new and improved and also helping us to think about what you
have was very helpful.
MS. BRETT: Thank you.
DR. CARR: Thank you. My suggestion is we take a break, but get back here
before 3:30 when we are expecting Karen.
DR. CARR: Most people saw this this morning. I am trying to think –
Josh, you are the only one. Because I am so fixated on all these visuals, I am
consistent if nothing else. I think as we are thinking about what we heard this
morning, the wonderfully exciting agenda that Damon presented and, I think, the
resonance of the things that we have been discussing, we are really – you
know, it is important to keep us grounded, to think about what is the spectrum
of things that we should be thinking about. How can we help?
One of the things we wanted to do is just revisit, if you could – if
you can stand it – the letter we wrote to the former Secretary about
changes that we would like to see in the use, utility and usefulness of data. I
think – we touched on a lot of those things this morning or a little while
I think there were a lot of things that we could say, but there were –
so, as I look at what we said – the steps to improve usability, use, and
usefulness of selected online HHS data resources.
One was the user-friendliness. I think we have seen that in the health data
indicators. I noticed that she gave us the data on bounce rate, pages visited,
and time. I think that we are having more page visited. I think we were at 1.3
at this letter. What did she say, four pages visited and three minutes? Bounce
rate is still high at 52 percent. It is helpful to ground us in that. I think
that the recommendations that we had in many cases resonate with what is
The second thing was use of data, data documentation, and metadata. Again,
that sounds like work that is work in progress. Ultimately, usefulness of the
data, timely availability, which we just discussed.
One of the things that didn’t make the final cut of the letter that Kenyon,
in particular, had been very articulate about is worth talking more about. We
will have it reflected in the minutes even though Damon isn’t here. It is just
the learning community that brings together – that is why I put this slide
up here. We need the top down. We need HHS. We need the communities. We need
the customers to know what the needs are.
The approaches – as we say the approaches, we are talking about data
generators, data aggregators, miners, integrators. Chris, I think, you do a lot
of this. Then the concept of the validators. I think that has come up a couple
of times. I worry in some way that you can juxtapose a lot of data, draw a lot
of confusion or conclusion, choose one or both. That is an emerging important
domain that I think we need to be paying attention to.
I am just looking down at the bottom. As we talked this morning, what are
the tools, the apps or the tools? And I put indicators because I think that is
a tool. We can mine the data. We can think about what is important. The Health
Indicators Warehouse can be a tool to quickly get what you want for
communities. As we went through it today, it is better than it was, but it
still needs work, I think just even as Kenyon mentioned, the resources that are
available with students or whatever to do some of the data organizing, et
At the bottom of this slide, then, is engagement and collaboration. I think
the work upcoming on the October roundtable is going to be generative for these
kinds of things. I think if we think about who are the people in the data food
chain, we won’t overlook or we won’t miss out on that.
Perfect timing. Welcome. You are welcome to take Damon’s seat at the front
of the table. This is great.
DR. GREEN: Let’s welcome Dr. DeSalvo. She doesn’t need much of an
introduction, but to be sure everyone is on the same page, she is the next
National Coordinator for Health Information Technology at HHS. It is a very
small job. She is the past City of New Orleans Health Commissioner and a senior
health policy advisory to Mayor Landrieu.
Examples of the sorts of work I know her for are very hands-on increasing
access to care for vulnerable patients, for IT solutions for emergency
preparedness. She graduate from the Tulane School of Medicine and the Harvard
School of Public Health. The most important thing I want to say about her, she
is actually a really nice person. We are so delighted that she is here. You are
nice to put up with all you have been putting up with in the last 24 hours and
still include us.
Karen, we want to listen to you and to learn from you what your aspirations
are to some extent and consider how NCVHS might enable you to be successful.
DR. DESALVO: Thank you for that wonderful introduction. Larry was on the
advisory committee when I was a Robert Wood Johnson generalist faculty scholar.
That was such a formative part of my life. I appreciate all the mentorship that
he and the other members of that group gave. I am indoctrinated with the
important notion that health is more than getting people to a doctor. It is
about where you live, learn, work, and play. I know Larry has done so much work
in that space. It has been great mentorship. I think maybe I should stop there
because Vickie looks really happy with what I said.
Thank you all for accommodating my schedule. As you probably heard, we ended
up spending a little extra time in Minneapolis, which is, I think, the nicest,
friendliest city I have ever been to. I love it. If I ever get stuck anywhere,
I was delighted for that. I appreciate you making time.
I wanted to just give you some sense about where we are at ONC and where I
see the field of health IT and how we are working to evolve ourselves and in
partnership with the community to go forward. I would be remised if I didn’t
start by thanking this committee historically for inventing ONC. I have had the
rich opportunity to know a lot about the history of the ONC, read the report
when John Lumpkin was chair, meet with him, and speak to him, personally, about
it. I talked to a lot of the original folks.
The reason I start there is because ONC’s roots are in public health and in
community health. A lot of the language in the original thinking about the
importance of standardizing data capture around people’s health and health
care, thinking about how it could be put to positive use to improve their
lives, and the ways that ONC was conceived as a coordinator, as an opportunity
to bring together the federal partners, but also bring together the private
sector partners to really think about how the data for a fifth of our economy
could be put to – captured and put to really good use is something that
speaks to me as a person who is not only trained in, but spent time in local
public health. It is just great to think about how we can go back to some of
those roots as ONC and begin to broaden our focus.
When I got here, we were at this pivotal time at ONC that we had been
intensively focused on the ARRA funding and on the Meaningful Use program for a
few years. Very appropriately so, it was billions of dollars and a major ramp
up in programs and spending and culture change in the country. The efforts
around Meaningful Use were very successful.
Stage I has brought widespread adoption of electronic health records. At the
hospitals, it was particularly successful. I think we all understand why. They
have bigger departments and, generally, budgets. The bigger you are as an
organization, the more likely you can adopt HIT. Providers, I think you saw
some numbers this morning from Judy. Two thirds or more of providers
participating in Meaningful Use if they are eligible. Across the country,
really, the majority of doctors’ officers and clinics using HIT in the form of
electronic health records.
In exchange, in some pockets, some aspirational models in our country are
doing extraordinarily well. There are some places where it is more nascent. We
have a lot of work to do to connect the EHR systems. Again, a great deal of
I think some real clear examples across the country, even at the state
level, where the promise and power of health IT is being brought to bear. I was
in Minnesota. I think that is a terrific example of a state that has been very
successful in a collaborative fashion, Maine, Kentucky, parts of Oklahoma.
There is really, I think, an opportunity for us to look at prototypes that are
working, but then also remember that we have to raise the floor for everybody.
We are coming off of the – I have said it so I will just say it to you
all – the adolescent years and the big budget of ARRA. It is really an
important time for ONC to reflect on our roots and to think about the future
and to think about how we become more efficient and effective at what we do.
That means, internally, as an agency, as an office, that we are working on
reprioritizing our work, doing deep inventories to understand where we have
work that may be synergistic together, where we may be doing things that don’t
matter as much anymore because it has been successful or because it is not on
the high priority list, and realign the way that we are running the agency and
communicating with each other, flattening the organization because we are very
matrix and entrepreneurial. The communication through the silos that grew up
quickly over time made some of our work a little challenging. We are morphing
and evolving in that.
Also, in some cases, like in Judy’s putting the portfolio of nursing out
front as the Chief Nursing Officer and changing the name of that group to the
Quality and Safety because that is what they had been doing. They are not the
Office of Chief Medical Officers. Some of it is labeling, but reflecting more
of what we are actually doing as an organization.
While we are doing all of that, it is also time for us to refresh the
federal HIT strategic plan, which is one of our responsibilities and to work
with the partners in the federal government and beyond on a really high
priority that comes up all the time in the environmental scan, which is
interoperability. It is an interesting push and pull. What has happened is with
all of the success of IT adoption, we now have some new desires and challenges
that you wouldn’t have had if you weren’t capturing the data in the first
For example, on the challenge side, the user interface, the desire for
doctors, nurses, and others who actually interact with the electronic health
record isn’t as powerful and strong as we would want it to be. I think most
people would want to see the IT systems evolve at the front line. On the other
hand, there is a lot of data that has been captured. Freeing that up so it
connects care for patients so that it is inclusive of new types of data,
patient-generated, genomics, proteomics, wearable information, I think, is a
We see that it is possible to collect. We want to make sure that it is free
to be put to appropriate use for care and science and beyond and public health.
I think there is some really – I think that begs other questions about
privacy and security and the need to make sure that IT systems are enabling and
supporting care transformation and healthcare delivery, but also making it
safer and better quality.
The interoperability piece is a tie that binds, if I could describe it that
way for the committee. It is not the only thing that we are thinking about, but
it is a way that we can weave together the work of ONC and the call to action
that I hear currently from the private sector, from our partners at the VA, the
DOD, the SSA, federal employee health benefits, and many others. It is one of
those really fun challenges that is solvable because people have solved it.
There are places, as I mentioned, in the country where it happens.
It is not perfect, but we have shown that the technology and privacy and
security and governance and sustainability issues have some – there are
some really good ideas out there about how to work it. What we haven’t done is
woven those together, those successful places and/or brought up the floor for
the places that haven’t come along.
Honestly, we haven’t gotten even into some of the granular areas that I
think we are going to have to go. There is this big governance conversation
about how is the data going to move, how is it going to move with equity and
with an opportunity to make sure that it is appropriately used not only for
care, but for broader population, public health, and science purposes. As you
get really deep into the standards piece, we have some standards we haven’t
standardized. It is, I think, preventing us from having the fundamental
building blocks to really do things like quality measurement, not just quality
measures, to have an opportunity to share a set of data that meaningfully
informs care and can help identify people most at risk from a population level.
Unfortunately for everyone at ONC, thinking, you know, there is this really
big frame, but there is this really deep dive we have to do. I think that may
help you all understand, also, why we are trying to really focus on
interoperability and get ourselves and all of our efforts to some end goals,
which we are beginning to outline.
We did, in the area of interoperability, put out a white paper last week. It
is really meant to be an invitation to anyone who might have an interest in
seeing that information is captured and shared and put to use. It is meant to
provide some direction around some areas that maybe there has been some
fuzziness. For example, that query response is a part of the portfolio of
interoperability and that Direct isn’t the only place that we are going to be,
although, it may have use cases. We have to consider new types of data, not
just electronic health record information because it is coming. It is already
there. There is not really a nice way for it to fit. We need to be inclusive
and thoughtful about the patient’s voice and vote in this data, but other
sources, as well. We have to be able to take advantage of our existing
infrastructure, but begin to evolve in such a way that we have a robust
structure that is flexible and adaptable to future needs.
We begged some questions about governance and sustainability and some of the
other technological pieces. We are concurrently working internally in ONC to
align ourselves around interoperability in those building block areas. We have
laid out five. We can share this paper with you if you want it.
Then working, also concurrently, with our FACAs as a first pass of getting
some subject matter expertise, but back to the public health vein, for our
interoperability work or our national priority setting around health IT, I
think it would be a mistake if ONC set those priorities or even HHS without
getting out into the world. I have learned in every job I have had, but
definitely this one in the last five months, it is easy to get preoccupied with
the fire burning in front of you or the tree right there that you are dealing
with. It is one thing – I think we can look at the forest beyond. We can
help ourselves do that to think about planning for an interoperability today
and in the future, think about what the bigger set of priorities are around
health IT that encompass interoperability.
If we really want to understand what is that vista beyond those mountains,
we need the community to help us see. I need to know what the innovative folks
are thinking about in the market. I need to know what potential use cases are
that we are not really considering. We don’t have to invent all of those. We
don’t have to have every solution. But we have to build a platform and a set of
rules of engagement and think about how this is a public good and what kind of
sustainability model we are going to have that allows it to be as inclusive as
possible going forward.
That is the kind of work we are doing. It is foundational. It is meant to be
a platform upon which things can grow. As we move forward in this, we just
wanted to get ourselves a little restructured and aligned to do this work
again, this work that you all set out for us years ago of being a coordinator,
a convener, a policy maker, and this is concurrently – we are doing this
within ONC, with our federal partners, and we will be ready to start engaging
much more with the private sector in a structured fashion as we go forward to
make sure that we are not – don’t have blind spots and that we are being
as thoughtful as possible and as inclusive as we can.
I will stop there and maybe take some questions. I can probably talk for an
hour. I guess I should say, explicitly, thank you all for creating ONC so I
could be here to do this really fantastic work. It is really exciting. I just
– the energy around data in this country – let me say that properly.
It is energy around data in some parts. It is energy around information for
others and energy around knowledge for others. In all of those spheres, I think
there is just this really great excitement about the potential and the promise.
Folks are using their imagination in exciting ways. I just want to make sure we
are protecting, as we should, people’s data and we are creating the right
technology platform that allows it to grow and flourish.
DR. GREEN: We have a habit of people putting their tents up. This time, just
put your hand up if you want to ask a question. I will ask the first. Walter
Suarez just had to leave. He asked me to ask you this question. What can we do
to align the work of the NCVHS Standards Committee with the ONC’s HIT Standards
DR. DESALVO: That is a great question. Next week, at our Standards Committee
meeting for ONC, you are going to – we are going to open a dialogue about
focusing the work of the S&I framework to this point about – there has
been a lot of optionality and experimentation. Can we get ourselves focused on
a path forward respecting all of the work that has been done?
Engagement, I think, in that public conversation that we have I think it is
next Tuesday and we will find out for certain the date, would give some eye on
where we would like to see the conversation go, in terms of that S&I
framework. I will tell you directly that I have a particular interest in
understanding if we, all of us, can come to a common – a set of common
data elements that would be the fundamental building blocks for the things that
I don’t know much about what your standards committee has worked on, but my
guess is that everyone has some set of things they are using, but they are not
exactly overlapping and, in some cases, less aligned than we might have even
thought. I will use gender as an example or sexual orientation and gender
identity or race and ethnicity, but you could even think about blood pressure
and the ways that that is measured and registries in CDC and other places.
It seems really simplistic, but if we – if the standards and technology
people do continue to believe that is the right way to go, I would like to get
very focused on that, meaning not inventing a lot of new things, but just
saying, gosh, if these are the things that really matter, could we just come to
some agreement that we are going to measure them all and collect them all in
the same – a data dictionary, Larry, is what I am trying to say. Whoever
is the furthest along and has the best option to offer, I think, is what we
ought to start think about grabbing. It will reduce a lot of confusion for the
marketplace and the vendors. That is a tactical thing to work on, I guess, but
hopefully, that answers your question.
MR. DAVIS: One of my thoughts coming out of Datapalooza and thinking about
what you said about interoperability was along the lines of the sort of
entrepreneurial community, the innovators that are utilizing public resources
of data for their various innovations, but, often times, they are very, very
interested in how they can in any way interface with any EHR product that is
I wonder is ONC thinking through, as you used the term, interoperability,
the fact that it is not simply the EHR to EHR, but it is also sort of, for lack
of better words, after-market products or innovative interventions that are
coming in. Can you say a little bit about what interoperability means with EHRs
and the entrepreneurial endeavors that are out there?
DR. DESALVO: Yes. This interoperability to ONC is much more than EHR vendor
to EHR vendor conversation. I think we would really be missing what is
happening in the world if we just thought about that. In the white paper, we
point to the Jason report as something for people to look at as a potential
model and evolutionary path. Part of the reason for that is because it speaks
to the idea of opening architecture and generic – or, rather, APIs that
allow for more flexibility in querying of data, but also the user interface or
exposing of the data – you know, what is the performance layer on top.
I think that is the kind of approach we want to take. I said it before and I
want to underscore it again. If we only use EHR data to understand the health
of our country, we have really missed something. Health is not just about
healthcare. I think for this body and even for us, we have a responsibility to
have a broader view. We absolutely have to have on and off ramps that allow for
that with the appropriate authorization, privacy, and security, et cetera. It
is the future. Frankly, it is kind of here.
DR. COHEN: Hi. I am Bruce. My day job is with the Massachusetts Health
DR. DESALVO: I used to work there.
DR. COHEN: You did? I am in the Bureau of Health Statistics, the head of
research. I am very excited that you want to reinvigorate the public health
focus of ONC. I was just wondering what your thinking is about what the key
public health data streams that you want to incorporate into this new effort
DR. DESALVO: So two responses to that. One is it is not just about –
the public health element of this is about the tools of public health, broadly,
the approaches to problem solving and inclusiveness of community, the
epidemiologic approach, the really being thoughtful about the questions you are
asking before you just start mining data, that kind of thinking.
The second piece of it, though, is the informing of the public’s health.
Having come from local health department where we had really no data – it
was pretty stale by the time we got it. It was almost impossible to understand
in real time if we were making a difference in the community’s health. That
opportunity to provide a window into both chronic disease and communicable
disease, amongst other issues, is powerful and important and must be done.
so far as important from the public health community and I don’t want to step
on that because they are pretty aligned, it is a mix of immunization records,
lab reporting, some surveillance, particularly around important areas like
– whether it is respiratory infection or sexually transmitted disease.
Some of you all may know I have a particular interest in disaster preparedness
and response and recovery and survival.
DR. COHEN: In that order.
DR. DESALVO: Yes, well, you have to survive all of it. That is why I sort of
threw it at the end. At the end of the day, the opportunity for clinical and/or
claims data to help identify people most at risk in our community from whatever
that is is an obligation. We want to work with them on that.
I think, also – this is such a great – first of all, IT is, by
nature, multidisciplinary. Is that the right term now? When you go to meetings
about IT, everybody is there, including in Minnesota yesterday Commerce was
there. I love that. It is a different view on the world, bringing different
ideas. It helps form a common language.
I think that IT is also then a great platform for public health and medicine
to start to really talk again and solve some problems about health, which is
something a lot of folks are working on, but it is a wonderful, I think, venue
for that. I would like to advance that idea, as well. Plus, medicine has all
DR. ROSENTHAL: What is beyond the vista? What does success look like in five
years? You ask anyone in IT what do you want to do? Interoperability, sure. You
can spend years doing that. It is a huge hurdle. That is a step towards
something, I am assuming. What is that a step towards in your vision?
DR. DESALVO: Thank you for asking that. I probably should have mentioned it.
I think that if I were to prioritize the use case piece, I think that
nationally, we have a nation priority and an obligation to reduce the cost of
care, but only in such a way that it actually improves quality and then
ultimately improve health. The reason I put it in that order is because the
cost pieces, though good right now, could go up again. The more money we spend
on health care, the less we can reinvest in not just public health, but all of
the social determinants. We are making tradeoff choices.
IT, interoperability, not just because you see the CAT scan result and you
don’t repeat it or make the person miss work again to go get it, but because
you can change delivery not just through telehealth, but through asynchronous
communication and different models of care that we haven’t even figured out how
to pay for yet. IT is just a really powerful way to improve the care system,
which frees up dollars to help us apply in other ways, which are going to also
There are some concrete, definitional metrics that we are working on that we
speak to a little bit in the white paper, but we are going to get a lot more
concrete with folks about how we will know we have been successful. What is
fascinating is there is not really – even around the country, it is the
use case piece that people grab onto about whether we were successful, not just
the movement of the data. That is great, but in some ways, if you just get
focused on moving the data, you could declare success as just faxing something
if you are moving information. The success is also – there is success in a
data dictionary we can all agree to and you can move the little bits from place
DR. ROSENTHAL: Can I ask just a quick follow up question. That is what
everyone would ask. You have hardcore metrics and use case and that is great.
When you lie awake at night and say five years from now I hope I am wildly
DR. DESALVO: If my husband doesn’t yell at me when I go home about his
electronic health record. We had like this Mary Matalin/James Carville level
fight on Capitol Hill. I had to tell the waiter – I’m like no, really,
it’s okay. We are just talking about his electronic health record and how much
he loves it.
I am sorry. I am being silly. It is not actually that silly because what he
is saying to me is there are expectations of entering data into a system that
doesn’t have the usability that I want and I am not getting information from it
to help me take care of the people I am trying to help. It is also not –
he wouldn’t necessarily think of the bigger frame, but as a doc, it is just
– it is not bringing it home for him. It has to be – he had to move
away from it just being a Palm Pilot to actually being something that is
knowledge and moves the dial. We will get more specific as we go along.
DR. CARR: I don’t mean to jump the line, but I have been saying that all
that we are doing is great. My role as Chief Medical Officer for Steward
Healthcare, 2,700 doctors, private practitioners. They have the same question
and trying to move forward. We will be increasing your numbers at the end of
this month for Meaningful Use II. Raj Sanderaj, who is on our full committee
and is a private practitioner cardiologist in Las Vegas, he often has a
different point of view than policymakers and specialists around the table.
That voice has to be addressed. I am very – I think this is very excellent
that your husband is there to help you.
This was my summary of Datapalooza, but we started with the top down –
we used to talk all about the apps.
DR. DESALVO: He is becoming quite famous. He is always asking me to quote
him accurately in his description of his electronic health record, but I told
him that wouldn’t be professional. I have to use other words.
DR. CARR: It just sparked my imagination because it adds to your credibility
unbelievably. That is not in any way to take away from your credentials, which
are extraordinary. Fantastic. It is wonderful to have you in that role. When I
am thinking about Raj and knowing you have somebody who is boots on the ground
working this through, I think it is very appealing.
DR. DESALVO: Let me share with you also that – without going into the
long story – I have purchased and implemented three electronic health
records in the clinic environment and purchased an electronic health record for
a hospital that we were doing in New Orleans just before I left. I left before
we had to implement so I left that work to other folks. I have used them. I am
still active, though, not in the last few months. I have some sense of it, but
you are exactly right. It is not the same. I am not making my living trying to
interact with an electronic health record. That is a different game all
DR. FRANCIS: I am Leslie Francis. I am at the University of Utah. I co-chair
the Privacy, Confidentiality, and Security Subcommittee of NCVHS. Also, in that
capacity, I have been our liaison to the Tiger Team. I am part of the Tiger
Team for the Policy Committee.
I think we have done a pretty good job of getting a sense of what each other
is doing. One nice thing that happened was the Accounting for Disclosure
virtual hearing last fall, where our group participated with ONC. Some of what
has happened is most of the work on electronic medical records,
interoperability, and privacy questions, that is all at ONC. We have been
focusing much more on public health and data availability and privacy
questions, data stewardship questions. We did a letter about a framework for
thinking about stewardship and we are working on a toolkit for communities when
they use data to be appropriate data stewards.
I would appreciate any thoughts you have about whether that sort of feels
like the right way to have laid things out or ways that we could better
interface with ONC on the privacy and confidentiality side of things.
DR. DESALVO: Those are a couple of great comments/questions that you made.
Let me just start by saying, like most doctors, I probably didn’t have that
same legal frame and appreciation for security and privacy, but I have been
very educated by Joy and her team, really rich resources, plural, there. I have
a – I think a sense in my stomach that data is moving fast and the kinds
of data that we are compiling around individuals and otherwise is beyond the
bounds of what we have rules to well define.
To your stewardship question, I am not sure I know exactly what you mean by
the term, but that kind of gets to this issue of who is hosting whose data and
for how long and for what purpose and how do people have control over who sees
it and for what purpose. Lots of really important questions. It is a major
priority for us is what I want to say. It is a place where we have strength and
depth and experience and passion.
I would love to work with you on that. To Walter’s question, maybe we need
to have some way that we have an offline sit down and just understand how
things are aligning. I mentioned it at the outset that ONC is undergoing some
structural realignment. So are our FACAs. I want to make sure we don’t get a
click off in such a way that we mess up some relationship there and figure out
what that new piece looks like so we make sure we stay connected. Thanks.
DR. GREEN: Let me build off of that. Terri reminded me just a minute ago,
there were some statutory requirements we both have to address. We need to
harmonize that. Do you want to be more specific, Terri?
MS. DEUTSCH: I am Terri Deutsch. I am the Senior Technical Advisor at OESS
and CMS. I am also the lead staff for the Standards Subcommittee. One of the
things that you mentioned earlier is the need to be standard, have
standardization where everyone is doing the same thing. In the Standards
Subcommittee, we hold hearings and we hear from the industry and we make
recommendations to the Secretary so that we can do that, so that we can have
the same things happening and people using the same products.
One of the statutory requirements are a review committee that we look at the
standards and the operating rules that have already been issued and have a
venue that we can look to see if they need to be corrected or a new one done
and, hopefully, lessons learned that we do any new ones or we recommend any new
ones that we are not repeating any of the mistakes from the past.
I think there are many opportunities that the different subcommittees and
NCVHS and the fact that we have the statutory requirement, we will be
contacting you. We are waiting for the Secretary to do the assignment. There
are very great opportunities to make sure that we are all in alignment and
would appreciate anything you can offer to help us in that direction.
DR. DESALVO: Great. Consider it done.
DR. GREEN: I don’t know if you used sit down as a technical term, but that
sounds like a really good idea.
DR. DESALVO: Yes, talk. Let’s talk.
DR. STEAD: I am Bill Stead from Vanderbilt University. I appreciate your
comment early on about the fact that we had both the back end problems, if you
will, the interoperability and standardization pieces, and the front end
problems. I would like to know what your thoughts are on the front end side a
From my perch, the electronic health records, where we have done the best at
getting information together, the patient has actually become lost in the data.
We don’t have user interfaces that present cognitive support in a
context-sensitive fashion. We have also tended, as we have deployed this
systems, to rigidify workflow actually as we get information, we want to
change. I didn’t know what your thinking was about how we got, if you will, the
kind of tenure vision you have laid out around interoperability around
cognitive support and workflow redesign or anything like that. Is there
something like that in the vision?
DR. DESALVO: That is a great question. I would have to say that I think it
is implicit, but it has not really be something we have talked deeply about
yet. I appreciate you raising it. Clark’s third law, right, said technology
sufficiently built is like magic. That is not where we are right now with
health IT and EHRs. This is a new thing I learned since being the coordinator.
The reason I raise it is because it is the most – I use the phone as
the example all the time. It is a story about Brian. We went to dinner. We were
trying to decide if we were going to eat inside or outside. Brian said, well,
let me check my phone and see if it is going to rain in the next hour and see
where we should eat. It just dawned on me that if someone had said something
like that in 2001 even, you would have probably wanted to check their whatever
– thought that they had lost their mind. It is just commonplace now. It is
magic. It is not a phone. It is an enabler. It is a way you help make social
decisions in the moment. EHRs are not there, but they have come really far.
You touched on something very important that I want to just grab, which is
that standardization is very – is great. There are things that we should
standardize. There are data point like potassium that make sense, but the
context and the narrative of somebody’s health and life, you cannot standardize
that. Patients and clinicians don’t want to lose it. I think that is right. It
is a balance that we have to try to achieve.
I believe that changes in technology – so if we drive things like
opening architecture and exposing even generic APIs and if fire(?) is the thing
that we think it might be as a technology and can really advance the interface,
the user interface and accelerate that, if ONC creates the right platform, the
right regulatory environment, that can flourish.
It is going to really advance the kinds of products that are available and
the way we can modify products beyond what we have today. We have been trying
to be really thoughtful about what is the right place on the dial for
regulation to make sure stuff is safe and secure and we are advancing, but that
we are also not getting in the way of the innovation. What I am trying to say
is ONC is not going to solve the user interface issue or the workflow issue. It
is going to get solved in the environment. We have to make sure we are allowing
for that to happen.
DR. STEAD: And incentivizing it happen.
DR. DESALVO: That is right. There are lots of dials and lots of arrows in
our quiver that we can use. We have to make sure we are thinking about where
DR. STEAD: Thank you.
DR. MAYS: One of the things that I want to try and figure out is you are
hearing the themes around the table of all the different things that we do. In
the workgroup, in particular, we are kind of designing ourselves to be able to
move nimbly, move quickly, and be able to be a resource. I am not sure the best
way to find out how we can be helpful to you.
We can give you briefing information about who we are. We can give you our
kind of what we have done. When you said to Larry about talk, what is the best
way for us to figure out – because you are reconstructing some of the ONC
committees. Part of like when – you know, I said I would be the workgroup
chair, that was one of the things I was concerned about, some of the
duplication that is there and making sure that we could align ourselves so that
we knew what you were doing and we aren’t working in parallel, but instead we
are working in some integrated fashion.
If you can give us a sense of how we can best do that as you start to
recreate so that we have alignment, it would be great. We are also kind of
taking off in a pattern where we are trying to be helpful to the Department.
DR. DESALVO: I am sorry I am stumbling a little bit. The reason is because
though I know, generally, about the work that you do, I don’t think – I
honestly didn’t see all of this interconnectivity until we were just talking. I
apologize for that. It’s a lot of moving parts. It has to work. There is too
much work to be done. There is not enough time or money to go around for us to
However you all think is the best way to make sure I am up to speed and
educated with my team – we can get a little deeper about some of the
realignment and what is happening with the workgroups, make sure that we are
clicking, and there is so much to be done. We definitely won’t be able to do it
all. I want to make sure that we are thinking about what we need to do
You all have a brand and a stature in the health and health care world,
also, that would be great to tap into if I could be so bold. ONC doesn’t have
that same audience. I think understanding – the frame, also, about health
broadly and data agnostically, it would be extremely helpful to us. I think in
some ways, people, when they think of ONC, they think about electronic health
records. We want to be about much more than that. As much as we can work with
you to get there, we would appreciate it.
I don’t know. The first thing I will tell you is make sure you know Ayame
Dinkler. She is my Special Assistant, who makes sure I do the things I am
supposed to do. She will make sure that we find a good way to follow up.
DR. GREEN: Debbie Jackson, would you raise your hand? We are underway. I
invite you, until we prove ourselves unworthy, assume that we have no conflicts
of interest. We are not trying to stake out any particular territory. We would
like to meet our statutory requirements. Vickie did a very nice job of
describing where we are.
This committee is virtually on fire right now around the theme of helping
communities become learning systems for health. We think that alignment with
you and ONC is critical.
DR. VAUGHAN: Thank you so much for joining us. I appreciate all of your
remarks, especially around place and health with my JIS hat on. HIT is an
opportunity for public health and medicine to reinvent and refresh some
enduring and critical – mission critical issues and challenges. I
especially want to touch really back to getting out into the world.
I had the privilege of sitting down earlier this week with a group of
California safety net providers. Safety Net hospitals, representatives from
throughout the state. They chose me – oh, you are going to D.C. – as
the person to carry their message. I am going to try to honor that.
We talked about some for the EHR challenges that they are having with very
scarce resources, feeling like they are getting very, very much overrun by
vendors. Beyond that, trying to think about how to really steward the data that
they are getting ready to generate, let alone carry what they already have
Given those limited resources, but passionate, deep commitment, they are
trying to think about talking with each other, which is not necessarily always
what they have done very well. That applies to so many Safety Net hospitals
across the country. Just to, perhaps, put forward the notion of convening those
really critical providers as more people come on with ACA and to the
Medicare/Medicaid space, to pool their challenges, pool their wisdom, and pool
their opportunities for solutions. I think it is just a tremendous opportunity
for public health and to really improve the outcomes for communities that
challenge by all sorts of competing social determinants of health.
DR. DESALVO: Was that a softball question? Thank you for carrying that
forward. In New Orleans, for the last nine years until I got here, my work was
around helping to build a clinic safety net in a city that hadn’t had any and
bring those providers together to both cooperate and compete. They were
disparate entities and, in fact, advanced the number of Federally Qualified
Health Centers, which are typically competitors, but they helped each other
They shared data in a RIO information exchange and are all on electronic
health records, all living off barely a couple nickels and some friends. We
were like cleaning our own toilets to get through the day, but we really,
really wanted to advance the care of the community. It is called
I will just stop there because I probably shouldn’t even tell you more about
that. Ayame could probably give you more information so that you know the
rules. I was on the Board. Great group to think – they looked at other
places around the country like Austin, ICC, the Integrative Care Collaborative.
San Francisco has this place in country they should look to to find out how
they do it. That is one thing.
The other part I would just share with them is, depending on where they are
in their payment model – this is an innovator, public health, and safety
net and person-level goal, this project that I want to do, which is so little
Johnny has asthma. Johnny is in and out of the hospital, in and out of the
clinic. Nobody thinks he is getting his drugs. Mom is not following through.
You know the negative Johnny view. Well, technology has a way that on top of an
inhaler, you can drop a little GPS thing to know where little Johnny is when he
is using his inhaler and how many times he is using it at what time of day,
which is really helpful for the clinical record for the doctor to say, oh,
well, he really is following through. Maybe we don’t need to escalate the dose
or maybe we do need to escalate the dose or adjust the meds.
Imagine if the local public health department – and providers are
worried about it. Maybe little Johnny is uninsured or maybe he has run out of
Medicaid visits or he has run out of whatever. He is worried about that strain
and they don’t have access to their clinic because they are clogged. What if
all the little Johnnies and Julies in the community – you are a mapping
person. Roll all that data up, blind it into a map, and you give it to the
local public health department, who then goes and says, wow, all of this is
happening at this same – I will pick on a school – this same school.
Why don’t we just go work with the school and do an environmental assessment
and understand that, actually, they have an infestation problem or a mold
Individual providers can’t know all that data. The public health department
doesn’t get that data because it doesn’t exist, but it is technologically
feasible. It could reduce some of the burden on providers and it would help
– public health has the tools and the relationship to actually advance
care. Little Johnny might get better. It may happen one day in this country.
Right now, in this country, I don’t know, but it all exists. If it could be put
together, maybe they could do something concrete like that and think about
ways, not necessarily with asthma, but I just happen to know it exists. I get
excited about it.
I would also say that, at this point, it is in my DNA to advance equity and
to be inclusive. I grew up in public health clinics. That is where I got my
care. I am a doctor today because of the National Health Service Corps. They
funded school and I did my payback. When I am thinking of the world, I am
thinking of the world and the tent is being inclusive. I am thinking about
raising the floor. I am thinking about everybody comes along. There is not a
divide. Data and information is so democratizing.
Everything that we are building and planning and thinking about at ONC is to
do it in such a way that it is inclusive and that it is sustainable and that it
is affordable. I don’t mean just the cost of care, but the pipes and everything
else. You all can help me keep straight with that, too, and that would be
DR. CARR: Thank you. This has been phenomenal. I think it laid the
groundwork for the conversation going forward. We just admire what you’ve done
so far and your fortitude for taking this on and really believe in your
capacity to make a difference. Thank you so much.
DR. DESALVO: Thank you guys, really, very much. I appreciate it.
DR. CARR: So next steps. A couple of things, I think we are close to
wrapping up at this point. Damon, was this meeting helpful for you?
MR. DAVIS: Yes. First, for just gauging initial enthusiasm for some of the
activities that we are going to be engaged in across the Health Data
Initiative, but also in recognizing where we now have an opportunity to
continue the conversation. I envision these future meetings being more focused
conversations on varied aspects of the Health Data Initiative, in my opinion. I
would love to be able to bring things like the strategically relevant data
conversation here or other efforts.
DR. CARR: That is great. The next meeting of this group is in September. If
there are things before then, obviously, just reach out to Vickie.
The second thing is we didn’t quite get to go through the Health Indicators
Warehouse real-time. I guess we need more technology to make sure that we are
connected here. I do think that we need to just keep looking at what we keep
talking about and sort of do a little use case. I would love to see the FDA
data. We will get to see SAMHSA/CMS data. If we just sort of tee up those, one
is probably enough, one or two. We meeting in September, November, February,
June are the scheduled meetings. As I said, we can make other ones.
The other area that I wanted to just touch base with with leadership from
NCVHS Full Committee is the intersection with the working group. There is a lot
of very exciting agenda planned for the year ahead for the Full Committee. How
would you see the working group fit in? Let me start with Bruce.
DR. COHEN: What we were talking about this morning and what we are talking
about this afternoon are in the same place. I see the conversations being
integrative in a variety of ways. Our focus this morning, Damon, was on having
data providers meet communities where they are at, in terms of data engagement.
I feel we have a lot to say, both from the community perspective and,
hopefully, helping data providers like HHS figure out dissemination strategies
to provide information that is useable for communities. This is just building
right on your strategy. I feel the work in totally in the same spot.
DR. CARR: I think – you were out of the room briefly, but when we were
talking about the resources – well, you came back in – for how to
prioritize, we have this community workshop coming up in October that we ought
to be carrying that thread along with all of the activities of and what would
this mean for data, data availability, data prioritization.
Leslie, I wanted to say in your leadership role in Privacy, what do you see
from the Full Committee perspective with the working group?
DR. FRANCIS: Well, it seems to me that it is really important to keep
thinking about what are the – I am going to call them technical, but also
the kinds of policy questions that we need to think about as data availability
goes forward. I am deeply committed to achieving both. My greatest fear is that
if we somehow screw up on the data availability side, yes, you got it. The
example that just comes up over and over and over again in my brain is the
Texas and Minnesota debacles about blood spots, which was an enormous loss of
tremendously important public health potential resource.
I worry – the dark side of me is really scared that in the enthusiasm
of getting it out there, there are going to be baby and bathwater problems. I
can’t design the technology, I can just keep pushing the question.
DR. ROSENTHAL: This is one of the things when we were going to go around the
room in reaction to Damon, this is some of the stuff that came up in the
earlier material last year. I put a couple quotes from some of the Comp Sci
people. Even at the HDI tech track, I brought in the Harvard and Hopkins people
on the comp side, medical management.
Part of that session was geared around this. Very specifically, number one,
there are some things worth looking at, not purely synthetic files, but there
are very specific comp sci-like, very common usage, like in sensory device, et
cetera, et cetera, where you can present mosaic effect, forward thinking, and
retain statistical integrity at a designated grain.
That is something we haven’t really talked about so much, but that stuff
exists. That was the core of that session. What you are talking about, there
are ways to do that. That might be worth chatting a little bit about that. We
did a bit of that last year.
Then you could do something totally crazy, too, if you wanted to think
outside the box. Again, I keep saying like I want to donate my data for
research. I would love a green button where I can say have access to it,
basically, waive any and all rights. As I toy around with this stuff, there are
a lot of other people who you might be able to create – my point is there
are ways to kind of handle this, both from the technological and on the policy
I think the core piece of it – if I can say two things, like to Damon’s
piece, the public/private stuff, what we were talking about, getting feedback
from the users, I sent you some stuff on that. All of that is interesting, but
I think you basically do it yourself, internally, relatively easily. There are
very specific things you can do around getting like assets and having people
comment on it in a very structured way, at least easy technically.
You don’t have to do sort of external people’s analytics. That is great, but
you want to start with your core assets. Then if you do that, you can do some
very interesting things around that, part of which is like use means of control
to require other people to do the stuff, whether it is funding – so if you
are granting funding, let’s see the ERD. Or actually doing something besides
machine-readable. I see machine-readable is what has been accepted and worked
pretty well as being something like somewhat separate than a published ERD,
I would propose that is something we should strongly think about, another
piece of legislation around machine-readable is great. It is like 2005. There
is some specific stuff we should be doing on top of that.
DR. CARR: Welcome comments, suggestions going forward.
MS. NILSEN: I am really excited by the idea of weaving all of this together
because I think – I sit in multiple groups. I see it in one place and
another and another, but they don’t end up hitting. People say, well, now this
isn’t happening and then it is. We are not utilizing things as well as we could
across disciplines, across skills. I think this is where it could be exciting
in a way that nowhere else is able to pull things together.
DR. GREEN: Of all the problems that beset the U.S. health care system,
principal among them is fragmentation.
DR. CARR: What I was saying is where are the intersections? That is what I
was asking you. I think we have heard a tremendous amount today. I know we have
some great note taking.
DR. GREEN: Terri and Jim and Debbie, they are on the case right now. They
are the people at work on sorting out what our obligations are and how the
Secretary wants to resolve this.
DR. CARR: So I am leaving. I am very delighted to have been a part of the
first two years. I see a very, very exciting collaboration emerging now. I
really wish you really the very best of luck. I know it is going to be
tremendously successful. I am going to turn it over – but, wait, it comes
with one of these.
DR. MAYS: Kenyon, are you still on?
DR. CROWLEY: I am here.
DR. MAYS: Do you want to comment?
DR. CROWLEY: I think this was a really good meeting and very exciting. Thank
you for all of your work, Justine. I am looking forward to Vickie being her
leadership. I was actually in your neck of the woods yesterday.
I agree with everything that has been said. I think the progress on multiple
fronts, but HHS out ahead, in terms of data, is wonderful. We just have to keep
pushing on all of these things and increase the usability, the effectiveness,
and the efficiency for the entire stakeholder community, including developers,
researchers, innovators, the whole gamut.
The communications plan, which was referenced earlier, takes more shape
– you have to sort of reflect on that from sort of business school
marketing perspective. I think communication is going to be key to how you
market it, not only how well you build it. If no one know about it, it is not
going to be ask useful.
I appreciate, Justine, you bringing up earlier the – your pushing on
the learning community aspect. That is key, too. Most innovation if you can
facilitate meaningful interaction across these stakeholders and make sure what
is coming out of the system gets shared back into the system, it creates a
feedback loop of knowledge development.
I want to cross-reference the idea about sort of pushing the envelope around
that green button or a way – if people want to be relaxed with their
privacy restrictions to create more valuable data, if they are okay and we
could do that in a way that is appropriate, then I think we should push the
envelope in that kind of way. With that, I will just close there and thanks
everyone for your work.
DR. MAYS: Great. Thank you. One of the things that I think today showed is
– you know, we talked earlier in the workgroup about the integration. I
think it is wonderful, Damon, to have you here and to know that we are going to
have you here each time. I think we are going to be better at helping with the
agenda of HHS and having Karen DeSalvo discuss with us alignment and what have
Going forth, this will be our expectation. We will have a call with you
before the meeting, try to set up an agenda, get a sense of some of the issues
that are on your list, Jim’s list, Greg’s list, and then see how it is that we
can deal with those in the meeting and bring – we have some of the best
and the brightest here, in terms of the minds that are sitting around the
table, then, to see how we can be responders.
Our sense is that much of what we are going to do will be kind of within the
context of the meeting. You can call on us in that context. We will do as much
as we can and kind of other times for planning, but I think we are going to
kind of use the notes, give feedback, and really try and take care of your
issues as you bring them to us. We will be happy to receive your agenda.
MR. DAVIS: Thank you very much. I speak for myself and all of the folks in
my office and Health Data Leads across the department in thanking you guys for
your leadership, Justine, especially, for your years of leadership here, but
especially, as we sort of advance the Health Data Initiative. As Karen sort of
said, ONC hit an adolescence and now we are really advancing things. I think it
is thanks to the leadership of folks like yourselves. I really appreciate the
offer to continue to be as contributory as you are towards the agenda that we
have. Thank you.
DR. MAYS: Let’s thank Justine.
DR. CARR: Mind-blowing. Eye-opening.
DR. ROSENTHAL: We owe you a cake.
DR. CARR: I’ll take some bourbon.
DR. MAYS: I am going to end this meeting at this point because we are
starting to talk about bourbon.
(Whereupon, the meeting adjourned at 4:40 P.M.)