[This Transcript is Unedited]

Department of Health and Human Services

National Committee on Vital and Health Statistics

Work Group on Data Access and Use

December 3, 2014

US Department of Health and Human Services
Hubert H. Humphrey Building
Room 705-A
200 Independence Ave, SW
Washington, DC 20201

CONTENTS

P R O C E E D I N G S

Agenda Item: Call to Order, Welcome, Review Agenda

DR. MAYS: Let’s get started for today. Thank you to everybody for being here. We are about to start the NCVHS Work Group on Data Access and Use. I am the chair, Vickie Mays, University of California Los Angeles. I have no conflicts about anything as far as I know.

Walter, can we start with you and just come all the way around.

DR. SUAREZ: Good afternoon. This is Walter Suarez with Kaiser Permanente, member of the Data Access and Use Work Group. I am also a member of the Full National Committee and co-chair of the Standard Subcommittee and no conflicts.

MS. KLOSS: I am Linda Kloss, member of the Full Committee, co-chair of the Privacy, Confidentiality, and Security Subcommittee and a groupie of the Data Use Work Group.

DR. FRANCIS: I am Leslie Francis, member of the Data Use Work Group and no conflicts.

MS. BRADLEY: Lily Bradley, staff to the Committee.

MR. DAVIS: Damon Davis, HHS IDEA Lab, no conflicts.

DR. COHEN: Bruce Cohen, member of this committee, member of the Full NCVHS Committee and co-chair of the Population Subcommittee, no conflicts.

DR. VAUGHAN: Leah Vaughan, member of the Working Group.

DR. ROSENTHAL: Josh Rosenthal, member of the Working Group, no conflicts.

MR. CROWLEY: Kenyon Crowley, member of the Working Group, no conflicts, University of Maryland.

DR. KAUSHAL: Mohit Kaushal, member of the Working Group, no conflicts.

MS. JONES: Katherine Jones, CDC, NCHS.

DR. QUEEN: Susan Queen, staff to the committee.

MS. BERNSTEIN: Maya Bernstein. I am the lead staff to the Subcommittee on Privacy, Confidentiality, and Security.

MS. JACKSON: Debbie Jackson, National Center for Health Statistics, CDC.

MS. LUCAS: Jacqueline Lucas, National Center for Health Statistics, CDC.

MS. COOPER: Nicole Cooper, staff to the Committee.

DR. MAYS: Let’s get started then. As you know, part of our way of proceeding is that very much like Jim gives us an update, we have decided that we are going to have Damon do that in terms of as the chief technology officer for the IDEA Lab.

We are going to two things. One, he will give us an update and two, you should ask for any updates about things that were discussed in the last meeting because we had said we wanted to make this bidirectional. Let me let you start.

Agenda Item: Updates and Follow up Status from Office of the Chief Technology Officer/HHS IDEA Lab

MR. DAVIS: Very good. Thank you very much. Once again, this is Damon Davis. A couple of interesting things happening out of the IDEA Lab and the Health Data Initiative, the first of which is many of you may be aware that HHS has what we call HHS Entrepreneurs Program, which is a series of innovative initiatives to try to bring experts from outside of the Department, into the Department, in order to solve a problem that we have identified. Typically, this is an invitation to the private sector and the individuals who have these certain expertise to come in and solve these problems.

My point in raising that is for the first time, we have had an individual come in who is going to be looking at database linkages across the department with the idea being that as we have already discussed earlier in the prior part of the meeting today and at other parts of these meetings, the idea that there are many different components of HHS data that are valuable in their own silos of excellence, but could actually be made more valuable by virtue of a connection to some other HHS or other data.

The new entrepreneur who has come on board — his name is David Portnoy. He comes to us from Chicago, Illinois. He has some extensive experience with enterprise-level, database architecture and — linkages skill sets that we think are going to be valuable here.

As a bit of additional background about the program just so you get a construct as to how he is going to be operating, David is going to be traversing in the department, talking to some of our major agencies like NIH who, as you know, has a BD2K, Big Data to Knowledge initiative, talking to the National Center for Health Statistics, Centers for Medicare and Medicaid Services and many different others to try to look for some of these efficiencies and effective linkages that are going to add value to the data both for the department’s own benefit as well as for the benefit of those in the public who are utilizing these data and could actually find higher value from the linkages I am speaking of.

David is part of this Entrepreneurs program and the entrepreneurs are only here for about a year to 18 months. They get the air cover of the Office of the Secretary and are really charged with taking the fastest approach to finding the solutions that are going to have the highest impact. I wanted to just shed a little bit of light on that set of data activities that are happening.

Another thing that has been going on behind the scenes is you have heard me allude before to the — I see a tent going up. Perhaps I should stop and ask for questions on that and then I will go to my next piece. Go for it, Bruce.

DR. COHEN: The investigation of the data linkages. Are these going to be linkages using direct identifiers, name, and social security number or are they going to be algorithms developed using indirect identifiers?

MR. DAVIS: That is a really good question. I think that that will be something that David, the entrepreneur, will determine. That will be based on what type of data he has actually determined is going to be valuable for this particular linkage. Obviously, that is a very loose nebulous term. It would determine the type of data that we are seeing the value in, what the actual value opportunity is in connecting that data. If you could do it in a de-identified format, but it could be even further value back to the department if it was viewed in some say administrative way then potentially you would want to have the identifiers in it. It remains to be seen.

He is really truly on a voyage of exploration right now trying to figure out even how HHS operates, let alone where we have some valuable data assets that he may be able to bring additional value to.

DR. ROSENTHAL: Will he be looking at things other than PHI, for instance, linkages if we are talking HRR to contract or PNI for Part B or Part D release, things like that, which do not necessarily have privacy in the same way as members? But that is a core piece of the linkages. We are always reconstructing as we looking at graphic linkage or taxonomic linkage, et cetera.

MR. DAVIS: Everything is within purview right now. I would encourage those of you around the table and on the phone, et cetera to send in these kinds of ideas because again, he really truly is in an era of exploration right now, trying to identify what is deemed valuable by the data owners and program owners as much as by someone like yourself who has a different bent, a different angle, a different lens through which they see the data and therefore have alternative ways of viewing value by virtue of data linkages. You can feel free to shoot me.

He is actually in the HHS Global. It is simply davidportnoy@hhs.gov. You can also drop him a line, but feel free to CC me so that I can help to coordinate any understanding as to whom you are that is suddenly reaching out to it.

DR. ROSENTHAL: It is linkages. It is not refreshes or anything like that specifically.

MR. DAVIS: No. I think within his purview would be the idea that if data needs to be refreshed in order to be more efficiently linked in order to have that higher value then potentially yes. That is all within the sweep.

DR. COHEN: Maybe he could come to our next meeting to update us.

MR. DAVIS: Sure.

DR. MAYS: One of the things I was going to suggest is I think it might be useful for us to send an email to the entire committee because I know that there are some very specific needs that people were talking about earlier. Maybe what we can do, Bruce, is not only ask him, but after we get a sense of say if it is CMS or something else, we might even want to invite that person or something to be here so that we can just close the whole loop. Since he is fairly new at this, I think the CMS or whoever the person is that is here can help to make it go a little faster.

Lily, let’s make sure we send everybody an email, asking the Full Committee for input.

MR. DAVIS: Excellent. In an alternative work stream, you have heard me talk about the identification of strategically relevant data assets, the idea being that HHS has a clearly stated set of strategic goals. If we, as a department, are not actually liberating data or making data available that is in support of those strategic goals then we clearly are not doing our data job.

A couple of folks in my office have been working to try to coordinate with the various operating divisions across HHS to identify the data sets for which those offices have been labeled as responsible for these strategic goals. We have gotten I would say about 50 percent through the responses so far. The response rate has been pretty good. There are obviously some agencies that are focused on other things like Ebola right now and are therefore not necessarily thinking about the strategic plan and their data as a place to focus any energy. That is another activity that continues to evolve. I am hopeful that I will have a clearer update for all of you as to what that looks like, where we have identified opportunities and challenges in our data. It may in fact be that we want to have this group think through any gaps that are seen on the matrix of the strat plan versus the data assets, what you think might be valuable in that space.

Another thing that we are just generally going through is our broader coordination with data liberation over to OSTP and data.gov and things along those lines. They have a lot of updates to a metadata schema that has Common Core data elements across all federal entities. They are going to be updating those Common Core data elements in a refresh of the Common Core Metadata for Project Open Data that is going to go live in February 2015. What that means is there is going to be significant coordination across the healthdata.gov platform in order to set up the infrastructure that is going to allow us to now input data in this new format, retroactively update data that is going to meet with the new format as well as interfaces with some of our automated systems. CDC, CMS, and others have automated JSON files that they produced out of their catalogs that we are ingesting. We obviously want to make sure that they too are updating to the new metadata schema in time for February 2015.

Go ahead, Leah.

DR. VAUGHAN: Is that now or will that be on the GitHub?

MR. DAVIS: Project Open Data in its entirety is on GitHub. The updated metadata schema is on GitHub. They have developed tools that will allow you to do a validation of your own JSON. They really are trying to make it a lighter lift for all, not just those agencies that are advanced, but those who are trying to catch up. They are providing tools so that everybody can be on the level playing field in terms of catching up their metadata schema.

I am trying to think if there is anything else right now. I guess really the data discovery and evangelizing for the data story continues to be an activity that we are really heavily engaged in. I think one of the shortcomings of our office right now is an inability to do all things. We go to staff meetings and we hear about reports and data sets and things that are coming out and we try to chase those down. But similarly, you can talk day in and day out about really cool data elements and as a data geek, you will think they are cool. But if you are not able to tell a real more human story about how the data are used, you are doing the availability of the data a disservice. I really am working hard to try to find some of the stories that we can feature on the healthdata.gov blog that how folks are innovating on the data, how the department is using its own data in order to do really interesting and cool programmatic things.

I was struck earlier, Susan, by what you said. I took notes on it. But there was a Medicaid Undercount Project. I thought that was fascinating. I would love to be able to revitalize their story, have it told on healthdata.gov to show the collaboration between our various operating divisions and departments as to how we are using data to address issues just like that. As you all think of anything along those lines, please feel free to share stories and contact people and make reference to any of these interesting factoids that will bring some color to the data.

MR. CROWLEY: I think that is a great idea. If you are looking for existing programs that might provide lessons of how to get those stories together, I know ONC had their patient help ID stories. They had a whole site and infrastructure around gathering responses from the field. There might be something that is reusable from that.

MR. DAVIS: I am really glad to hear you say that. I used to work in ONC. Before I was in their consumer E-Health program, I was in their communications office. One of the things that Dr. Blumenthal wanted us to do was try to find some of the stories about how health IT is utilized. We called them health IT stories from the road. I did the same thing where I was out trying to find all of those stories of where electronic health records had been meaningfully impacting care. It is a constant refrain of how do we tell the story of how insert your subject is impacting health or social services. Thanks for raising that.

MR. CROWLEY: It sounds like they have the right person.

MR. DAVIS: Yes. Are there other comments, questions about anything that I have said so far? Anything that I did not cover that you wish I had said something about?

DR. ROSENTHAL: A question about plans on refresh basically. This is kind of one of the core tenants. If we release something — say Part B and Part D is a great example. We release this. Now, we can do all sorts of wonderful things and literally look at every single physician and say are they unnecessarily performing surgeries, unnecessarily prescribing lots of referral patterns. We send that thing out. Presumably, with the market or communities actually using that in some meaningful way and potentially incorporating that into their workflows, the immediate question is not only how recent is the data, but should I invest in using this whether it is a technical implementation or interventions. If I do not have any foresight or visibility into whether it is going to come out again. Obviously, some of it is constrained by budget. I get it. But that refresh piece is one of the hugest roadblocks to mark adoption broadly speaking.

DR. DAVIS: I appreciate what you are saying and I have heard that comment from you before. I think that that is one of the things that we would love to have better information on as part of a broader agenda for better metadata. It is as much about the frequency of the data’s refresh and actually expressing that in the catalog so that you know very quickly that I just missed a refresh. It just happened January 1 of 2014 or whatever versus absolutely no information about the timely delivery of this data. That is a metadata set of fields that I think we should be a lot better about. Coinciding with that in understanding the reliability of the data, I think would also be introducing information about the collection and curation mechanisms so that you understand the restrictions or the opportunities in the data versus just having to guess. There are a lot of folks who are not as well versed as you in many of the data elements and they are going to be coming to health data, social services data from other industries and they will need to know more and more about it.

DR. ROSENTHAL: And definitely on the metadata and the curation, but even kind of a basic business question to your storytelling. What question does this data answer? Number one. And then number two, do I have any plans? Just basic business 101. I am not talking huge technical implementation — developers. If I am a data owner, I am going to release a set. Does this answer a meaningful question? If so, what is that meaningful question and can I state it and then how do I disseminate that?

Aside from the technical details around it, literally, if I am going to release this and I am going to ask people to use it, am I ever going to see an update of this? The Part B and Part D is such a great example. 200,000 downloads in five minutes. But the market meaning communities broadly speaking. Sort of failing to adopt that into mainstream workflows. I personally have heard from probably no less than three dozen entities considering it saying I would love to build something around it. It is groundbreaking. But right now, it is two years old. I have no visibility into whether it is ever going to be updated to persuade me to use it. If I am going to release the thing, do I have a plan for ever having a refresh? If not, should I state that upfront?

MR. DAVIS: That is a very good point. I am going to note that down as something that we should follow. I appreciate where you are coming from on that.

DR. MAYS: It also might be something that when we put into the principles on the guiding principles on accessing views that we get people to say when they first put the data up what the intent is in terms of the refresh. That makes them think about it as soon as they are putting it up. Make sure that we will get this into the —

DR. ROSENTHAL: That is such a great — because you get immediate buzz and it is getting picked up in the New York Times and that is great. And then kind of what happens to it. Does it all go away? And understanding obviously it is around budget and you only have certain allocations, but if you are forced to think about, you know it is a best practice going in, then all of a sudden that might change some of the dynamics around.

MR. DAVIS: Part of it could quite literally be just the function of making it a required field in the catalog. You cannot complete your catalog entry and so you have that. Even if you say not intended at this time, you are forced to state it.

PARTICIPANT: You have to say something. That is exactly right. We could certainly work on that.

DR. MAYS: But I also think the other question that you raised or the other comment you had is like what questions can this answer. That has gotten to be this hot marketing thing. I know in terms of our websites and these university websites, now, they are starting to say write a question about what your work is about. And I thought what. And then they sent us to other websites. They came back because none of us wanted to do it. They came back and said being able to see what the question is and people understanding where the question can go is better than a list of this is what I do.

DR. ROSENTHAL: Hot marketing.

DR. MAYS: I am just going to tell you that is the — this notion about saying something about what this can do is getting to be, I think, something that people actually want out there as a part of the face of an explanation.

DR. ROSENTHAL: It is kind of the flip side of the stories that it has done, what the potential of it is to do and that forces the owner, the disseminator to actually think that through. That is kind of what you do in other fields outside health care. If I am going to put out whether data, I am going to say this answer. Is it going to rain outside? Should I bring an umbrella?

MR. CROWLEY: And I have noticed that many of the HHS data sets have that information as part of it. Several of the data sets will say what are you trying to answer. This is what you can answer with a set. This is what has been done. This is the types of elements. Maybe it is more of putting up more focus on that or making more explicit or making more consistent.

DR. ROSENTHAL: Or a requirement. You definitely have — I routinely deal with probably 120 different public data sets out of HHS and there are probably 12 of them that come to mind where you see that. Making it a requirement even if you say nothing. Same thing with intended frequency. That could become part of a reasonable metadata infrastructure as well saying show me all the datasets that have intended refresh annually, quarterly, monthly.

DR. COHEN: We are going to talk a little later about data dissemination as one of the foci of our work. I think this a great conversation about if we do some kind of survey of HHS data sets about how they disseminate data, clearly, this is stuff that Bill Davenhall started us thinking about. What should be included in those questions? What do we think all data sets should have whether it is metadata or tagging or just dissemination strategies? This is a great start.

You are all welcome to join my subcommittee.

DR. MAYS: Let’s do a couple of other things before we actually move on because we do have a person coming in at 2:30, Chris Boone. One of the things an email popped up about is Datapalooza. Is there anything that you want to say about the next Datapalooza because it is popping up already?

MR. DAVIS: I think the man who can answer that is Chris Boone. Let’s ask him all about his Health Datapalooza when he gets here.

DR. MAYS: Any other questions in terms of the update?

DR. ROSENTHAL: Can I just throw in an update? I am actually co-organizing the data tracks, all of the data sessions or Health Datapalooza. I am a co-organizer. I brought in Niall Brennan who is the chief data officer of CMS. If any of you are doing anything meaningful with the public data or know anyone who is, please do submit an abstract proposal. We are actively seeking them as we construct the tracks.

MR. DAVIS: I would absolutely second that. I think that is one of the biggest challenges about the tracks is getting the people to come out of the woodwork with their really high-quality utilization of the data sets.

PARTICIPANT: Linda and then Leslie.

MS. KLOSS: Thanks. You mentioned Niall Brennan in that new role. How well you coordinate the work of the innovation lab with what new initiatives might be unveiled at CMS. Earlier in our national committee meeting, we were talking about the desirability to get more of the CMS particularly Medicare and Medicaid data available for various purposes. At any rate, I just wondered what the coordination would be.

MR. DAVIS: We stay in pretty close contact with CMS for pretty obvious reasons. They are one of the main data producers out of the department and therefore a frequent example of what to do and where to get data and so on. We try to stay in close contact with them. My boss, Greg Downing, is frequently on emails with Niall even prior to his appointment to this particular position partially because he is just such a phenomenal advocate for the utilization of the data in different ways. Rest assured, we are constantly looking for the avenues through which we can get some additional information as to what is to come. We are also feeding ideas as well and trying to take the friction out of some of their data release activities to the extent that we can. It is a monolithic agency and sometimes they need the opinions of folks like you all, but also some innovative thinkers outside of this organization to provide input. We try to give them that so that they can get a signal as to what is desired.

PARTICIPANT: That might be a good moment to —

DR. ROSENTHAL: Could we bring in Niall perhaps and ask him a couple of questions?

DR. MAYS: I don’t see why not.

PARTICIPANT: I don’t either.

DR. FRANCIS: Two quick points about Datapalooza. I do not know whether you are the best audience for this or Chris. First of all, when I went to the Datapalooza, there was very little about data stewardship or privacy. There was a whole lot about data dissemination and use. It might be interesting to try to do a privacy tract or a privacy techie set of innovation questions. Related to that, we did actually something a few years ago that the National Committee did. We have this cool new toolkit that is going to be getting launched in maybe a month or so on community health data stewardship and what are all some of the tools. Would it make sense to have a session on the toolkit?

DR. ROSENTHAL: That is a good Chris question. There is a community tract pressure.

MR. DAVIS: I am glad to hear. I think those are points better asked to Chris as the organizer.

DR. FRANCIS: I will just need to find out how to make sure we put it in.

PARTICIPANT: He is coming in.

PARTICIPANT: He will be here.

DR. FRANCIS: I know. I just was not sure whom the right person was to do it. Thanks.

DR. SUAREZ: The same can be said about the community health framework and the community health initiatives and the roundtable that we had this year. I think there could be an opportunity to include something on the Datapalooza as well. But we can talk about it when Chris comes.

DR. MAYS: When Chris comes, the intent is for it to be just a sharing back and forth. He is not doing a presentation. We are not doing a presentation. I think it would be very important that you actually bring these things up and describe them to him because part of what we want him to understand is what our terrain is that we have been working in and getting a sense of theirs as well and seeing where there are sweet spots in terms of work. If you all can remember to do that when he is here, that would be great.

DR. KAUSHAL: I think we should talk to Chris, but also can you forward me an email just outlining what you said? I am on the committee as well and I will forward it to them. Thank you.

Agenda Item: 2015 Strategic Work Plan Discussion and Development of Goals, Scope, Product and Consultation Needs

DR. MAYS: Great. One of the things that is typical for the committee to do in December is to talk about its work plan for the next year. Just so that you know, one of the things that was discussed earlier is it is not just are we doing 2015, but what is going to happen is we are going to push out. We have been asked to think about by quarters, what products, activities, hearings or whatever in terms of the broader committee that they will be doing.

What you should have — Damon, do you have one? It will be up on the monitor as well. What we want to think about is and we have pretty much talked about it, but what we want to think about is where we want to go, how fast we want to do this, and what we see as the things we want to achieve. Now, we are not the subcommittee. We actually have flexibility, but at the same time, we do not want to build such a huge agenda that we are talking about trying to have hearings and what have you.

I think what we should talk about is what it is that we think we can accomplish in a consultation mode. That is whom we are as opposed to a contemplative mode where we have everybody coming in. We listen and we contemplate what the next steps are. But instead, we should do it from a more consultative standpoint because as I saw the plan for the three subcommittees, it is going to be tight over the next year. I think if we stay in our lane, we can do two things. The goals that we are setting out as well as being an asset to the subcommittee. I just don’t want us to build too much beyond what our real role is as a work group.

When I began this, I talked to each of you. We had a sense of what you would like to do going forward. And what I like is I still see that there is interest and passion about it. We did not pick an empty topic. That is our assistance to HHS probably best comes in trying to figure out within various data sets as well as entities within HHS how to push the envelope in terms of both access and use.

Two ways we are going to do this. One is we are going to generate some guiding principles. You have been doing that even today. That is great. You can see. Lily has done very well at collecting them. We have given you some of the things that you have said.

In an earlier session, I thought that Linda raised a very good point, which is we are not doing this — but instead there are some guiding principles that HHS in general tells all of its data producers. We should probably get a hold of that. I do not know if it is in some one place. That is it. I do not know how. You all know it. But whether or not it is written down somewhere is probably what we really need you to work with us on. When someone says, here is my data. Here is what I am going to do. I know there are rules. I do not know that there are principles. Susan knows this because she is very helpful when we are trying to figure out what can and cannot be done to know what the rules are. But I bet that there are some principles. But what I do not know is what is the level of HHS principles versus what is the level of the way NCHS does it, the way that AHRQ does it with MEPS. Can you tell us whether or not there is some place?

MR. DAVIS: If only. I do not think there is, but I think there would be a lot of value in pointing out the fact that there is not one solid place and therefore starting to create it. If we are going to ask folks to rely on it, we may as well be the ones that cultivate what we do find and amalgamate it into something that is going to be beneficial for all. Unfortunately, I do not believe you are going to find one single repository where that set of guidance resides.

This actually sparks an idea for me. One of the things that we have been thinking about is internal to the department, anybody who is even remotely interested in the data initiative does not necessarily have a single internal place to go. You have to hear by word of mouth that you should probably talk to the IDEA Lab or something along those lines.

My point being we have been thinking about having a data-related space on our intranet that would allow us to highlight things like this work group. The group of health data leads that I commonly rely upon for all of our data activities. Things like that. A source where people can go in order to get this kind of data. This sounds like the type of thing that would be great fodder to add to that mix. What are the sets of things that I need to be considering as I am producing and making data available? What are some of the policies that we have made adjustments to that are going to allow for the IAAs and DUAs back to the administrative data thing. Literally just what is their repository of guidance that anybody can quickly tap into. I think this would be a very valuable addition to that. I am glad to hear this because it encourages me to pursue that intranet site even more.

DR. MAYS: Susan, can I ask a question?

DR. QUEEN: Are you talking about one place where you would have either statutes or regs related within HHS related to privacy depending on whether it is claims data or statistical? I am not sure what you are talking about.

DR. MAYS: Not that because I think what happens is that different data sets will approach that differently. But it would be more — for a lack of a better term right now, it is kind of guiding principles like the things we were just talking about to think about. When you put your data set up, it is about just use and access. When you put your data set up, do you list questions that the data set answers? Do you have the metadata?

The question I was going to ask you is has this come up at all in data counsel because that seemed like it would be a place where you all might have some.

DR. QUEEN: Not really other than in responding to open data, but it was quite a while ago, looking at the issue of confusion over metadata versus data documentation and survey documentation and how things are provided. There is not a standard for even within a given agency. Survey X may have a different way of providing documentation than Survey Y. It has not come up to a great extent.

MS. KLOSS: I guess I have to say that kind of surprises me because it seems like such a foundational kind of governance process before you release data that there is at least some framework.

DR. QUEEN: What I would say is that the department would not necessarily be taking a position on telling a statistical agency or entity — the statistical agencies themselves — there are a lot of different committees and inter-agency collaborations on what you are supposed to provide. It does not even mean that within that set of agencies that you have a consistent way of documenting everything, but you have to document. But they may not all look the same.

DR. COHEN: It does not surprise me at all for a couple of reasons. One, I have worked in government for 35 years. And two, the target audience, the purpose for the different data collection systems vary so enormously. By definition, they have different practices and parameters. The question is is there a way to step back and have a higher level of conceptual framework that all the different purposed data sets could adhere to try to bring more order. People have been so focused on the use and promotion of their data sets. I do not think there has been a lot of strategic thinking about how to do that. This is a perfect opportunity I think to engage more broadly in that discussion.

DR. MAYS: Just before I call on you, I think the one thing we want to be clear about is that we are not going in the realm of any of the rules, the regs because that is going to be so — we will get ourselves in a bind. But it is more like — Bruce, when you were talking, it is more like the kind of broad vision, here are some things to think about. And I want to take us back and remember that we want to do this by these different use cases so that people have a clear idea. A data entrepreneur could go on and in a very short amount of time say okay. I can see which one of these that I can work with quite quickly. And a consumer goes on and says I need to go and get an intermediary that is at another level to help me because this is exactly the question I want to answer. But it says that I might have some difficulty. I want to go and get someone.

DR. ROSENTHAL: That is basically what I was going to say. There are these various initiatives and it gets bogged down. I do not want to say anything about standards, but it always goes into the weeds. The only time you touched on this has been in the realm of open data largely because now all of a sudden you have these different users and you have to answer the question. Now it is a really good time. If we could do one thing as a work group and basically be able to do that and say here are some axiomatic principles or call them something nicer. Guiding principles, best practices. But here is a strategic approach to not only disseminating it, but making it useful and meaningful or your for personas or whatever you want to do, not just useful for me, but useful for Bruce and somebody else I do not know and some other person.

And to Kenyon’s point, you find bits and pieces of this. I can actually tell you who the data owners are who do a pretty job of this. Chris Powers, anything out of his group usually has bits of this. It is pretty good. But basically just looking around and saying let’s apply even the best practices internally within HHS around this and say this is a place for a strategic thought, not that you won’t have — even if you do not have a hammer to enforce it, if you put it out and say this is a best practice and this is part of the culture. Are you releasing that? Have you thought about the best practice? Choice architecture.

DR. COHEN: The way I would look at it — there are four basic groups that we target data to. There are the entrepreneurs. There are the researchers. There are the individual consumers and then the community. These are the four target audiences that we package and disseminate data for. Any given data set can be targeting one of those audiences or all four of those audiences. I could go through an example looking at NHANES data and see how — dissemination strategies for NHANES would differ for each of those target audiences. Can we build a framework or a set of guiding principles for NHANES to target each of those potential data audiences that is conceptually broad enough, but clearly explicit so that all data sets or most data sets generated by HHS can fit into that summary framework that would help guide folks? They identify themselves as the user and then that information would be available for them.

DR. ROSENTHAL: That is not heavy technical architecture. That is sitting down with it and thinking about — if you want to translate applicability of data to different groups, if you just do basic question and answering frequency refresh and then say let’s go through the list of this. There are some wacky data sets that you would not think of for consumers — think of some particulars. But they do have great applicability. Being able to translate that can make the data so much more valuable.

DR. MAYS: I think what we would be doing is coming up with the principles and then a consultation with the group. At the end, what you want them to do is to look at the principles, look at the use groups. I think that is building the very group that you run. That is building what is the set of questions that we want to ask a person that is coming into to see us. I think it is exactly what you are talking about.

Realistically, not all of the surveys are going to respond very well I think to the consumer or maybe even the community because, again, I am not going to do today like who is the community. I think that we do not have a hammer to make them do it, but I think by having the principles, it makes them know that other people are being a lot clearer and they are going to get better use. I do not know if we will be able to demonstrate, but later it may be even better participation because you realize that is a survey I was able to find X out from. We were just worrying earlier about people’s participation. That more that a survey appears to be useful to people, the more that you call and they are likely to say yes in terms of participation.

DR. COHEN: I do not think we need a hammer. Whenever I want to know anything about a federal health data set, I go to the back of Health, US because the appendix at the back of Health, US has a three-paragraph summary of all the important data sets. I know a lot of people use that appendix as their bible reference. If we can create a reference document that we can direct people to do so they can quickly look up what they need, that will help them in their investigation.

DR. MAYS: I think what we are doing is adding something to our work plan. Lily, can you take us to each of the quarters for now so I can see what we are doing? I like the suggestions earlier, I think it was Alex or Linda, when we also say now how do we get this done kind of thing.

What is on here — I think most of you have this. Now, we are talking about not just guiding principles for data access and use. We are talking about needing to build a document and the document that we are talking about building is one that begins to lay out the information. We know already that HHS has some, but we just do not know what they are. If we cannot find them, what we need to do is a little ferreting around to see what it is we want to put in it.

It is a little more than we were talking about guiding principles for data access and use. We are now talking about making that an original document in which what we come up with is what we know is kind of untalked about operating procedures that we will have to find out from various people as well as what we think are some of the guiding principles.

And then you want to take this to the level of having a site of putting this on the HHS website. Bruce, you were talking also about going to the back of Health, US.

DR. COHEN: There are different groups of users who enter many different points of the system. It should be everywhere where people can access it. If people started healthdata.gov, it should be here is the reference. Some of it exists in the metadata and the descriptions. It is not in places where people routinely go in an organized, compiled fashion.

DR. MAYS: It says one of the things we need to know is something about how those various groups enter into HHS to look for information. We need to see if there is some way in which we can get analytics that say consumers tend to come through X so let’s make sure it is there. Other people come through Y. We want to try and figure out. It is not in one place because the users enter so differently. We want to do that.

Damon and then Walter.

MR. DAVIS: Just a quick one. This is Damon. The idea that there is no “hammer” I understand, but if we are going to influence people towards using some level of guidance, you have to give them something that you can refer back to, a dashboard, something that basically shows whether they are or are not compliant.

A perfect example of this is in the open data space, the Project Open Data team has created a dashboard for every single department in government. And on a quarterly basis, they basically highlight which agency has met any of the varied requirements. They give you a percentage. They give you a color code, et cetera. The point being there is a public shaming element to every agency’s response to the Project Open Data initiative. I see you Susan.

But my point is some sort of — if you are going to create the guidance, there has to be some tie back to actually whether it was used or unused. Maybe it is not like a public dashboard that says you are doing poorly as much as like a badge from the NCVHS work group that says it meets the standard. You then get put on the catalog so the healthdata.gov has the little green check that says meets the standard. I am just throwing it out there. I did not think of it until ten seconds ago. Don’t be critical of it. I think there is value in actually tying it back somehow so that it can be some way seen by people.

DR. QUEEN: Josh mentioned something that is really important at least from an agency perspective, which is low burden.

DR. ROSENTHAL: That is why he is in IDEA Lab. Dashboard, you are not a coffee pourer you are a barista. This is the best practice approved badge.

DR. MAYS: It is a badge to acknowledge.

PARTICIPANT: It is the seal of approval kind of thing.

MR. DAVIS: It is the type of thing that someone can quickly identify with.

DR. ROSENTHAL: For what it is worth, there are requirements. No one likes to talk about it. I have actually tried to do this myself in other instances. There is legislation requiring user feedback on specific data sets. I just want to be really clear. CMS, in particular. They are supposed to take user feedbacks. If someone comes to them and says we use this regularly. One hundred million lives do this sort of stuff across 35 of the 50 states. Would you like to know what our feedback is? They are supposed to listen to that.

DR. MAYS: What is the legislation?

DR. ROSENTHAL: This is a Lily question. I will find it for you. I have it.

MR. DAVIS: Even Project Open Data has a piece that requires you to document your public feedback.

DR. ROSENTHAL: You are supposed to accept and document public feedback. There are a couple of different pieces around that in different departments.

MS. BRADLEY: You are on the record —

DR. SUAREZ: Thank you. Just a couple of comments. I mentioned earlier one of the things that I think would be valuable as part of the assessment is not just assessment about data dissemination, but about the usefulness and the usability particularly of the data, how usable it is since that is actually one of our primary goals, which is making data more usable and hopefully useful as well. I think it would be valuable to look into how do we evaluate that and again not just from the perspective of the provider of the data, but more importantly for the user of the data.

The other thing that I wanted to ask as part of the work plan is really as we define topics, actions and products or outcomes — so we know the topics and we know some of the actions. I do not know about the outcomes or products. In other words, I am about to send out to the entire committee and work group the master table I guess of all the work plans of all the subcommittees. I could not really identify what is the actual product — a letter of recommendation in the other subcommittees in this case could be the recommended guiding principles or something. Is that one of the actual products that will be produced, finalized, and published in Q2, for example?

DR. MAYS: There were two products that we had talked about. One would be the guiding principles. I think we pushed that off to Q4 to finalize those. I think we had it in three and I thought it would take us longer.

DR. SUAREZ: I know that it says that, but it sounds like we are spending the entire year really developing it. I do not know if it will take a full year. Maybe it will.

DR. MAYS: I think because the agencies are so different that we are going to be better off with not doing it too quickly. I think what we can develop a lot faster is this sense of what our product is going to be in terms of consulting with the agency. The questionnaire that we want to develop in terms of identifying, kind of what some of the best practices are that we want agencies to do to come in and give us feedback about. I think that that is going to be developed first. I thought we would look at that to see whether or not — I think the pop subcommittee wanted to use that to push that out as well to see whether or not when they are talking to data connectors whether or not in talking with those data connectors that they have a sense of these things. We can be very ambitious and pull it back.

It is funny because once I saw how much we actually have, it was — Lily had it in Q3. I am the one that put it in Q4. Lily had a better sense of how much we have already done. It could be that we could think about Q3. I am always one that thinks everything takes a little bit longer than you originally planned to do and especially if you are going to subject it to needing to be approved by people.

PARTICIPANT: — a realist. Is that what that was?

DR. MAYS: I am calling myself — I put it in a box where I really think we can get it done because February is around the corner and that is the next meeting. Very honestly, we do have a lot done, but I think the devil is in the details.

I think that we can always shoot for three, but if we leave it in four, we will see what happens.

DR. SUAREZ: You said there were two products. Number one. What is the other one?

DR. MAYS: The other one would be the assessment and the other one then would be the guiding principles.

DR. SUAREZ: The guiding principles and then the assessment tools.

DR. MAYS: Yes.

DR. SUAREZ: And the assessment tool — with what quarter do you see it?

DR. MAYS: The assessment tool we need to have by probably the next quarter.

DR. SUAREZ: Quarter one of 2015?

DR. MAYS: No. That means by the February meeting. I think we will have it finished by quarter two. We will have a draft in quarter one because we cannot keep calling people in and not having any idea of what we want to ask them. I would say quarter one we are going to test drive it and quarter two we are going to try and finalize it.

DR. SUAREZ: I have quarter one as developing and finalizing the data assessment tool and then the data dissemination — quarter two actually conducting the data dissemination assessment tool consultation.

DR. MAYS: I kind of think of it as the same. We are just being technical. I am saying that in quarter one, we will have the tool. In quarter two, we will actually push it out the door and see what happens with whomever the group is that we are bringing in. And then we would fine-tune it. If it was something where I was ready to release it to the public I would say it is not until after quarter two when we see how it works.

DR. FRANCIS: Maybe I am totally not understanding what is going on. It seems to me that the assessment tool is a whole lot easier than guiding principles and here is why. The question of assessment is really topic areas. Have you addressed this? Have you addressed that? But the question of guiding principles is going to need to be at least three-dimensional and maybe more than that. It seems to me that the substance of the principles anyway is going to vary with the data in question and the locus of release. You can have a vacuous principle like protect privacy to the appropriate degree. But that does not tell you what you need to do to protect privacy depending on different types of data and types of user. It seems to me that I do not think there is going to be a single set of principles except that a very high level. But I think there could be a single assessment tool unless I am totally misunderstanding what you mean by principles.

I also wanted to say that I think there is another — we have not differentiated in the list of users at least in Bruce’s list government entities because communities and other government entities with a sub-division between federal and state also are different sets of folks to whom the data might be released.

DR. MAYS: Let me just respond to that because the use cases that I originally did, which differ from what Bruce threw out on the table really had to do with capacity. That was why you can put lots of dots. The first one would be like data entrepreneurs, data warehouses. They not only have the capacity. They have the resources and will move at a certain pace.

The next that I had was like this level two were people with skills. That would be the researchers. That would be the federal agencies. That would be people who have a skill set, not always the amount of resources they may want, but they have the ability to get the data and do something with it. There is different helps that they need. Researchers will tell you there is help. The federal agencies will tell you that there is help.

And then the third level that I had thought about was people who do not have those skill sets nor the resources. You are then talking about should this data set have a query system. Should the data set be a little table? You begin to have a sense that these extra things —

DR. FRANCIS: That is focusing on usability for the users and the user’s level of sophistication. When I start hearing things like guiding principles for data access and use, different users are going to be subject to different legal constraints, for example. I am just taking a very simple example. The kind of data use agreement that you might have in sharing with another governmental entity might be very different than you would have sharing with an entrepreneur.

DR. MAYS: But our customer is DHHS. In terms of the principles, that is whom we are going to call in.

DR. FRANCIS: But aren’t we talking to — our customer is HHS, but aren’t we talking to HHS about what they need to think about with respect to releasing their data?

DR. MAYS: Yes. When we think about what the guiding principles are is what the guiding principles are in terms of them dealing with their data as a federal agency. In terms of the next step would be when they go make other relationships, those relationships, I think, would be so varied that it would be very hard. Just in terms of what you need to think about as an agency that reports to the secretary, here is what we think are good principles that should be basic to every one of these data sets. That is the guiding principles for them to get people to be able to access to know it is there and then usability is a little different. Other people can jump into this.

MS. KLOSS: Just to clarify. When you stated it the way you stated it, you do not mean principles with regard to every aspect of their data. You just mean in regard to access and use because we cannot get into principles for all aspects of data gathering.

DR. SUAREZ: They are listed here. The principles are in these two pages. It covers stewardship, formats, metadata, social tagging, data communities, and top-down strategies. As I understand it, I totally agree. This is about principles —

DR. MAYS: Access and use.

DR. SUAREZ: — data access and use principles for federal data —

DR. MAYS: What we are trying to do is to stay within, one, our charge, but this lane about access and use. Now anything that spills over, I think is fair game for the Full Committee to if they want to discuss it there. But I think for us we should be sticking to HHS as the customer and as a customer, this is like the things to think about in terms of when you put your data out there.

DR. ROSENTHAL: I will take a basic high level. Also, just for context, this is done outside health care all day long. This is not new. It will end up being very high level. For refresh, we won’t say you need to refresh at this frequency or this frequency. We will say you actually have to think about refreshing before you drop a lot of money, time, and effort. If we are putting — Part B and Part D, you might want to state. Do you intend to ever refresh it if you expect marketed adoption? Before you drop a data file, you might want to say what kinds of questions can this answer. Basically, how would I use this if I were X, if I were Y, if I were Z.? That may be very high level, but it is not currently being done. Yes, it is like a tenth of the sources that you consult with. It is being done sporadically in pieces. If we can actually have that as a guiding principle across these data sets, they will be tremendous.

I was going to say Q4 realistically. I was not going to bring this up until now, but if we can also think about one way to capture feedback and then also potentially when we talk about metadata, we have done the 101 now and our training wheels are off and we know what we are talking about. This means this and relationships. Also potentially something around methodology as well, which is as important as entity relationships. We talked a lot about ERDs previously. Capturing feedback.

For instance, when I have seen the scorecard of this data owner gives good feedback and the email is listed and I can get a hold of them. Literally, two days ago I am looking at a data set. I email. I get the woman’s name. I email here and I say I am looking at your data set. I see the metadata. The column’s name is literally column. What does this actually mean? It means it is a column. Thank you. Any further questions, direct to blah, blah. This is like the fifth exchange with them around this. Technically, there is a data owner who is responding to me. On the scorecard, it is green. That is kind of the next level of thinking about it.

And then the other idea around methodology and what do I mean by that. I don’t mean fancy schmancy. To use an example, if I am interested in health care and I am going to set up service and I am going to do meta-care, commercial, I have to have network adequacy. I have to make sure if I am going to set up a plan that I have doctors who will serve as those people. That is a required entity of a business problem. HHS puts out data that answers that. They tell me who are the doctors, where are their locations. It is geo-located. It is a brilliant data design. I can go through and sift through it.

The question is I do not want to constitute adequacy. The business problem is if I want to launch something in a rural area or in a community, do I have network adequacy. I have a business and community problem saying I have to have adequacy. I have a data set. When I look at that data set, it literally to Kenyon’s point says you can use this to determine adequacy because I have a name of every doctor and it is geotech. But I actually do not know what constitutes adequacy. I send an email in and say how do you guys constitute adequacy. Do you do north versus south geocode by member location? Do you do average of distance to mile between provider and member? Basic stuff like that because it is a business —

DR. MAYS: What would be the high level of that? What I am struggling with is how different these various data sets are. The concept that one is not going to — the other set.

DR. ROSENTHAL: The high level is axiomatizing it. If you have a requirement and the data is supposed to answer a question through some sort of formula or a method, you have to state what that formula or method is. It is very similar to the entity relationship diagrams.

DR. MAYS: I need another example.

DR. ROSENTHAL: If my business question — I am going to put out a list of all of the speed limits in the areas where I have different speed limits. I am supposed to use this to determine where is a fast place to drive and where is a slow place to drive. I have to put out the basic formula that 75 miles an hour is faster than 25 hours an hour. It is true that I have released the data, but no one can use it. Literally, I have had 50 people ask how to do this adequacy stuff.

PARTICIPANT: It just seems like it is so done.

PARTICIPANT: That is kind of a thing. That is why you started with that column example because that is literally the same data set that has a column named column. Technically, they are meeting the requirements. There is metadata in that field. There is literally the word column on top of the column. Technically, every single provider, every single speed is registered: 55, 75, 55. We sort of know that 75 is faster than 25. We might say you have adequacy if you have a provider within 10 miles, 100 miles, 1500 miles of a member. That is never stated. When I have asked, it is proprietary this or that or we actually do not know, et cetera. That is an example of if you have a business question and HHS or CMS puts out a data set to answer that business question, we should probably — it seems so basic, but in the spirit of Leslie’s stuff that we deal with every day, that is another great example.

The same thing with the relationships. Not just the taxonomy, but what is the relationship of HRR to hospital service area. I have HRR and HSA in those files, but I actually do not know what the nature of the relationship is. I have to reconstruct it. They say HRR means hospital referral region. HSA means hospital service area. Are they the same thing? Are they a different thing? One, getting metadata labeled, but also something beyond that. Does that make sense? I do not mean to get too advanced too quickly.

PARTICIPANT: To me it was at first seemed so simple. You mean to tell me that is a real example. I think what it is is that I am making an assumption and there are some data sets that if I really think back when I was first beginning, it was like you really do spend a lot of time trying to figure out — as an example, is SES in this data set defined the same way it is in this data set. They named it that. And then you go to do your analyses and it is like why is this not working. And then you realize one had occupation and the other had earnings. We are looking at apples and oranges. There are lots of data sets that give you a name. You have to really have enough skills.

DR. ROSENTHAL: That is a long roundabout way of answering of answering Leslie’s questions. Can we have some guiding principles? And the answer is the stuff seems really basic, but I probably see it covered on 5, maybe 10 percent of the stuff.

DR. MAYS: I got it once I could retranslate it into when I first began type thing.

Let me just do one other thing and then I am going to call on Kenyon and then we have to take a break — and Walter and then we will take a break.

The capture feedback.

DR. TANG: Paul Tang. I joined late — point where I can make a comment about work —

DR. MAYS: Let me do this real quick. The capture feedback. This is Walter’s point about making sure we define some kind of metrics for the user. I think earlier when we talked about this, I talked about when people go to your data set and you ask them to fill out feedback. You also launch. This is probably part of what we could ask is how much we can ask them to launch metrics in the background. I do not know if the various agencies want to run the metrics or that is something your office would do. But in the background, you get somebody and you can tell they are this kind of user and you can see. It is almost like when you check out the grocery store. Did you find everything okay? You can try and figure out by the type of user how difficult it was and where they cut off. We do this in terms of surveys all the time. We probably need to do this in terms of users. That will help us with the metrics. I think we need to capture that as well. I would like your capture feedback kind of thing because we should do that.

Let’s take Kenyon and then let’s take Walter and then let’s take Paul.

MR. CROWLEY: There has been a lot of interesting discussions since I put my card up. I will use — the last in first out. On some of these points, it is really interesting about defining — run your level of data sets, the adequacy and different types of questions and interactions between terms and sets. I think this is also an example of how that is probably beyond the scope that any one HHS department or person could do with those resources. This is probably an area where we need to democratize the data more, democratize the learning of the data, democratize the findings around it.

For the example that you just had, is this one data set for SAS doing the same? That is the type of question that could be posed within the community forum on the data set users. Within that community and consistent with the next comment of the principles, there should be a moderator. I think there should be a moderator from HHS whose experience and expert in data who is monitoring these communities.

A nice analog to that — I think it has been brought up before. Something like Google products forum where you have a rich dialogue across all the different users of varying skill levels about different activities across the community while you have the expert from Google and Google products in the community providing the validated voice, answer to that. And you have multiple ones of these going at any time. You can amplify the process of getting to this well-described set of information within the data sets. It lives on. It is searchable. It also helps you define where if a lot of people are asking the same questions, then it is likely that this might be an issue that we need to tackle. Sometimes we think we have issues that are not issues and we do not think we have issues that are really big issues. This network community helps to answer that.

Our earlier comment was when we were talking about the guiding principles data access use and the data summation assessment tool. The thought that struck me is when we talked about the guiding principles. I want to make sure we use the right language. There are some existing guiding principles for the open data movement. OSTP — if you read those, they are very consistent with everything that we are discussing here around you are describing the data and how it is being made successful, how it is being reusable, frequency and timeliness of release. Some of these principles are guiding this whole effort.

If we were really talking about principles for HHS, I would just say that would probably be at least a starting point.

DR. MAYS: Lily, can you find those?

MS. BRADLEY: He has provided the URL here, but his computer is not online. I think everyone else is online. I am online.

MR. CROWLEY: It is project-open-data.cio.gov/principles.

DR. ROSENTHAL: There is good stuff in there. It is paragraph format, et cetera.

MR. CROWLEY: There are implementation guidelines.

MS. BRADLEY: — you can make comments on specific guiding principles.

MR. CROWLEY: It seems what we are talking about are rules.

DR. MAYS: I do not know if we are talking rules.

MR. CROWLEY: Maybe not rules. Rules are the wrong context in a government. Not rules, but maybe guidelines or a checklist for effective data release.

MR. DAVIS: I think you have raised a really good point. I was thinking about the question of whether there is a centralized repository for any of these kinds of principles at HHS. You are absolutely right that that there is the overarching set of principles that we too are following. It would be a good to have a broader conversation about what those are as it applies to what we are thinking about here.

DR. MAYS: Maybe what we can do is have a little bit of homework that we are going to pull those up and look at those before the next meeting. See what we want to have as the overarching and then maybe after that some of the other things that we think are important. Let me let you finish so I can get to Walter and to Paul.

MR. CROWLEY: Just on the dissemination assessment. Having simple rules with symbolism that helps people know what is and isn’t, but whatever that is. It is breaking those principles down to that. Simple things that you need to do well as a data release owner or stakeholder to make sure people understand it. I think we have the capability within this group and within experts that we have within our network to define what that is. That seems like a rational doable goal.

DR. SUAREZ: I think it would be helpful as we look at the principles to also develop the metrics for the principles. In other words, we have a series of principles, but then how do look into whether and to what extent people are actually using them and ultimately really how again as we look at assessment tools and things like that, how do we measure the overarching concepts and the principles of usability and usefulness and dissemination and other things? That is one element.

Now going back to the work plan, there are two more questions I had. One is we will have a dissemination assessment tool and there will be some assessment work done with the agencies. Is there a report to come out or is there a summary of those? Is that another product at all or no, this is more like internal information?

DR. MAYS: I would have to ask because — the reason I say I have to ask is because I do not have a sense that we are functioning quite the same way that the subcommittee would. I think that is one of the questions that I will talk with Jim about and figure it out.

But it also may be that if it seems like it facilitates this, it is like if your office uses this for something, I think that is the other thing is to have a sense. It is not us doing a report so that it goes to the secretary because we are doing it a little differently. I would be whether or not doing the — looking at the evaluation in some way has a usefulness to what you are trying to achieve.

DR. SUAREZ: It is a cross agency thing. It would be helpful for one agency to learn what the rest of the agency said about some of the dissemination. That might be one of the utilities of that type of a report.

MR. DAVIS: This is Damon. One of the things that could be valuable out of that process of evaluating and reporting is the ability to highlight the hot spots where interesting and cool things are happening and help those organizations where they are not happening to understand how those who are proficient got to be proficient. Less about the tattle telling on those who are not doing well as much as here are some bright spots. Let’s shine a light on those and help others to reach that same space.

DR. MAYS: Then I can see it as in our lane because what it means is that it is to help the agency. It is not going to the secretary, but it is to help the other agencies maybe to get where that particular agency is. If writing that up helps then I think we are, again, serving our customers well.

DR. SUAREZ: I am typing on the draft of this work plan. The other one area that we have on the work plans is activities. Do we, as a work group, plan to have any particular activity during any time of the year? When I say activity, I mean some sort of a workshop or roundtable or even hearing. Is there any activity concretely that we want to highlight during any of the quarters during the year?

DR. MAYS: Here is what I would say, which is I know that we are going to be bringing agencies into our meeting, doing separate activities. I am going to say that probably what we should think about is if it is in conjunction with the subcommittee, can we sometimes put things into your subcommittee as opposed to us ramping up? If we really feel that that would help us do something, I would say yes. But I think to store it with given how we construct it that we are better at making sure that some of our questions are at your hearings and activities as oppose to us mounting them alone.

DR. SUAREZ: For purposes of everybody knowing, I have completed the draft of this combined work plan against the draft that will be distributed to everyone for us to take the next several weeks to review.

DR. MAYS: Can they print it out for the second half where we are going to discuss the plan again? Is that possible?

PARTICIPANT: Sure.

DR. MAYS: Then the last hour where we are discussing this, it will be helpful. Thanks. That is really great.

Paul, hi.

DR. TANG: I just have a question. Since I have not been able to attend some of the meetings, just tell me if this has already been covered. As I look at the work plan, I see we are supposed to make recommendations about how to promote and expand the access to innovative uses of the data. This has a lot to do with what data in a sense and what to disseminate.

When we talk to community — we focused a lot on what is the original problem to solve. What is the underlying need for which use of community health data can be input to decision making or action?

I guess what I am not seeing is the needs assessment versus the data-oriented view. Am I just missing it or is that something covered elsewhere?

DR. MAYS: No, you are not missing it. If you were here, you would see Damon. And Damon is the repository of the needs assessment.

At the very beginning, part of what we were talking about is in terms of our approaches to working. In the consultation process with HHS, they have identified some of the realms where they need some assistance about access and use. We will be bringing into this meeting different groups like we brought in NCHS to talk about it. It was NIHS, right? We will continue to do that.

But Damon has a better sense of those groups that are in a development phase where they have both time and budget. There is an opportunity to really impact what they are doing. We think that if we were to do this for long enough, we will cycle around to several.

Let me let Damon who is the needs assessment actually say something.

MR. DAVIS: There is still going to need to be a little bit of a scan across the department as to where the opportunity lies here to find out who is in a position to even take advantage of these sorts of consultative services that we are offering as a group.

DR. MAYS: But you will do that, right?

MR. DAVIS: I would welcome the opportunity to do that. I think what we would do is I would love to have Vickie or the appropriate representative come to a health data leads meeting. You can dial in and talk a little bit about the things that we have covered here so that they have an understanding as to what is being offered and therefore what the opportunity is for them. They then can say back to us you should probably talk to this group. You have probably already thought of this, but you should talk to these folks also. They will then begin to feed us a pipeline of places where we can plug in some of the offerings here.

DR. TANG: I still might be missing something. How do we address what people don’t know that they need?

DR. MAYS: Are the people you are talking about, Paul, like the people outside? For this and the work group, we are saying that our customer that we are working with is really for — is HHS as opposed to — if we are talking about what the community needs are though you might be talking about the Subcommittee on Population Health. If you are talking about some aspect of consistency in the definition of something, you might be talking about standards. For us, it is to respond to HHS. In the event that there are segments that do not know that they need this, there is a set of encouragement. There is encouragement around what they should be doing with their data. I think that is the kind of thing that is coming out of the IDEA office.

MR. DAVIS: Along those lines, Paul, it may be — I am not sure if you were on when we talked about this a little bit earlier. An idea arose that it would be valuable to have a place inside the department where people who are exploring what it is that needs to be done in the data space could actually find the useful information like the services offered out of this group, like who their main point of contact is for their specific operating division such that they — if you don’t know what you don’t know and you start to look on the HHS intranet, for example, right now you are not going to find anything anyway. Do we build out a space where you can allow these tools to rest so that they can be discovered and then people can expand their search to engage this group, engage my groups or others? Obviously, you cannot help someone to know what they don’t know if they don’t know you exist. We have to make them aware that these services are available. Part of it is just having a place where the information can reside that they can get to it.

DR. TANG: — one more try at explaining what I am thinking about — it might be totally covered somewhere else. I think it is in the scope of what we are talking about — let’s have an imaginary person who has in her mind knows every data source in the federal government, HHS or otherwise. And then they will throw out into the community and just listen to issues that come up whether it is in the city council office or the health care provider or the community groups or the neighborhood associations. Just understand it goes with all this knowledge of what actually exists in — goes up and listen. There would be a lot of needs — “unrecognized” needs by the people who were talking, but that this data guru would recognize those as needs where access to corporate data could inform that person or body or policymaking organization.

DR. MAYS: But that is not our customer. That would be someplace else. Right now, for the work group, not the committee, but for the work group, it really is to deal with HHS.

DR. TANG: Let me just throw one more sentence — then you can tell me — but this data guru happens to know that there is one data set that if we actually either added one more to the survey or reformatted, create another table, or combine it with SAS data, I would have the information that all communities in America need. Is that something that HHS should be interested in?

DR. VAUGHAN: I am sorry. I am going to interrupt. Our guest speaker is here. We could pause this and it could be picked up after his presentation.

DR. MAYS: Paul, are you going to be here for when we do the rest of the discussion a little bit later?

DR. TANG: Yes.

DR. MAYS: I will come back to you then. What we are going to do is transition. I need to unfortunately give people a quick break. Can we make this less than five so that you do whatever you need to do, let us switch over, get everything up, and then we will start immediately? I am told I have to give the staff a break.

(Break)

Agenda Item: Discussion with Health Data Consortium

DR. MAYS: Okay, everybody, let’s get started. What I would like to do — let me talk about what this time is about. We have talked a lot about HDC and we have participants in our group who also have done things with HDC. Because they have a very similar set of work to their activities that we have, we thought it would be a good time to talk. We would like to welcome Chris Boone who is very new at HDC. He will tell us how new.

But Chris, what I would like to do if it is okay with you, is to actually start by having us introduce ourselves to you and then letting you introduce yourself and talk with us. What I am going to ask is that as we introduce ourselves is that particularly we have individuals sitting here who are on what is known as the Subcommittees of the National Committee on Vital and Health Statistics. NCVHS is a federal advisory committee. Our role is to be advisory to the secretary in terms of health data, health statistics. We have HIPAA and we just also got a new duty, which is the review of ACA.

What we would like to do is when we introduce ourselves is to have each of those chairs to say what they do. It will be a little longer introduction rather than hi, my name is kind of thing. But instead, if they could share with you what their subcommittee does, that would be great.

Let’s start with Walter.

DR. SUAREZ: I am Walter Suarez. I am with Kaiser Permanente. I am a member of the Full National Committee and co-chair of the Standards Subcommittee, one of the three major subcommittees. What the Standards Subcommittee does is primarily focus on all the administrative standards for health care transactions so health care claims, all the HIPAA standards and HIPAA transactions, code sets including ICD-10, identifiers including the national provider identifier, and then something called operating rules. We cover the realm of all the HIPAA standards.

In addition to that, we have been just named by the secretary about three months to be the review committee. It is called the Review Committee under the Affordable Care Act to review the status of implementation of all the standards and how effective they still are.

And then on top of it, we also work besides that on standards in the realm of public health and population health messaging so health information exchange with public health agencies as well as other administrative and standard related topics including things like virtual cards, the use of virtual cards for payment between providers and patients. That is the scope of work of our subcommittee.

MS. KLOSS: Hi Chris. My name is Linda Kloss. I am a member of the Full Committee, the National Committee. My background is health information management and I am an independent consultant. I am on the Full Committee though not on this work group, but I have stayed because I always find the work group deliberations so interesting and informative.

The Privacy, Confidentiality, and Security Subcommittee of NCVHS originally was convened to help craft guidance material for the roll out of HIPAA legislation. Now that there is Office of the National Coordinator, we collaborate with that office. We also take up projects that we think are forward looking in terms of privacy, security, and stewardship of data.

One of the things we will want to share with you during your discussion is a work product that the committee just approved a toolkit for community use of health data that reviews stewardship principles and we hope provide some useful samples and examples. Our goal now is to talk to everybody about how we disseminate this and get it into use and get some feedback. We will talk to you a little bit more specifically about that hopefully.

MS. BERNSTEIN: Maya Bernstein. I work in ASPE for Jim Scanlon and I am the lead staff to the Privacy, Confidentiality, and Security Subcommittee.

DR. FRANCIS: I am Leslie Francis. I am at the University of Utah. I am particularly interested in bioethics issues involving privacy. I am a law professor and a philosophy professor there. I was co-chair of the Privacy, Confidentiality, and Security Subcommittee until the day before yesterday when my term ended, but I am staying on as the member of the work group.

DR. MAYS: I am Vickie Mays. I am at the University of California Los Angeles. I sit on the Privacy and the Population Subcommittee and then I chair this work group.

MR. DAVIS: Damon Davis, director of the Health Data Initiative here at HHS.

DR. COHEN: I am Bruce Cohen. I am at the Massachusetts Departments of Public Health. I am on the Full Committee. I am also on this work group. I co-chair the Population Health Subcommittee. We are on a journey exploring communities as learning health systems. The two particular spots we are in now are really thinking about community data engagement to improve the quality of community lives, focusing on the health data continuum from the individual through the community, and trying to provide some sense to understanding that whole range of data that are used for community decisions.

DR. VAUGHAN: Dr. Leah Vaughan, an epidemiologist based in California.

DR. ROSENTHAL: Josh Rosenthal, member of the Work Group. Good to meet you in person. Background. PhD Fulbright to the Sorbonne’s Institute for Advanced Studies. I am probably here representing health care entrepreneurs. I have been a member of a team that has founded and successfully taken all the way to (?) health care analytic companies using public data, most notably stuff around Dartmouth Atlas for unwarranted variation. Also, represent or can at least speak for some of the kids coming in, guest lecturer at Harvard, Hopkins, MIT around health care entrepreneur boot camps, how to take public data and actually make a meaningful impact and a meaningful business model about that. And then the latest day job actually use nothing but public data. Probably 120 different sources or so nationally with providers, payers, and different people trying to use it too. I do triple aim and ACA type stuff.

MR. CROWLEY: Good afternoon. I am Kenyon Crowley. Thanks for joining us. I am deputy director of the Center for Health Information Decision Systems. We are a research center based on the Robert H. Smith School of Business at the University of Maryland, but we work across a number of departments of science, public health, engineering and others to facilitate the design and effective use of information systems in health care. That extends across a number of programs for education development working with corporate partners, but they do have a strong interest and a number of faculty folks on health care analytics and how can we harness these various data sources across community to create value from it.

MS. BRADLEY: Lily Bradley. I am at ASPE, staff to the community. I have doing all sorts of stuff with policy and improving health care at the intersection of policy and business. I am returning to the private sector soon.

DR. BOONE: Thank you everybody for those introductions. It is definitely helpful to really know exactly whom I am talking to. I was telling Vickie that no, I do not have to go through the HDC 101, which I now have down pretty packed because everybody asked. We will just focus more on exactly what my vision is for HDC and then we can talk about what the natural synergies are for this work group and potentially what we are doing.

But I guess more formally, my name is Chris Boone. I am the new executive director for the Health Data Consortium. I am now on week five on the job and I have it all figured out. I promise you. It has not been totally like drinking from as they say a firehouse or whatever so to speak because I did have some transition meetings with Dwayne prior to starting. I did help structure some of the things. But if you know Dwayne, you know he is a man of many words and I mean that facetiously. But we did cover a lot ground and he did equip me to be prepared to hit the ground running.

This is really meant to be a more informal, unstructured, more of a dialogue and discussion. I do want to just share — you know that HDC has been around officially as an organization for a little less than two years. We have done a whole host of things. I am challenged with coming in and building the organization outside of — building a brand of the organization outside of just health datapalooza. We are certainly challenged with building a sustainable business model for the organization, but more importantly positioning ourselves as being the parameter source for open health data and being that non-government organization that works hand in glove with HHS as well as the private sector in order to make that happen.

As you know, we are primarily focused initially on government data sets. Hence, the reason Damon and I are BFFs. But we do hope that one day that many of these data sets will be from the private sector as well as you can think about clinical trial data sets and the like.

When we think about the opportunities that exist, we know that HDC can do many things. When we talk about open health data, there are vast amounts of opportunities that we can be engaged in. Certainly, it only makes sense that we become very targeted in the things we do just because of limited resources, people, money, and the like. But also, it is about being effective. It is about really differentiating ourselves from what many organizations are now staking a claim in when we talk about this world of health data. That is it as part of building some of the things that we do.

I would say one of the key aspects of the vision that I have is really about fostering a community both on a local and global scale. Now you may say what exactly does that mean because it sounds really good. I will tell you what it means. It boils down to really focusing on the grassroots level, really working in communities, and really talking about the use of data and how it is affecting targeted populations. We will see an expanded effort to work with cities, municipalities, states in order to increase their efforts to be in this world that we know as open government or open data.

We will be very open to international collaborations and really fostering that global community and hopefully it would be great that if HDC could certainly be the global convening platform for open health data at least.

Other things that we are certainly focused on are this whole thought of innovation. We know that there are a host of what I would say traditionalists in the world of health data or health IT, but we know that the startup community is certainly the catalyst for much of the innovation that takes shape. We want to foster that community as well. And my buddy Josh can attest to that.

We will certainly continue to do — expanding our affiliate relationships. And our regional affiliates obviously have been there. We certainly wish to or aspire to have more affiliate relationships in different regions across the country. If you want to help us with that, that is good. We want to clearly define some roles and responsibilities as it relates to how we are building our relationships with affiliates and what they really mean. They can be effective and they can be equipped to do the things they want to do at their local level.

Another thing is about molding the young minds of tomorrow, as I like to say, which involves really engaging with many of the universities and really trying to give opportunities to some of these young data scientists and some of the things that we are doing and maybe projects, internships, fellowships. You can call it free labor. I like to call interns. But however you want to describe it works for me.

But I am going to stop there for a second and really see if you guys have any questions or thoughts for me.

DR. COHEN: Chris, thanks. One of the things that we are working in the Population Health Subcommittee is trying to connect what we call data connectors or data intermediaries with data providers and with communities. Essentially this whole new group of organizations in the Health Data Consortium is I think an excellent example. Communities have not been able to use data directly because it is too obscure, not clear enough, and not comprehensive so they can use it. There are a whole bunch of health data connectors that have arisen to support community needs, community commons, county health rankings, a variety of organizations that have served a really useful function to create a better connection between communities and the folks who hold the data.

I am really interested in your vision of how the Health Data Consortium can provide value and my focus here is on community coalitions so they can be doing better community needs assessment so they can be doing their projects and programs to really improve their communities.

DR. BOONE: That is an excellent question. We had an affiliate town hall two weeks ago, a week before Thanksgiving and that question actually came up and are really common and what they would like to see HDC be more involved in. And really what they presented was we would like templates for how we can do certain things or even how we can engage in certain relationships whether it be if they wanted to conduct challenges on a local level whether it be if they wanted to conduct any type of needs assessment or readiness assessments on the local level and they wanted that type of guidance.

I think that for us the relationships with the affiliates grew organically. And the reason I am focusing on affiliates is that would be our medium for instituting something like this.

Where we are right now is really trying to focus on thought leadership, which involves documenting best practices and disseminating that information out to many of these different affiliate levels or local organizations. It is developing frameworks that would enable them to do. We know that with limited resources, we cannot necessarily work hand in hand with every organization across the country, but we can provide them with guidance documents that would enable them to do a lot of this work and be a resource to them to provide that technical assistance.

For us, it is something that we are trying to position ourselves to do, but it does take convening these different groups and really knowing what works and what does not work. That will be the first step. I do not know if that answers your question.

DR. COHEN: I do not understand your affiliate network. Could you explain that a little more?

DR. BOONE: We have a cohort of approximately 11 regional affiliates. I do not want to call them mini-HDCs, but that is what I like to look at them as. They are groups that are focused at the state level that had been historically focused on more the start up communities and it is just convening these groups and they can talk amongst each other and share what they do well and what they do not do so well.

Many of these groups have expressed an interest in formalizing that affiliate relationship and perhaps conducting regional datapaloozas at their levels, which has been something that we have explored and we tried to see if we are at a place where we are prepared to do something like that effectively. But historically, it has been mostly on those groups more collaborating and convening with each other and sharing best practices. We hope that it can be a more formal structure. That has been what the regional affiliates have been.

DR. COHEN: That is great. As the National Committee and as this group moves forward, I think there is going to be a focus on making better connections at the community level. We would certainly love to mine your expertise and connect with HDC to get those resources out at the community level. That is great. Thanks.

MR. DAVIS: Before your arrival today, Chris, there was a little bit of discussion about the palooza, which you have already brought up and the idea that a discussion of communities at the palooza and their utilization of data might be a valuable addition. I thought maybe since both topics have come up, it would be an appropriate time to talk a little bit about what the current vision may be at least at a high level for the datapalooza is and then perhaps invite the rest of that conversation about community engagement with data to see where if it does fits into this year’s organization of the palooza.

DR. BOONE: Again, that is another great question. I think that HDC is at a place where we are almost at adolescence I would say. Any time you are getting to that time, you have to start to evolve. We do not want to move away from the edginess that has been the culture of datapalooza. But we certainly want to evolve into a more knowledge-generating meeting, less about the rah rah cheerleading session that data is great because I think we have all accepted that, but more about how best to go about doing certain things and really starting to engage the thought leaders and the practitioners about how do we move forward. Obviously, the natural progression is to move the direction of — we talked about startups earlier and really convening those and having them, expanding on the whole notion of shark tank, but more educating them on what they can do.

Another aspect is something that you and I have talked about before where we talked about the uses of data as it applies to special populations or very targeted populations and how it is improving care or even reducing cost for those particular groups. Obviously, that would involve looking at different communities, which there is an overlap with focusing on targeted populations and the whole municipality, state-level initiative, grassroots initiative that I was explaining to you earlier.

The vision for HDP is to become a knowledge-generating source and really pushing the envelope when it comes to the uses of data. An example I would give is the evolution of clinical trials and the movement towards more adaptive clinical trials and how that is taking shape and really starting to explore what that means and what the innovative uses of data that are out there and that should be considered as we move forward. That is really the vision.

DR. FRANCIS: I made the suggestion earlier that at least at the datapaloozas that I have gone to there really was not very much discussion about stewardship or privacy practices. I wondered there was any interest in having a track to look at that kind of question. I know we did a session that the committee did several years ago at the datapalooza that involved communities and data use and data stewardship. I was wondering whether there would be any role for somebody from this group with our toolkit. If we wanted to raise that question, how we might go about doing it.

DR. BOONE: Let me say. When it comes to datapalooza right now, there is a call for abstract. You can feel free to submit any and everything you want. As I expressed earlier, my challenge is really to broaden or expand and strengthen the brand of HDC beyond even just health datapalooza. When you talk about an issue such as privacy or many people talk about the issue of information exchange or the lack thereof in many cases, these are all policy-related issues that we historically have not taken a position on, but we will. And mostly because we have no choice. It is directly related to our very existence.

Right now, we will tell you we are in the throes of strategic planning for the organization. Much of it involves considering what our strategic priorities are organizational priorities are for the organization and what platform we are going to — and the particular issues we are really going to focus on.

One of them is obviously patient privacy and how involved do we get is always the question. Another is data sharing, information exchange. I am reluctant to say interoperability because it has such a negative connotation these days, which is another issue that we will continue to explore. Obviously, our bread and butter, which is the open data directly and that is where it is. If you think about those three priorities, that is where we are focused.

MS. KLOSS: Just to go a little bit further with the notion of this toolkit, it is really based on a stewardship framework. We are not zeroing in on privacy as much as thinking about use of data and what kind of range of stewardship functions should be in place and checklists and that sort of thing. I think what we can do is share the toolkit. You can see what the scope is. I think it could be of interest to the audience. Like Leslie, we have participated in past datapaloozas and it is a topic that I think we have articulated in a broad enough way that it could be of interest to entrepreneurs as well as public health or others or communities.

DR. MAYS: One of the things that Lily just did was she had your website up and we also have the handout that you gave us. I want to go on the focus area and kind of see if we can have a discussion about that. When you say HDC will and you have open health data, data sharing and human-centered health system powered by health data, can you talk a little bit about what your plans are in terms of how to do that? On our side, for example, we are working with Damon’s office. What we are trying to do is make sure that this work group’s primary charge is to increase people’s access and use of data. That is everything from how do we make it more friendly to can it be linked and used even better. Our approach in terms of some of that is for the Full Committee is really sometimes we will end up going and having hearings. We have a question. We want to find out. Is there some gap in terms of a policy? Is there an issue that we can comment on? And then we respond up to the secretary with our findings. I want to just get for people a better sense of how you do our focus in ways in which then we might see if there are things that overlap for us both.

DR. BOONE: The key thing to focus on if you are looking at that handout is probably the foundational priorities or strategies, as we call it. That really gets into more of the how at a high level and how we go about doing things. Do advocacy education through raising awareness. Education is not — I know it is obviously very broad, but it does involve a certain level of educating the consumer patient as well. We want to be sure that there is — we all know that there is a discomfort when you talk about the use of health data for any purpose especially amongst the patient population.

We are obviously working very hard and diligent with Damon and his team at HHS to do that to obviously expand the number of data sets that we have on healthdata.gov to obviously identify additional data sets that may not be available and encouraging that they are.

We certainly have advocacy efforts as it relates to responding, but to a certain degree, we are not trying to position ourselves as a lobbying organization, but there is some advocacy emphasis that we have as far as commenting on certain issues. I shared a number of those earlier.

We also hopefully will get more — not hopefully — we will be more involved in more thought leadership activities, as I described earlier, where we talked about development of frameworks that would help states and cities who are interested in embarking on that journey. We will certainly continue to conduct many of our challenges for the startup community and hopefully even expand our efforts beyond that, which is equipping them with some of the tools and educations that they need to further the use of the data.

On one hand, we talk about deliberation of data, which is working a lot with the governments, the agencies, but then there is also the use of the data at the local levels. That involves really reaching out to the startup communities. That reaches out to the other researches that are interested in order to encourage that use of data. We are doing that just through multiple mediums. We have the challenge series that we have been doing. We are doing the webinars. We are doing obviously datapalooza that everybody knows about.

Right now, we are in the process of conducting a feasibility study for doing a multi-state challenge with California Health Care Foundation that would be involve one particular data set, how could it be used across multiple states if they standardize it and how would it move the needle. Really, it is more focused on moving the needle, liberating the data and moving the needle with the data.

I guess when it comes to these innovative ways, we are still exploring what other ways we can actually get involved with or engaged with that would actually assist us in our movement efforts.

I do not know if that really answers your question, but it is a very broad question to answer at this point.

DR. MAYS: I want to be mindful of your time and we started a little bit late. Do you have about five minutes for us to see if there are some things going on around the table here for you?

DR. BOONE: Sure.

DR. MAYS: Let’s just go around the table. If you put your tend up. Josh?

DR. ROSENTHAL: I basically just wanted to ask a question, two parts, one on your view of business model, not just for HDC, but for these entrepreneurs and a really specific question around how you might be able to help them out by lowering the bar to get into it. Just for context, you release weather data. You put it out. It is great that the government is not trying to do it themselves and you are going to let a thousand flowers bloom and that is fantastic. You do that with weather data and it booms. You do that with geo-location data and it booms. And you do it with health care data. And just looking at objective metrics with financial analysts, it is kind of a thud, meaning there are not a lot of startups that are successfully generating revenue. There are not a lot of startups that are actually having sustainable market impact, depending upon on how the market, whatever metrics you look at.

Part of that is around at least this is my take — it is a perverse market. It is not like weather geo-location where you can use the public data and everybody knows what it is. You put it on your phone and you have this cool app and it tells you it is going to rain outside so you bring an umbrella. I have done a couple of times myself, but then working on these entrepreneurship boot camps with kids at the universities. They come out and they basically say I have a great idea. It is a way to keep people out of hospital beds to reduce bed rate. I am going to walk into a hospital CEO and tell him I have a fantastic protocol to keep people out of your hospital, which in a fee-for-service model is like walking into the Hilton and telling people I am going to keep people out of your hotel or walking into Chevron and telling them I have a water-powered engine. There are these perverse incentives floating around. It is not like personal take. That is like Dartmouth Atlas supply driven or whatever.

There has not been a resounding — the entrepreneurs have a tough time figuring it out. It is not a data issue. It is as much a business model issue and applying health care data to perversely incented sometimes ecosystem. And then part of it is — the legacy folks — public data potentially just intermediates them. A good friend, someone we work with, David Winberg, Dartmouth Atlas for unwarranted variation. He writes a little article in the British Medical Journal a month or two ago. He says this one public data source that this committee puts out, behavioral risk factor surveillance system. It actually out predicts risk stratification prediction than any claims-based system, than any EHR, than any PHR. If I am a legacy big analytics company, I put a lot of money into doing this claim stuff and I have this little tiny data set that any kid could download and out predicts my stuff. I might be somewhat hesitant to participate in this in terms of if I am not that innovative and it is going to disrupt me. If you put the public data out, objectively most analysts would say there has not been a really great response in terms of startups getting into it partially because it is perverse, not in a real market way for startups. Partially because it disintermediates in a real specific way.

What I am always trying to think about in this committee is how can we make it easier for these startups. There are some particular ways of lowering the bar we might want to be thinking about, some of which you have already said like best practices, like putting out basic metadata, like putting out basic — relationship diagrams, stuff that you do outside health care all day long.

I just was wondering if you could us your thoughts on how you think about not only HDC business models, but helping these entrepreneurs beyond a challenge. Because you get a challenge, a couple of hundred thousand bucks and then what do you do. You do not have a business model. How do you actually make value out of it and then how do you avoid disintermediating the bigger players and then how do you see some of the things that this group does as a way to potentially lower the bar to get these thousand flowers to bloom just like they did in weather and geolocation.

DR. BOONE: When it comes to the business model of many of these organizations, that is more of an exploratory venture. What we have is convening power. And certainly, what we want to do is build more of a platform, an infrastructure that would enable these guys to really get together and share those types of key lessons learned. If you think of something like south by southwest where these guys just really convene and they just network and they do all these things. I think that is what we have the power to do.

But when it comes to many of the — just an initial observation when I think about many of the startup organizations, it seems to be more of a BDC type of business model. I wonder. I think the opportunities are more in the BDB. Certainly, you start thinking about what innovative approaches can you take to be more BDB and less BDC. That is just my observation. If there are ways that we can actually capitalize and encourage that type of development then I think that that is where we will focus.

DR. ROSENTHAL: Is that what you mean by evolving going from adolescence, let’s build a cool lab —

DR. BOONE: Absolutely. Let’s talk about usability and get into some real needle movers. And frankly it is great to be another — the opposite of this is — knock on those that choose to be a consumer app that is focused on I am going to measure your heart rate, but I think that Apple and these guys have really pretty much taken that market over with their own Apple Health and some of the wearable technologies that exist out there. I think there is a huge opportunity and more focusing on how do you use the data for research purposes and really enabling more of these opportunities that exist. Obviously, life sciences are a very ripe area. They have a particular interest in patient engagement. And maybe that is something to consider. It is different things to really explore that are non or unconventional opportunities and figuring out what works. But I think that involves really understanding what the needs are out there, really bringing in a diverse group of individuals who could introduce problems to you that you could potentially solve with open data.

DR. MAYS: Let me just ask a question because I am trying to find where we are probably going down parallel tracks and what we might be able to see if there is some way that were enhanced value to each. It sounds like you have a network in terms of the 11 groups that you have out there. I am wondering. We tend to have hearings. We tend to need to outreach to people. I am just trying to get a sense of whether or not that is a place where we have questions or issues. If we sent it, whether or not those groups could respond. If we wanted feedback on something and we said part of what we are trying to think about is to respond to data linkages to the secretary whether or not you have a way. If we were to contact you, you would have a way to be a voice for these 11 groups or something like that.

DR. BOONE: Ultimately, that is what we want to do. We are actually holding a number of different summits. Obviously, with these different targeted groups such as the startup community having a summit with these guys and really hearing what they say, that would enable us to really have the information to come back to you and say we have convened these guys. Here is the report from that actual meeting where we talked about all the different challenges that are associated with data whether it be — everybody complains about the challenges with HIPAA and the fact that it needs to be I guess lack of better terms modernized in a way to reflect current technology and some of the things that are out there. Identifying specific technical challenges and certainly we can provide that information to you.

We want to be — I do not want to say we want to triage issues, but essentially that is what it is. We want to be able to triage certain requests or concerns from these various communities or stakeholder groups and bring them to you. That is how I see this relationship being fruitful for you and for us as well.

DR. MAYS: Great. What I want to do just in respect of time, could you each say what your question is and then let him answer it that way? Why don’t we start with Kenyon? Say what your question is and then have each of you say what your question is and then we will have him answer.

MR. CROWLEY: Your comments about creating research assets for students to utilize peaked my interest. I work at the University of Maryland. I know there is a need for more — I think we call it innovation sandbox for students to be able to access data and use that and share it. I would just be interested to hear your thoughts in that area and then just to offer my assistance if you do decide to pursue student engagement.

Both of our student data analysis classes in the Smith School and the School information sites, both were oversubscribed by 100 percent. Students, I think, are getting that this is the new world. This is what I need to know.

DR. SUAREZ: I do have two more focused questions. Any work done around metrics to evaluate the usability and the usefulness of data. I found it interesting that I was looking at some data, some Internet resources and there is usability.gov actually maintained by the Department of Health and Human Services. It is usability.gov. They talk about all the usability that you want to know about. What is usability? All sorts of things. It is an interesting website. I did not know if you had developed some criteria, some sense of the usability and usefulness.

And then the other question is about establishing and I think you also mentioned some of that — establishing standards for metadata of the databases themselves and providing those so others can use them.

MS. KLOSS: Mine is a simple question. In your resources or website, do you link to other sites? Could we do some linkage to work products of this committee? Even though that is a little more passive way of collaboration, it might be an interesting way to see what interest there is in some of those topics.

And then do you do any third-party webinars? Are you doing any of that online education?

DR. ROSENTHAL: What do you think of the job so far?

DR. BOONE: I am still here. I love the job. I love my job. I have no complaints there.

Let’s start first with the university of relationships. First of all, if you are going to volunteer, I am going to take you up on it. I am just going to tell you that now. Again, it goes back to free labor.

Right now, when it comes to university engagement, our partnerships, at least I have not clearly defined what that means yet. I just know that it is something that we want to do and certainly working with you to really conceptualize what that means would certainly be imperative because you can tell me what the needs are. I can tell you what I am thinking. We can somewhat figure out the best model for what works and that becomes mutually beneficial.

We do know that this whole job of chief data officer seems to be something that many of the, I will say, millenials seem to be interested in. It certainly furthers our mission. I am supportive of that and I want to figure out the best way to make that work.

When it comes to metrics around availability of data, we have not gone above and beyond and I think you guys were talking about it earlier at OSTP with their criteria that they have established for open data and how they define it. We have not gone above and beyond that to really establish any metrics for usability. I certainly think that is a wonderful area to explore and potentially research.

When it comes to standards for metadata, we are in the throes of planning for a summit that is focused on data governance where that particular issue will come up and be discussed. Obviously, you can never resolve that in one meeting, but certainly, we can initiate the dialogue to get that ball rolling.

We do link to other websites and certainly have no problem with linking to any resources that you guys have available. We do host webinars and where we engage third parties to be featured presenters, speakers and what not on those particular webinars.

I have already answered your job satisfaction question. Now my approval rating is a different question. Are there any other questions?

DR. MAYS: I think probably in terms of what I heard is does seem like there are some synergy things that are there, things that we greatly appreciate in terms of the space that you are in and a way in which we might be able to have that kind of relationship.

I am wondering for you if there is something in terms of the different communities we have, the things that we have talked about that seem like they might be useful to you. We are a FACA. There are some things that we can do and some things we cannot. But for the most part, in terms of the usefulness of some type of information source, ways in which to bring issues, ways in which to push issues out, do you see any value added in us having — share that with us. We can see it. They are smiling so we can see it.

DR. BOONE: I was not familiar with this group prior to Damon bringing it to me. Knowing that this group exists and that there is an opportunity to actually push issues and actually get that direct feedback from you guys is certainly a value. I think it would be a value to even our members and stakeholders as I can push that information directly to them.

One of the advantages they find for HDC is that we have such a close relationship with HHS and the various committees. I want to leverage that in every way we can in order to get that direct feedback and take it back, but also bring information to you guys so you can really hear I hate to say what the field is saying, but what the people out in the world are saying outside the beltway.

I have a particular interest in all the different subcommittees that everybody is a part of and learning more about that. It just may be getting access to some of the meeting agendas. I am interested to hear what some of the key discussion points have been in the past. That will be helpful.

DR. MAYS: Great. Last word.

DR. SUAREZ: Now that we are getting close to the end of 2014 and getting ready for 2015, what would be — you will probably cover them, but what would be your top two or three major issues or areas of work that you see are the challenges of 2015 for health data initiatives?

DR. BOONE: I would answer that question, but it may be a bit premature. Can we talk about this in two months? I will give you my top issues in 2015. We have talked about some of it already. I think for us as an organization this is obviously not confidential, but we have to build our sustainable business model and we have so much area to cover. We will have to really identify critical areas of focus that give us maximum output. It becomes more of the 80/20 rule. We can do many things, but it does not make sense to do it all and it does not make sense to do it all right now. We have a prioritization challenge. That becomes a bit sticky because we have such a diverse group of stakeholders that are interested in many different things. We want to be able to address those things, but it is next to impossible for us to do it at this point. We are going to have to identify what the key priorities are that advance what our mission is today. Those will be the things that we focus on.

DR. MAYS: One of the things we will do is I will ask Lily to send you what all our little URLs are where you can find our reports. You can find letters that we have sent to the secretary. You can find — I think our plan will actually be up. You can see whatever it is that we have online what we are planning to do and then there can be a follow up from that if you want.

PARTICIPANT: — live somewhere where people can just easily access.

DR. MAYS: It does live on the website. We have a new website. The only reason I was saying that I was going to ask Lily to do is because it is a brand new website. I do not know if you have seen it. But just to make sure that to navigate it and find it. Then you can get a sense of — there is our new website. That stuff will live there. The work group will get good about where it is stuff is going to live as well. You want to get familiar. I just want to make sure that Lily can direct you right to where our stuff is.

I want to thank you for taking time out of your schedule. I apologize. We are running a little late, but I want to really thank you for coming. I think that this has been a treasure for us. I am glad that it was a suggestion that we get to talk to you when you are early in this process and you get to know us. We have to know about you as well. We are going to really extend a welcome to you and your job, but also a welcome to you in a sense of what it is that we do and to feel free to send us emails to comment, to contact us. I am sure Damon is also available to talk about it at the level beyond just the committee. Thank you very much for your time. We wish you well in your job.

DR. BOONE: Thank you.

Agenda Item: 2015 Strategic Work Plan Discussion

DR. MAYS: What we want to do is to go back to our work plan discussion. Part of why we want to begin it early was so that if there were things we wanted to flesh out before Chris came, we could do that. But I think we have a pretty good sense of where we are headed.

Now, what we want to do is actually talk about doing the work. We have talked about it at this level of this is what we are doing, but now to say who is going to do what. I think Lily has our timeframe up there that we can also work on.

And then I want to make sure I save some time to do the convergence even though I think we have been doing it along. Subcommittees may have a different idea, but I do not think we are too far off.

And then I know that you want some time to be able — that is why we put you up there first to be able to talk about your launch.

MS. KLOSS: I just want to brainstorm.

DR. MAYS: Yes. Now, we are going to talk about the mechanism in terms of actually doing the work plan. We have suggested two things. One is the guiding principles, which I think we have had the most discussion on and the second is the data dissemination assessment tool. Here is what we have done so far, which is we brought in NHIS. They have agreed to be our little test case and they just told us things about their data set. We want to store it up coming with questions that we want to send to them before they come. This is getting at this whole notion of what the assessment tool would be.

We want to do this in a sequence. We want to secondarily start at talking about what kind of metrics that we would want to use in terms of them beginning to think about assessments that they would do so that they can get some feedback on the usability of the data set.

We tried to break the group into two separate groups. One, people that would work on coming up with guiding principles and people that would work in terms of the assessment tool. I do not think everybody got a job. We want to make sure by the time we are finished that everybody has a place in which they are working.

Let’s start by talking first about the assessment tool as opposed to the principles because I think we have talked a little bit more about the principles. We actually have a starting point in terms of the principles. I do not think we have as much of a sense of what we want to do in terms of the assessment tool.

Here is the purpose of the assessment tool. The purpose of the assessment tool is to work with the various HHS groups and find out what they are doing in terms of their data access and usability. The purpose of that is for us to be able to give them feedback about ways in which they might be able to tweak, change, add, enhance, whatever the words are that you want both their access and use.

We also want to use this tool to have a sense of being able to look across HHS in general and to try and learn whether or not there are some patterns that we can ascertain that certain data tends to be more usable for group X where other data tends to be easier to use for group Y so that we begin to help HHS not just by doing this with each group, but that we are doing it by looking at — if a new group were to store it, we could say from jump street, if you are new and you are doing data X, it is very easy for you in terms of access and use with this. But you may want to think about these other things. What we start doing is being a little bit more proactive. The assessment for the group is what they are doing now. But what we hope to get out of the tool is to be able to leave some information for HHS about as new things start and as they use different mechanisms. For example, getting data out by social media or getting data out in some other way that people do not typically do that we have advice because we have learned it from using our assessment tool.

The assessment tool is here and now. But we want it to have implications going forward. Does that make sense? That is the goal.

DR. COHEN: I was initially charged or volunteered to begin this process. I started devising a data dissemination assessment tool and I will send a draft around and folks can — if this is where we want to go, that is fine. If folks have other ideas, that is fine too.

There are essentially two sections. One is the availability of metadata and tagging. And the other is release strategies in terms of data dissemination. Some of the concepts or questions for metadata tagging include content, variable, shared definitions, format, periodicity, refresh rate, interoperability. Some of the kinds of release strategies moving forward are modes of distribution, which is static reports, individual-level identified data, individual-level de-identified level, aggregate data, web-based query system, use case examples, availability of technical assistance for use, target audiences that we discussed of entrepreneurs, researchers, individual consumers, communities, other government agencies, and other characteristics such as suppression requirements, data use agreement requirements. Another release issue is how they capture feedback from users, evaluation, metrics for usability, and finally novel or emerging dissemination strategies via social media infographics, integrated stories, et cetera.

I have developed a framework of questions to circulate around assessing the state of dissemination for data sets in HHS. If this is valuable, I think it will be not only valuable to get feedback for us, but maybe this will work for characterizing data sets that we can eventually roll into some kind of postable document.

How many people want to play in the sandbox with me? Walter. Kenyon.

MR. CROWLEY: I will jump in your sandbox.

DR. SUAREZ: Just a couple of comments. I think it is great the two categories. Two suggestions. I would think that in addition to availability of metadata, one of the first parts is really availability of data itself. How available is the data itself? And then one of the characteristics is does it have metadata to become available?

The other suggestion is in addition to the release strategies, I would probably call it more dissemination than release strategies, but in addition to that, I would create a third category, which is usability because I think that is a separate element. In fact, the two areas to focus on from your introductory comments as I noted them was the purpose of this assessment is work the various HHS work groups to see what they are doing with respect to data access, dissemination practices, and usability. I think a very specific category with questions that we would create is usability.

DR. COHEN: Great. This is just a really very — but it would be great to build on it. I am going to be gone for the rest of this week. Next week I will type it up and send it out.

DR. ROSENTHAL: I do not want to be in a group, but can I just share things?

MS. BRADLEY: I was also just thinking that as you go through the assessment tool, there might be things you could leverage. Did you list it on healthdata.gov? It is actually a great place.

DR. COHEN: Try to read this? I will be back Monday.

DR. MAYS: Lily, I am trying to find my list of who is in which group.

MS. BRADLEY: It was Josh and Kenyon and then it was Leslie and Bruce and then others TBD.

DR. MAYS: Leslie and Bruce were in the group, which is actually trying to pull together the assessment tool. Josh and Kenyon were in the group about the principles.

DR. SUAREZ: The fourth category I would suggest actually, which is really part of what needs to be done with respect to data is usefulness. There is availability, dissemination on release, usability and usefulness.

DR. MAYS: Can you talk about if I am asking an agency about the usefulness, what would I be asking them? I am saying it because everybody is going to say yes, it is very useful. But what is it I really want them to tell me so that I can also judge usefulness?

DR. SUAREZ: The first question is how do they capture feedback regarding the usefulness of the data. Do they capture it? Just like do they capture feedback about the usability, how easy to use the data? Do they capture information feedback about how useful the data was to achieve the goal? What were they able to do with it and was that what they were expecting? Did it fall short or did it meet expectations?

MR. CROWLEY: In essence, you can almost develop a set of usability heuristics for open data. There are well-defined heuristics for system effectiveness and usability such as Nielsen’s heuristics. And there are other folks who have taken a shot at applying those towards data use. Part of the goal of that could what are those heuristics for usable data.

DR. MAYS: Are there best practices about that that we might pull up in terms of that that you might share with us?

DR. SUAREZ: In terms of evaluating usefulness?

DR. MAYS: No, he was saying that they were doing it at another level and that now they are trying to do these same heuristics in terms of the data itself. I was saying whether or not there was — if this is being discussed or are there recommendations of what to use.

DR. CROWLEY: I am not sure how well developed it is and there are people looking at it. I am saying that would be one approach is to use some of the existing heuristics around system design and also some applications for use of systems with data.

DR. MAYS: In terms of the system design issue —

MR. CROWLEY: Jakob Nielsen. He is the most cited person in that typically.

DR. MAYS: Lily and I will work on that.

In terms of the start, I think it is a great start. I think it is a great way for us to start to build it out. We have, I think, a pretty good start in terms of the other side, which is what some of the principles are. Let me also ask. Are there any others in terms of the assessment tool that we want to throw in? I think what we are going to do is pull these two things together and see what we have. We seem to be going pretty well in terms of the development of them both.

Thank you, Bruce. I think that was a great job.

DR. CROWLEY: I will just say as we put together, why don’t we put it together in a shareable document.

DR. MAYS: Yes. We are going to use the site that Lily created. It will be easy then to even make adjustments. It was a Google doc site.

Let’s turn to the information that we have around the other things that we came up with about the guiding principles. Let’s just go through that and see if there are some things we want to add. For this, we talked about stewardship. We started off with that as a principle format, metadata, social tagging, data communities, top-down strategies, and caveats for the principles.

Could someone tell me in terms of — I do not remember who actually generated data communities. What are we talking about in terms of data communities towards the principles? It was Kenyon.

DR. CROWLEY: I think both Josh and I and other people — data communities – the different stakeholders around any particular data source whether that is the users, the producers, and users may be across a number of different categories that they have a place to interact with each other and the data and having a feedback loop around their experiences with the data that can inform the future development of the data and the resources around it.

DR. COHEN: Community of practice for data users.

DR. MAYS: Did we just converge with them?

DR. COHEN: I am translating it into terms that I use.

DR. CROWLEY: It is a community of practice.

DR. MAYS: We have convergence.

Are there questions or comments about any of these seven that we have? I think these look like good starting points. We will flesh them out. We will try and see how they work.

What I want to do is also to determine how to do these use cases because I think it is one thing to just have these, but it is like how we are going to add dimensions to them in terms of what these levels are. We have talked about it, I think. Linda, you were helping us. I remember that Bruce had one level and I had another level. Can we talk about that?

MS. KLOSS: I actually hadn’t seen these. I am just seeing these for the first time. It seems like there is maybe one that we could add or one category of principles around cautions or something. I thought that was caveats to the principles. I think that there are cautions in the data. If they are being accessed and used —

DR. MAYS: Early releases —

MS. KLOSS: It is known to be incomplete because the goal of timeliness trumped the goal of completeness or other kinds of accuracy or integrity characteristics of the data that need to be brought forward. So people just don’t have to discover that once they get into it. They have some clue that the originator understands they are making this data available. It is not perfect.

DR. MAYS: I think that is really important relative to the push for the releasing data, provisional data. I think that we are going to see the department really need to do that sooner and sooner. It does have to have a caution. It might be useful to have the caution be early.

MS. KLOSS: I can almost even envision you could in the principles, describe levels.

DR. ROSENTHAL: It is like levels of competence and the tradeoff can either be like expediency or its levels of competence for a particular target within the data. If you are going to do top-down, you are going to gain benchmark, but you are going to lose some specificity versus if you are going to do specific, you are going to lose some benchmarks. There are levels of tradeoff and getting the data out versus competence. And then there are levels of what your objective — I release this data set with the goal of national benchmarks. All right. I am going to lose the 5 percent in the tail like for specificity, but I am comfortable with that because that is the aim.

MS. KLOSS: I think a principle would be that you would need to reveal any idiosyncrasies. Let’s say the releaser excluded certain cases because of small numbers. In the middle of this data set, the United States transitioned from ICD-9 to ICD-10. It has mapped so it is not as good as if it were all in one.

DR. ROSENTHAL: And it sounds insane, but what you see when you are using the data is you see these blank cells and your question is is this a blank cell because of the idiosyncrasy that is known or is it a blank cell because it is just whatever happened. You have no way of knowing that. All you need is a thousand percent.

DR. MAYS: I agree. I think that is good to add.

DR. SUAREZ: One comment about when I was reviewing the principles — of course, when dealing with guiding principles and just principles in general, there are different levels and scope. There are some high-level principles that are overarching and more generic, if you will. You can create usability. Actually, I was looking at a table of the ten topics on the bottom of Page 2 there. Inside those are the many possible things.

And then there are guiding principles that are more granular, almost structural requirements. Those are the ones that I thought were more like the ones listed in here. For example, formats or metadata. Those are to me at a level that is a little more granular and more technical, if you will, than guiding principles that are more generic and overarching. I do not know how to call them. They are more high level.

DR. MAYS: One of the things that they identify is that HHS does have them and I think it was in those two days. I think we are going to pull that and see if those are like the overarching ones. And then we are getting a little bit more granular with my sense. Lily.

MS. BRADLEY: I just have a general feedback. We have gone through different iterations of this brainstorming, but just generally, when I look at a list to evaluate whether or not I think there is a coherent message and it has been well thought through. I am looking for a list that has the same type of thing in it. It is sloppy work to have something that has car and van and truck and train and then all of a sudden you have an apple. What was this person thinking? They haven’t thought it through. It is that kind of analogy that this has very different kinds of things. Right now, until it is parsed out and things are categorized and sorted properly that it does not carry the same weight so that process.

I do not actually have a particular preference about where it is going to be or where you should start. I just think that wherever it lands, it should land in a very kind of sane type of thing in the list. It is like parallel radical structures.

MS. KLOSS: You could write a principle that says data will be released with description of any cautions to how it is used, known cautions about quality or completeness or timeliness and write it at that level, which is one sentence or you could be much more specific. I think you are probably going to stay at that high level at this stage.

DR. ROSENTHAL: Some of it will be consolidation. I think some of what is going on as an illustrative example as kind of replacing axiom. When we go through and sort it, we will probably have to have some illustrative examples. When I say the statement for shot thought should be given to frequency. That sounds great, but nobody knows what it means. How I would envision it potentially is axiom illustrative, specificity if you want it, illustrative example, potential analogy and then use case application by the four different types. However you structure it.

DR. MAYS: Do that again because that may be something for us to think about.

DR. ROSENTHAL: Axiom or call it a guiding principle statement. First principle axiom and then specificity if you want to — Walter goes through and makes it real and detailed. And then illustrative example. When I say refresh rate and people won’t use it. Part B and Part D. Great fanfare. There is a great hue and cry in the market. Should I use this if I do not know if it is going to be refreshed? Then analogy. What do I mean by that? The speeding example earlier. What do I mean by method as part of metadata if I do not know that 75 is faster than 25 and I am trying to figure out how fast I should drive it does not help me. And then potentially use case and multiple use cases for the four types or personas or whatever you want to call them. I am not saying we have to do that.

DR. MAYS: Again, I want to put that in the brainstorming.

DR. COHEN: — actual health data. So your principles. Start at the beginning. The axiom.

DR. ROSENTHAL: Okay. Axiom. When releasing the data set.

DR. COHEN: Let’s go with Linda’s examples. You need descriptive information related to —

PARTICIPANT: The example we gave was explain any blank fields.

DR. COHEN: Let’s talk about mortality data. The explanation. The convergence to ICD-9 to ICD-10 created discrepancies and you need the comparability ratios for trend analysis.

DR. ROSENTHAL: Yes.

DR. COHEN: Next.

DR. ROSENTHAL: Illustrative examples. That is kind of the example.

DR. COHEN: When you are looking at Alzheimer’s rates, there is a huge difference because that was not a viable code in ICD-9.

DR. ROSENTHAL: Analogy. I actually see this as a mad lib game honestly for applications. I am not joking. I am totally serious. The data owner does not say what do I do. You literally fill up the mad libs. But it makes it real.

DR. COHEN: Difference would be hospital discharge data does not use ICD-10. It only uses ICD-9 so looking at trends and hospital discharge data are different than looking at mortality data. I do not know whether that fits your analogy.

DR. ROSENTHAL: That is a contrast or some sort of explanatory rubric, analogy, or compare, contrast and then use case. If I am community member or if I am releasing a set for a community member, what does that mean?

DR. COHEN: It means when all of a sudden you see an epidemic of Alzheimer’s disease. It is because of coding changes.

MS. BRADLEY: I would still say that is a different — you have to still make sure that the axioms are of the same —

DR. COHEN: That has to be at the same level of abstraction. They all need to be —

MS. BRADLEY: You cannot go for metadata to we should encourage communities of practice across —

DR. COHEN: Yes, you are absolutely right.

DR. MAYS: It is a good structure to at least begin with to see if we can do what you are saying.

DR. ROSENTHAL: You have the seven things and then you do a mad lib. I am a data owner. I am going to release this like blank for blank. This is a little more abstract.

DR. COHEN: First of all, we need to evaluate these seven to see if they are —

DR. ROSENTHAL: I should warn my users that blank are known issues. As far as refresh rate, I anticipate blank. I am not phrasing it well, but you know what I mean. They literally fill the thing out like fill in the blanks, mad libs.

DR. COHEN: I think we need to take Lily’s precaution seriously. Are these seven concepts at the same level of abstraction or can they put at the same level of abstraction?

DR. MAYS: I think that that is probably I think some work to do because I think — it is interesting is that I think Lily has a very good point and then your structure allows you that you all are talking about. It is like doing Josh’s mad lib. If you work it through, it helps you to see if it is falling apart to some extent.

DR. ROSENTHAL: Jim and Greg. We came up with some mad libs. This is what we are going to do. We think this is where — fill up a mad lib as far —

DR. MAYS: Did you ask about mad libs —

MS. KLOSS: I do not know what it is.

DR. MAYS: I want to make sure I go back.

DR. ROSENTHAL: Fill in the blanks.

DR. COHEN: When you have little kids in the back seat of the car, you give them this story that has blanks and they fill in stuff to create their own story.

DR. ROSENTHAL: A twist on that. Pretend it is fill in the blank. For known issues like they will have to watch out for blank are known issues. But then you have a list — missing data, mismapping. Then you have a list. You kind of help them. I am a data owner.

DR. COHEN: — mother’s house and grandmother gave blank.

DR. ROSENTHAL: The benefit is if you persist all the answers — so I am a data owner. I say this answer is blank business questions.

DR. SUAREZ: They can actually click on the space and it opens up a whole bunch of options.

DR. ROSENTHAL: Something like that. I have done it with little cards before. You literally say I am going to use blank data source and you list the five things or I have to watch out for blank. I am a data owner and I have never actually asked the question. What question does this data answer? I can have 10 or 20 or 5 options and say this helps reduce cost. This helps pick — you lower the bar for the data owner to actually fill out the mad libs.

DR. MAYS: When you do this, it really makes you throw things out and think about it. It is used sometimes in terms of I do not want to say it is just brainstorming, but it really pushes you to make a case that makes sense. The more creative you are, the more interesting it gets.

DR. ROSENTHAL: Isn’t it on your bucket list? You do not want to do a letter of recommendation. You want to do a bucket list. Here is the user guide. Mad libs.

DR. MAYS: I think that we have a very good start on the two pieces. I think we have a good start in terms of what the guiding principles are. I think what will happen — what I would say our next steps are — when Lily and I can get a good set of the draft of these two things, we are going to send them out. Some of you need to say what group you want to be on, whether you want to be on the guiding principles group or a subgroup or an ad hoc group or whatever we want to call it and some of you want to be on the assessment. Mo, which one would you like? Bruce? Bruce did a very good job. We know which one he is on. Linda? We have put Leslie on the — which one do you want to be on? Do you want to be on the other one?

MS. KLOSS: I would like to not be on one.

DR. MAYS: Okay. We will do a pass for right now then. You have to get your toolkit out. Walter?

DR. SUAREZ: I think I mentioned assessments.

DR. MAYS: We are good then. I think we have done fantastic with pushing this along.

Agenda Item: Convergence with NCVHS Activities

DR. MAYS: Let’s try and go because it will give you just a little bit more time. Let’s try and move to the convergence with NCVHS activities. The reason why I put on this schedule is that we are the work group and while our primary customer is HHS, we also want to be helpful to the committee itself. Part of what we want to do is ask the committee chairs the ways in which we might be able to do that.

We are going to let Privacy go first because we can be helpful today to Privacy. Linda, why don’t you talk about what —

MS. KLOSS: I am just going to send this to Lily if I may. Lily, can you pull this up? I am sending you —

DR. MAYS: I think you have to send it to Janine because that is their —

MS. KLOSS: Janine, you have it, right? The latest toolkit? Can we bring that up?

Let me set the stage. Two years ago at the end of 2012, the Privacy Subcommittee wrote a letter to the secretary outlining a stewardship framework for community data use. It was a letter with stewardship principles essentially for the scope outside of HIPAA.

And the feedback we got from the roundtable in 2013 suggested that it would be helpful to these communities if they had some case examples and they had some resources on how to apply these principles or this toolkit. The committee approved last month a toolkit. It is still in draft form. We are still polishing it.

But one of our goals — because it was a lot of work, we think it is also valuable, but we are looking for broad dissemination. Our question to the work group is to help us brainstorm on ways that we could disseminate this toolkit whether it is splitting it apart electronically by linking to the Health Data Consortium, through the university programs. I think we are just looking at ways to get this out. It is not going to do anybody any good if it is on the shelf. It is a 40-page document. It is not simple although we have tried to write it in a non-technical way. It is not written at an eighth grade level. It is really written at a level where somebody advising a data user could use it as a checklist and so forth. If we can get it brought up, you will see what I mean.

This came up yesterday that we were going to do a communication plan. Vickie had suggested that we do a more formal launch of this and we thought that this was a perfect time to add to our list of ideas for how we disseminate this as broadly as we can.

DR. MAYS: Let me just add a couple of pieces. One is that we have a new website, a beautiful website. Debbie and all the people who are responsible for it, it looks wonderful. I am assuming that we can do things like make announcements about new things coming up within the department. Damon had to go take a meeting, but within the department, they have — Lily knows. Can you say all the social media in terms of ways in which you can announce things. You have Twitter accounts. What else do you have?

MS. BRADLEY: We have social media accounts. There is a full registry we could look at which ones. My first question would just be which audience we were trying to reach. There is something official that Debbie — we use GovLoop, don’t we — GovDelivery platform.

MS. JACKSON: Yes. It is GovDelivery.

MS. BRADLEY: People who sign up.

MS. JACKSON: Katharine Jones spent a lot of time testing — an effort pulling — we are hoping that we can get our products out in more expeditious kinds of ways so that people will know it is coming. We do GovDelivery.

In the agenda book, in the e-book, you will see in the minutes that we already have a list that you were pulling together, Linda, for the traditional contribution for organizations. That is where we will start and making sure that we are covering our basis. Then when we get to the website, then that would be even more of a push out and launch. We are looking at from this group that has more innovative insight of connecting to people, just that second and third kind of level of push out and launch for this toolkit.

What we know is also going to be a bit different than our regular reports in that you are looking for some kind of interactivity and another level and stage. But for right now, it is a document. It is an item that will be going out as far as the report. The main thing is to get this on people’s radar and it can continue as we go along.

MS. KLOSS: Could you go to Page 2, Lily, and just increase the size? Let’s just look at the table of contents.

DR. MAYS: The specific things we are looking for is what they would like to do is to have a launch to actually do a media blitz to announce this is available to try and answer what Lily was asking. This was actually in response to having had hearings in which individuals came in that were community stakeholders. They want to make sure they get it out to community stakeholders. Is there another specific population?

MS. KLOSS: Again, community, definition of community. Who is that? And realistically in many communities, this kind of support to community health initiatives may be coming out of universities. It may be coming out of public health. It may be coming from a variety. What we would like is to get as much exposure to this. You can see that it begins by describing the fact that if you are taking responsibility for data, you have to look at it over its life cycle. It defines what stewardship is. And then it goes through principles including accountability, openness, transparency, and choice, community and individual engagement and participation, purpose specification, data quality and integrity, security, and de-identification.

Ideally, the way I would love to see this on a website is that you could click on the topic and just read that and so if you are concerned about de-identification. It has three or four pages about de-identification. It has a number of references. And then there is right now as this is formatted, there are a series of checklists that if your advising a community, you could make sure you have dotted the I’s and crossed the T’s that you need to do. But if it were a live document with hot links, you would bring up the section on de-identification and you would also get the checklist regarding de-identification with that and any references. You would even be able to extend your search by clicking on those references and get right to those. Right now, it is just a Word document. My view is it could be so much more useful if it were a live document.

And then even better yet, if we could solicit some ways in which this was used, how it was helpful, what is missing, what additional questions are coming up that we have not addressed. It really could be a live resource, which the discussions that we have now in population and in data use about communities of practice fits right in with that if there is a discussion group around de-identification and questions that are coming up there. That is all grand and probably beyond our current capability.

But in the short run, it is once it launched a public document and anybody could take this document and do what we envision. That would be fine. I think our interest is in getting broad dissemination and raising all boats for those that are not really skilled at data use and at least asking the questions about whether they are being effective stewards.

DR. MAYS: Let’s see what suggestions that we can get. Not even assume yet — again, we are going to stay in brainstorming level. Not even assuming yet that they cannot be done, but instead let’s take and get you ideas and then see the extent to which the department can do some of them. Or we just talk to HDC and others. They might be able to do something different.

MS. KLOSS: We can do a lot of links.

DR. MAYS: Let’s get the ideas out. Josh.

DR. ROSENTHAL: A couple of different things depending upon what you do. One, you can literally take any public document — I do not know how this applies to public document. Usually you can take and like as a private individual there are all sorts of people that take stuff and turn it into Kindle books, which are linked. It is usually about $200 or $300 to do that. You hit one of the people. I have five people that do it. I have done it myself. You say here is a 60-page book. Turn it into a Kindle doc. You put the thing up there. Usually, they want to make sure you have rights of authorship or something like that. Once you have that and then you put the thing on Kindle as an Amazon shop, you can put that on there as open source versions, which are not Amazon specific. They are called Moby docs or Moby books. Once you turn it into an e-book, which cost like literally $200 or $300 to have a shop do it for you, you can put it in all these distribution networks. That is number one.

Number two, if you break it into web pages and hyperlink, which is literally like ten minutes of work. You literally cut and paste and hyperlink the think depending upon how you want to structure it.

DR. MAYS: Would you hyperlink then in the way that she talked about, which is each piece.

DR. ROSENTHAL: Yes, you can set it up however you want to. You can say I am going to have each page — there is an e-book format, which has ecosystems and distribution and people that read that. There is an HTML website format. There is also depending upon how crazy you want to get in terms of feedback. You could do things like slide chair or something like that. You could convert to Power Point, which is long, but admittedly these people do it. If you do that sort of stuff or something snap guide if you could distill it is where you have a really interesting group of people doing things like that.

Once you have that posted and that is the anchor of your ecosystem, then you can solicit feedback outside of it. Then you can put something up on CORA or whatever alternative you want to use, and here is this thing, this Moby book, this URL here. We are looking for feedback. Post the thing wherever you want to.

DR. MAYS: One of the things I know that the surveys do is that before they let you get it is that they get you to commit to saying here is how I plan to use it. And then at the end of it, it will come up and ask if you will take a survey and usually it is very short. It is about what is easy, did the pages move fast. They do all that kind of stuff. That is something else.

DR. ROSENTHAL: Or you could put a gate on it and actually just build a list.

DR. MAYS: I am concerned because I do not know that the community wants the federal government — that is the only thing I am concerned about is that then that may turn some people off. You have to give your information.

DR. ROSENTHAL: Typically, you would say like an opt in where it says I want to be contacted for feedback or something like that. If you are actually using this — I am a user and I get these Power Points. I have five things that bother me about it, but I am not the kind of person who types on it or whatever. If you want to ask me my opinion, I would be more than happy to share it.

DR. MAYS: I would agree with that.

MR. CROWLEY: Great work pulling all this together. I know it is always a question of what is the rub and get it out. Try to distill everything down to one page. Just the key items, what it is about. You can have all the other items to follow that, but it needs to be something where someone can just look at. I know what this is about. I know it is meaningful to me and that is what it is. You may even consider putting a two-minute video with it.

MS. KLOSS: That is not a bad idea. I think it would lend itself frankly to an infographic because — we lost that page, but you will see that there is a diagram that guides the whole thing.

MR. CROWLEY: Do a blog post by someone — or someone who has credibility and gravitas. Have them do a blog post for you why it is important. The secretary would put it on her blog. The Health IT buzz blog and it gets a lot of traffic, but just a blog posted. Once you get that initial blog post out —

MS. KLOSS: Can anybody use that?

MR. CROWLEY: I get updates from it.

MS. KLOSS: — ONC —

MR. CROWLEY: Look and see. Just say what are the lists of all the HHS blogs. Which ones have the most readership and target some that have relevant facts for the community? Once you have that blog post, it has a link to the article, then you can start posting it everywhere. Have everybody in the taskforce put a shared on their linked pages. Have everybody tweet it. If for the different conferences that are going on around that have anything to do with data stewardship, get in that feed. Put on the conference, follow that, put it in the conference pack. Put it out. Are you doing stuff with data? You need to know about data stewardship.

PARTICIPANT: We need to have a tweet-a-thon.

MR. CROWLEY: I am not sure what that is. But what you could consider doing is once you have a really strong tweet about this, do a promotion. Identify who are the top influencers around data or data stewardship, data management, public health data, and do a Twitter campaign. They are not very expensive, but you can make it so that this stewardship tweet shows up prominently within their activity feed. Those would be a few things.

DR. MAYS: Lily, you know how to do that.

MS. BRADLEY: Which last part are we talking about?

MR. CROWLEY: There is a promoted tweet. If you need help doing that, I have a graduate student who does that for us. She optimizes who are all the influencers in this space. You can actually pay a little more to get into their tweet streams.

The second part was just the conferences, which the audience you are trying to reach participates at, get it into the conference, those conference tags during the conference. Datapalooza would be one.

DR. MAYS: The NCHS conference. NCHS has their conference and then you could also ask HDC to do it and then ask the HITT group that is in APHA to do it.

MR. CROWLEY: You do not even really have to ask them to do it. If you just have a Twitter account, you just use their hashtag during their conference and it shows up. Technically, you are supposed to be at the conference.

DR. MAYS: Any other suggestions? Is that helpful to you?

MR. CROWLEY: Do you have a PR agency? If there is budget for PR, leverage it.

DR. MAYS: That is what I am saying. I think what we need to do is come up with the ideas. And then once we come up with the ideas, I think Debbie will then talk to the department. Personally, what I think is it would be great to see if you launch this what the response is so that you can figure out for the other products of the committee what to use. For the first time to do it to see the kind of professional space that the committee is in, the update for that I think would be useful. Very honestly, I do not think anything in here — I do not think we have talked about anymore than maybe a thousand dollars.

MR. CROWLEY: If you are up for putting money behind it, even a thousand dollars, Twitter is great. For sure, explore that. But there is a basic checklist. Just Google it. What should I do for social media campaign? There are Google ad words. There is Twitter. There are five or six different —

PARTICIPANT: Is there a checklist like that?

MR. CROWLEY: Yes, there is. Just Google. I can send you one.

PARTICIPANT: A checklist for social media.

MR. CROWLEY: Basic campaigning. You are talking about a thousand bucks. My strong suspension is you are going to get a much better response off of a LinkedIn account, but what do I know. I would actually do all of them.

PARTICIPANT: The committee does not have a LinkedIn.

MR. CROWLEY: If you are willing to pay a little bit of money, all sorts of options are open. And if you are willing to do an executive summary and have a narration where you as a human being don’t just say what is in the thing, but say this is why we produced it. This is what is meaningful and here is the thesis. Then you open up all sorts of options. All I would say is think it through a little bit. Be really systematic. There are basic checklists around it. I want to opt to one channel or one medium. I would AB test some stuff. I would literally throw out five things.

DR. MAYS: I agree. I think then you can see in this space. This may be the experiment for the committee.

MS. JACKSON: We have been leaning towards something like this for a couple of years. We are just not quite knowing how to get our hands around it. For someone like Chris Boone to be on board and indicate he hadn’t heard much of the committee, things like that where people realize this is one of the hardest working committee that people do not know about. We want to change that in 2015 with these mechanisms, with these avenues, with a great product that we have that you do not want to just leave on the shelf. A lot of us are in offices. We do not even have shelves anymore. That is old school. What we are looking at now is how to really get it out. We will look at different mechanisms and then go from there.

DR. MAYS: You are welcome to bring it back for a discussion.

DR. ROSENTHAL: You will actually get very easy simple metrics. If you are doing Google ad words, you get to see where the stuff comes in from. If you are doing LinkedIn, you get to say who responds to this. Who reads this? Is it policy people? Is it government people? Is it market people? What is their education level? What is their income? What is their intended use?

DR. MAYS: You really do want the metrics. I agree because that is going to tell you who is doing it.

MR. CROWLEY: Think about being a little provocative with some of the marketing. You do not want to get sued for your data release or you do not want to end up on the New York Times for being a data dummy. Get your own hashtag. Don’t be a data dummy or something.

MS. BRADLEY: He has a lot of followers.

DR. MAYS: Let’s kind of take this and ask that at the next meeting in February that you all come back with a plan between you and Debbie and Jim figure it out.

MS. KLOSS: I hope we will have launched by then

DR. MAYS: — you did use. Whether this that you think you can and if there is part of this that you want more ideas about. We move nimbly so if you have some questions, we send it out in the email. You get responses from whomever and we just let you move on.

Walter, in terms of the work group, anything in particular that you see as an interdigitation that you want us to be thinking about?

DR. SUAREZ: Sure. Our subcommittee of course focuses a lot more on the HIPAA standards. That line of work seems to have less of a connection to what I think are more critical connections to the work that the data access and use committee or work group does. Here are four areas. It is all about standards. Standards for the format and content of the data basis, the terminologies that are used and the vocabulary and terms and those kind of elements, the metadata and this last one, the API. I think that is really new from the JSON report to the ten-year plan to everything that everybody is talking about. The API is the way to go when it comes to interactivity and interoperability with data. And ultimately the ability to create API interfaces that are able to be accessed on those databases will improve I guess the accessibility and the usability of the data itself.

DR. MAYS: What would you like us to do? I only got three of the things. If you send it to me and send it to Lily then I will make sure. What is it you want from us?

DR. SUAREZ: I am more like what we can help you with. I think those are areas that need to be explored and worked on when it comes to accessibility and availability of databases and federal databases.

DR. MAYS: That is great.

DR. SUAREZ: The other element that I was thinking about was — there are two major topics that we are going to be looking into next year. One of them is health care transparency, the whole concept of transparency. The extent of how much we as a national committee get into it, we are to talk about. But it is a topic that we wanted to bring up and have a discussion. I think there would be some connections with respect to transparency and access to data through the health initiative.

DR. MAYS: I agree. That was one of the ones that when that came up I said I thought the access and use. I do not think that people know about the ACA review piece. Can you just kind of briefly let them know what that is.

DR. SUAREZ: The Affordable Care Act had a provision for the secretary to establish a review committee. The review committee’s function is to evaluate the degree to which the current standards used in HIPAA transactions in these areas are still and continue to be responsive to the business needs of the industry and effective to achieve ultimately the purpose for which the standards have been created or whether we need to move to newer versions and additional changes in the standards or newer standards.

The secretary designated the National Committee on Vital and Health Statistics to be the review committee. The Standard Subcommittee within the National Committee is going to operationalize the function of the review committee and it is going to begin to implement the series of work items including no less frequently than every other year, but probably will look more like every year hearing on all the inviting the industry to talk about how well the standards are serving the industry and the industry at large and health care as a whole. That is the new function that the National Committee is going to have is a very specific task.

The good news is that the National Committee had been doing this for some years already and this is a very complementary, somewhat of an expansion, but it is very complementary to what we have been doing.

Again, it focuses very much on the standards side. But as we discussed with the National Committee earlier, yesterday, and this morning was there is also the question about the evaluation, not just of the standards for the Affordable Care Act, but actually the evaluation of how the Affordable Care Act is going. There is some opportunity to consider that as another topic for the National Committee. In February actually, we are going to hear from or that is the plan now to hear from the Department of Health and Human Services their entire strategy to evaluate the implementation of the Affordable Care Act, the health reform, and all the components. There is some agency-wide strategy to monitor and evaluate that. They are going to be presenting that to us. We are going to see if the National Committee would be in a position to work on that and provide feedback to the secretary.

DR. MAYS: In the February meeting, you may want to pay attention to whether you want to come in earlier to actually see that. If you look at the website, you will see when that particular thing will take place. If that is a value to the work that we are doing here, which you can see that that we are being asked to do that. You may want to pay attention to when that is and to be at the meeting. That is where this health transparency piece is coming from. I just wanted to make sure you understood where it comes from because otherwise you would not have heard that as an agenda item.

MR. CROWLEY: Transparency is going to be one of the major shifts in all of health care for the next five to ten years.

DR. MAYS: We just got that as our agenda. That is what I am saying. For those of you who have an interest, you may want to be here to actually hear HHS come in and discuss that. Thank you very much.

Do you want to talk about populations? We kind of see where some of it, but I want to give you an opportunity.

DR. COHEN: Sure. The work we are doing on community data engagement is really right in the wheelhouse of what is going on here in our conversation. We are going to be holding hearings, roundtable workshop on community data literacy. We think about that in a bidirectional sense. It is understanding from a community perspective how communities need to receive information that they can really use better to do their work, but it also is understanding what we need to tell the federal government data providers about how they need to disseminate and repackage information that is more meaningful to communities. I think the focus of some of the dissemination assessment tool and some of the activities that we are doing here really will help us in the Population Health Subcommittee hone in on the issues particularly with respect to the data, the federal agencies, and what they are doing and what they are not doing with respect to providing meaningful data for communities to use.

Certainly, I invite everybody to be engaged in the community aspect too, but the work here directly is related to the federal data provider piece.

Susan, do you want to add anything? You all know Susan. She has been a key person in helping us organize the roundtables and really come up with a solid set of objectives. I do not know whether it is posted on our website, but the set of slides that I presented today gives a really good summary of the roundtable for the folks who were not there. It was really very robust discussion.

DR. MAYS: If not, they will be there.

The other thing that we try and do near the end is allow you to make these announcements about things that you have going and things you think people should be attending and just in general raising our IQ on data access and use and technology issues. I knew the datapalooza because I saw the request for the abstract went out. I wanted to make sure, but we did that early. Are there other things that people want to make announcements about?

Agenda Item: Next Steps, Work Group Member Announcements

MR. CROWLEY: I do not have an announcement, but I have a request. One of the things we are working on right now is a public health information technology maturity index, looking at the different structures and processes that underlie public health in terms of the information systems that are used. One of the things that is coming up with that is Adelphi with some of the — officers and CIOs, the different public health departments and systems. If you have any recommendations of good people who could provide feedback on what is that optimal in a perfect world if we could get there, public health information technology infrastructure, I would be interested in getting that feedback.

DR. SUAREZ: I can quickly give you a few. We have a national organization called JPHIT, the Joint Public Health Informatics Taskforce, which includes pretty much the leadership across public health on health information technology and information infrastructure. Through there, you will be able to access a lot of the leaders that are moving forward sort of the health IT infrastructure agenda for public health. JPHIT, Joint Public Health Informatics Taskforce. I believe we have a website. You can just Google Joint Public Health Informatics Taskforce or JPHIT. You will get to it.

DR. MAYS: Is there a way to send an email to the group or something?

DR. SUAREZ: Absolutely.

DR. MAYS: Then the other person would be who is not here is Bill Stead who does informatics and may have some suggestions as well.

DR. SUAREZ: Another important point I think — thank you for bringing it up. The National Committee and I do not know if you are familiar with it, but in June of this year a few months ago we sent a letter to the secretary out of a hearing that we had last November a year ago on public health information infrastructure with very specific recommendations about advancing and repositioning or advancing the health information infrastructure and public health. There is a series of recommendations. It is on our NCVHS website as well.

DR. ROSENTHAL: Health datapalooza. I am organizing the data track. If any of you know anyone who is doing it.

DR. MAYS: What are the deadlines?

DR. ROSENTHAL: I actually have no idea what the deadlines are. I am the worst person to ask that. Whenever they feel like it. But seriously, the data track is one of the toughest to staff. If you are doing anything reasonable, have any students that are doing anything reasonable, know anyone who is doing anything reasonable, please have them submit an extract.

DR. KAUSHAL: For the timing as soon as possible. We can always find a place to put it in. It is more about quality than quantity.

MS. BRADLEY: I just wanted to make sure that you got to see this Google drive. You guys have seen it. You guys know how to use it.

PARTICIPANT: Do you want to send me the invite again?

MS. BRADLEY: I will send it out again. And then you do have to — did you ever see this as well? There was a response from the secretary to the letter that we sent. There was a working group that had sent a letter to the Full Committee and the Full Committee sent it out to the secretary.

DR. MAYS: Is this the one that Justine sent?

MS. BRADLEY: Yes and then there was a response from the secretary.

DR. MAYS: Okay.

MS. BRADLEY: I suppose I should send that out as well.

DR. MAYS: Send it to us and then we can look at it. Was it a good response?

MS. BRADLEY: Yes, it was positive.

DR. MAYS: I think we will look at this and then see if there is anything. Do we respond back to the secretary when we get responses? Thank you or thank you for the response.

Seeing nothing else, can I get a motion for adjournment? We have a second. Thank you very much everyone. I think we have ourselves a great start on accomplishing something that I think can make a difference. Thank you everybody.

(Whereupon, at 4:45 p.m., the meeting adjourned.)