[This Transcript is Unedited]

Department of Health and Human Services

National Committee on Vital Health and Statistics (NCVHS)

Working Group on Data Access and Use

May 28, 2015

Hubert Humphrey Building
200 Independence Ave., SW
Washington, D.C. 20024

TABLE OF CONTENTS

P R O C E E D I N G S (1:15 p.m.)

Agenda Item: Call to Order, Welcome, Review Agenda

DR. MAYS: I am going to call the meeting to order. Thank you, everybody. We are now moved from the NCVHS full committee to start our afternoon with the workgroup meeting. I would like to welcome everybody. We are going to go around the table, have people introduce themselves, say if there is any conflict. Then we will go to the phones and have people on the phones let us know who is on. Let’s start with you.

DR. FRANCIS: I am Leslie Francis from the University of Utah. I don’t have any conflicts, and I am a member of the working group.

MS. BURKE-BEBEE: I am Suzie Burke-Bebee, and I am with ASPE.

DR. MAYS: Vickie Mays. I am a member of the full committee. I am the chair of the working group. I have no conflicts.

MR. DAVIS: This is Damon Davis, Department of Health and Human Services, no conflicts.

MR. STEAD: Bill Stead, member of the full committee and co-chair of population health, no conflicts.

DR. COHEN: Bruce Cohen, member of the full committee, co-chair population health, member of data workgroup, no conflicts.

MR. CROWLEY: Kenyon Crowley, University of Maryland, member of the working group, no conflicts.

DR. KAUSHAL: Mo Kaushal, member of the working group, no conflicts.

DR. MAYS: On the telephone is Paul?

DR. TANG: Yes, Paul Tang, Palo Alto Medical Foundation, member of the working group, no conflict.

DR. MAYS: Anyone else on the phone? Okay, Lily, Google Hangout?

MS. BRADLEY: Lily Bradley, I am part of the committee, no conflict.

DR. VAUGHAN: Leah Vaughan, member of the working group, no conflicts.

DR. MAYS: Great. In the room.

MS. JONES: Catherine Jones, also known as Vanna White, NCHS.

MS. HINES: Rebecca Hines, NCHS.

MS. JACKSON: Thank you, Vanna. Debbie Jackson, National Center of Health Statistics, CDC.

DR. MAYS: One of our members isn’t here for a very good reason, so we want to say congratulations to Josh. Josh just had a baby. Best wishes to Josh. Josh, if you are trying to dial in, don’t worry. We are good.

Then we also don’t have him here, but he probably will come in a little bit later. We will wait. We will congratulate Walter when he comes back in, so that we can do it when he is actually in the room.

If there is anything, Paul and Lily, as we go through, if you don’t have, just remind us, and I will make sure that we send it to you. I think you have the couple of things that you need to have for the meeting today, which is the tool kit, slides, the agenda and the pop letter. If you don’t have that, just email and let Marietta know, and she will make sure that you get it.

What I would like to do is start by making an adjustment to the agenda because we have a couple of people that need to kind of rearrange schedules. Damon needs to get out. I also know that Bill needs to get out.

What we are going to do is to actually start with Damon. Then we will move directly to Bill, so that Damon can hear Bill’s presentation. Then we will go back to the agenda as it stands. Is that okay with everybody? Good? Okay.

Damon, if you could start by talking about the updates from your office and any kind of feedback of any of the other things you presented in terms of things we have commented on, that would be great.

Agenda Item: Updates and Follow Up Status from the Office of the Chief Technology Officer/HHS IDEA Lab

MR. DAVIS: This is Damon Davis, HHS IDEA Lab. I am going to start with a bit of a technical update in terms of HealthData.gov. I just wanted to let the group know that we recently released a contract for the revitalization of HealthData.gov. The organization that is going to be supporting the website now is called NuCivic. They are operating off of a DKAN. It is Drupal and CKAN combined together.

The phase we are in right now is basically what I would loosely call a beta launch. We are trying to basically get the catalog converted over to the new platform in time for HealthData.gov to be at least presentable for the Health Datapalooza that is coming up starting on Sunday, May 31st.

Some things that you will see with regard to HealthData.gov’s revitalization. There will obviously be a new skin on it to modernize it, make a cleaner presentation of the datasets and the data assets that we have there. We are also going to build in some data visualization tools. We are going to build in some dashboards for the data’s utilization, so that you can have a clearer understanding of what some of the, for lack of better terms, hot data sets are, the ones that are most interesting to a variety of people, as well as a little bit of the ability to reference other datasets that people are interested in.

I think one of the things, Lily, that you named out of this work group a long time ago, was the idea that you want to sort of either upscale or help people to understand what others have used, the Amazon approach to datasets. Other people who have viewed this dataset have also viewed these.

We are taking cues from a PowerPoint that Lily put together with this workgroup a long time ago, Data.gov, influencers. Also folks from across the data use population, anybody who is coming to help Data.gov. Because this is something that is still under development, there is still a lot of opportunity to take some cues from those in the user community as to what it is that they want, need and desire.

It is an exciting time. You are going to see a very basic form of HealthData.gov, and it is going to be built up over time. I think that is a solid approach to sort of doing a more agile development life cycle that will allow us to focus on one thing at a time, make it great and then move to the next thing. That is HealthData.gov for us.

As I already mentioned, the Health Datapalooza is coming up. It is going to be beginning on Sunday, May 31st, with a variety of activities. Some of them are international. You may know that the United States Department of Health and Human Services has a memorandum of understanding with the UK NHS England. A part of that is sharing data, looking for means through which we can take lessons learned, take best practices and actually match up our data against each other or what have you, and try to find new insights into things like obesity and other comorbidities.

There is going to be a challenge that is going to be announced at the Health Datapalooza. We are going to be doing a lot of work to sort of encourage and incentivize the developer community to find our datasets and utilize them more fully. Specifically, one example of what is going to transpire at the Health Datapalooza.

Walter and I are going to be teaming up to co-moderate what we call the DataLab. Originally, I wanted to ask Vickie to do it as the chair of this working group. Unfortunately, she is not able to participate, so Walter has graciously agreed to step in, in her place.

The DataLab is basically something that you guys have heard about before. It is the opportunity for the department to tally its data resources, but also for the innovation community to step forward and say, yes, we have used that dataset. Here is what we used it for. It is really a means through which we can show a more longitudinal look at the data. Here is the reason it was collected and curated by the department. Now that we have made it open data or publicly available, here is the alternative use that someone on the outside has found for it. It really is kind of a fun session there. That is one of many things that is going to transpire.

You will also hear some really cool and very interesting announcements out of the Centers for Medicare and Medicaid Services. This secretary, the new secretary, Secretary Burwell, this will be her first Datapalooza. She is going to give a keynote on the third day, which I believe is Wednesday, the 3rd, if I am not mistaken, or Tuesday. She is going to give a keynote on the final day.

Just to signal a couple of things that you will hear from her, what she is going to try to convey really is the idea that this is an unparalleled time in health care. You have got the convergence of many different things happening at the same time. We have got our high-tech and open data and all kinds of efforts to do really innovative and creative things in health care that are going to advance health care in this nation.

Her speech is going to be very inspirational and help the folks in the audience to really understand that this is an opportunity for you to really dig in and show what it is that your creativity can bring to innovation in health care in order to maximize what it is that we have the opportunity to do in terms of the convergence of all of those things that I talked about, inclusive of the ACA, I should say.

I am trying to think if there is anything else out of the health Datapalooza. While I am thinking, I guess I will just take any questions that somebody might have.

DR. MAYS: Let me introduce because they want to ask. So let me just get introductions for the people who came in late. Then they can do the no-conflicts, and then they can participate.

DR. FULCHER: Chris Fulcher, University of Missouri, no conflicts.

DR. SUAREZ: This is Walter Suarez. I am the chair of the National Committee and member of the Data Access and Use Working Group. I don’t have any conflicts.

DR. MAYS: We wanted, on behalf of the Work Group, to acknowledge our new chair and to congratulate him, and to say we are very happy and looking forward as a work group of working with you. Congratulations.

DR. SUAREZ: Thank you very much. I am very much looking forward to our continued work through the work group with HHS and helping advance access and use of data.

DR. MAYS: Okay, let’s start with Leslie.

DR. FRANCIS: Very simple logistical question. Will you be able to listen to Datapalooza online in real-time?

MR. DAVIS: I don’t know that. As someone who has been there for almost every one, I don’t recall if there are elements of the show that you can listen to. I feel like there may be some portions that are livestreamed. My recollection may be for those who are in HHS. We sometimes will livestream pieces of it from the TVs here. If you would like me to find out, I am happy to do so.

DR. FRANCIS: That would be great, or even if it is going to be available letter.

MR. DAVIS: So many of the slides from the various breakout sessions are all available online after. Whether the recorded session itself, at least the main plenary sessions are available. I think many of them are. They are broken out into different pieces, so you won’t get the whole show in one shot. But you can certainly sort of find and parse through the elements that you are most interested in.

Questions about the Palooza or HealthData.gov? I would love to underscore that this is a beta, very early version of HealthData.gov. Therefore, I would love to have this workgroup’s input as to things that you would be interested in seeing on a data cataloging platform in a new version.

I would highly encourage you next week to poke around, see what kinds of things are of interest, see what kinds of things you feel like you should be seeing and you are not. Also, just recognize that this is the early version, and we are going to build it up as we go. It is a great opportunity for you to provide input. I would love to task the members of the workgroup with kicking around in there and seeing what kind of things you find and what things you think you should find that you are not.

DR. MAYS: I was going to say maybe what we could do is, in the context of some of the work that we are doing around the guidance, to actually look at it and see when we have a discussion about the guidance on access and use, to look at what you have and also then to make direct recommendations to you.

MS. BRADLEY: I had a question. There was the data entrepreneur that came in and built something. Were you able to use any of that? It was a sort of demand-driven, like figuring out what people wanted.

MR. DAVIS: The gentleman’s name is David Pontius(phonetic). He is our entrepreneur in residence. He is building a demand-driven, open data platform or interface.

He has done most of it in GitHub right now, as he has been trying to sort of figure out how to make that work. I am glad you brought that up. It is probably fair to give an update there, as well.

I think the question you were trying to — go ahead.

MS. BRADLEY: — intersection with the update to the website.

MR. DAVIS: That is exactly what I was going to get to. They have been separate and individual activities until now. Just as a reminder, he is trying to figure out how it is that we can have this demand-driven approach to data, where the public is able to ask the Department for specific datasets in specific forms.

Then there is a sort of pathway to have the responsible party for that dataset be contributory towards releasing that data in the form or the closest form that can be made available. What Lily is asking is there a way to link up those two? Is HealthData.gov and demand-driven open data going to be one in the same or part and parcel. That is what we are trying to get to.

David is very interested in this early stage development of the platform. We have not gotten to a point where we can engage him in making the process of demand-driven open data be part of HealthData.gov, only because it is so early in HealthData.gov’s relaunch. That is certainly one of the things that I want to fold in very quickly here because his entrepreneur and resident’s time is going to end soon. We want to build this in as something that is going to have longevity well beyond his tenure here with us.

He is doing a couple of things. He is interviewing people, PMFs and detailees and others, to try to find folks who are potentially capable of helping to take this thing over when he does leave. We also do want to try to make it an integral part of HealthData.gov, so that there will be a backbone to it, as well as the sort of process and set of responsibilities for those of us in the Department.

MS. BRADLEY: Do we know who is going to fill in as technology officer?

MR. DAVIS: We do. I can’t say. It is coming, so I am glad you brought that up, as well. That is another thing that you are going to hear about very soon is that there is going to be a new CTO announced here at the department.

DR. MAYS: Before Datapalooza or after?

MR. DAVIS: We will see what happens. I am sure it is going to come out before Palooza. It is just not official yet.

DR. SUAREZ: I was wondering if, with all the efforts that are going on and sort of starting this new phase, is there a formalized evaluation process, sort of with metrics? Has that been established? Is there any way that actually might be one of the things that we can contribute to, as well, is how to evaluate all of these incredible amount of activities and efforts that are going on.

Part of the concept behind the entire Datapalooza, of course, is this disruptive nature of the entire entrepreneurial process of going in multiple directions from multiple perspectives. I am not trying to force a rigorous methodological evaluation process. Has there been any process identified to evaluate? Four years from now, two years from now, say, well, this is what we have achieved.

MR. DAVIS: I think I am hearing a couple of different things in your question, one of which is sort of an evaluation of the success of the relaunch of the platform and whether it is, in fact, effective or impactful. The other that I think I am hearing is more along the lines of the advancement of the health data initiative writ large.

One of the things that I have heard out of this workgroup is the idea that obviously the datasets need to be more liquid, more machine readable by default, those kinds of things. I think the other thing I am hearing is an evaluation of whether we make this conversation on HealthData.gov, yet we still a number of datasets that remain in PDF form a year, two years later.

On the first part, the platform itself, we have attempted to take a bit of a data-driven approach to how it is that we are revitalizing the platform in that we have taken recommendations from the users of the platform in the past. We have also taken things like the Google Analytics and some of the input from specific users to try to inform the approach of the platform. We don’t necessarily have the evaluation on the backend that is going to say we have delivered on all of those issues.

On the second part, I think we can build in the piece where we try to evaluate a specific operating division on its machine readability of its datasets. If you go into this new phase, and you had 52 PDFs and whatever number of Excel spreadsheets and APIs, as you exit this phase and go into the next, are we starting to see this conversion of those datasets into alternative forms.

I think that would be a really good thing for this workgroup to be a part of, that overarching conversation that basically says, are you all making advancements on the data initiative, not just on the modernization of the platform that supports the catalog. I think that could be a really good thing to have the Work Group work on. I know that you have talked before about doing that evaluation, giving report cards or something incentivizing a stick or a carrot that would be beneficial.

DR. SUAREZ: At some point, we talk about providing feedback. In my mind, that was a different evaluation. That was an evaluation of certain parameters of the specific dataset saying, sort of giving a report. How useful, the utility, the usability, the usefulness, all those factors of creating a sort of report card that then can be reported back and the dataset can be adjusted or improved.

That is one, but it is really more the larger picture of evaluation. Are we really making headway or progress. With your okay, I think Vickie, with your okay, I think that would be a really great part of the kind of work that we can do with the Data Access and Use Work Group.

DR. MAYS: I think that is one of the things, like when we were looking at are some of the things that would be helpful to you. We have already kind of gotten to that. That can be one we can put on the agenda for ourselves. Great.

DR. FULCHER: If it would help Damon in terms of we could have our team look at the website from different perspectives. From our Center, we have got the data folks, the GIS folks, who would use it in one way. Then we have, through our non-profit, the way they use it as a standard end user, looking at usability or the API or the data download. If we had some buckets that you want us to focus on – because I know your team is probably focused in a way around usability or taxonomy that you really want feedback on.

What I ask is that if we could have basically five main areas that we could really provide a response on. There is nothing like a timeline. Usually we have a meeting and say, yes, can you look at the site. It would be good if there was an expectation of when you would like feedback. Then we could do that with you.

MR. DAVIS: I love that idea. I think that would be really helpful. I don’t have a timeline necessarily, but I could certainly provide you about five things that would be great for you to review.

I need to get through the Palooza, through the data catalog conversion and some other sort of low-level things that are going to add functionality before I get to the point where I am really ready for the robust set of usability testing. Even in the version that you are going to see in the next couple of days, that is not even the version that you are going to see a couple of days later. There will be continually evolving functionality on the platform that is going to be really beneficial for us to all examine at the appropriate time.

DR. FULCHER: We will start hitting it at Datapalooza, and we will wait until that time, as well. The one area around Google Analytics, we have been using that for many years and looking at it. The funders want to know how many hits you had on the website, et cetera. We have really gone beyond that to much more user-centric metrics. It has really helped us in our evaluation and monitoring.

We are working with KISSmetrics. That really helps us understand how people are navigating through a website and what they value the most in terms of data and data download. We could provide that as part of some feedback or what we have experienced as well. Do you have that flexibility to do that, or is it just Google Analytics? Are you open to other options?

MR. DAVIS: No, I think we would be open to other options certainly. We want to make sure that we are providing the best user experience possible, improved sort and search, some of the tools that are going to allow for you to interact with the platform in a way that is going to help you have a data-driven approach to finding data. The taxonomies, even the key words that we are using, there is a lot of stuff that is going to have to be done.

Some of these things, we will be fixing the plane while it is in flight. The department is working on its own set of taxonomies, for example. There are just so many different pieces that would need to come together to have this be the absolute best thing possible. I think we are just going to continue to work on it and hack away on it. We are obviously interested in as much input as we can get.

DR. FULCHER: When our team looks at search and across all of these different categories, we all end up running out of the room because it gets so complicated. Maybe we will just have our team talking with your team sometime. That would be great.

DR. MAYS: David, what would help us is, a little bit later in the agenda, what we are actually going to deal with is short-term and long-term things that you might be interested in. Can you think of things that the workgroup might be useful to you in accomplishing, and kind of give us a sense of what the scope of that work would be, and whether it is something in which it is just advisory and we could have a discussion, a phone call and it is done. Or things that may require a little bit more work that we need to kind of plan out.

MR. DAVIS: I don’t know if I am able to answer that right now. I would love to take some time to think on that. I apologize. I feel like you have asked me that before, and I also feel like I run from thing to thing to thing to thing. I never have a chance to sit down and try to take advantage of the offer that you have put on the table in terms of supporting the site and the health data initiative.

Perhaps what I can make sure to do after the Palooza is sit down and actually have a conversation with you or some others, if you want, and try to hash out what some of those opportunities really are. If I don’t dedicate time to do it, it is not going to happen. I think we should sit and strategize a little bit as to what the real opportunities are.

DR. MAYS: What we might do is pull Jim into the conversation, so that we can get the whole of HHS. Figure out what is in the IDEA lab, what are other things in HHS. I am sure then we have Walter in the conversation. Then to see how these things either come back in the workgroup because there are a few things, I think, Walter, that may be again kind of for the committee as a whole in terms of what we are hearing. Why don’t we do it that way? We will follow up like that.

MR. CROWLEY: It sounds great, the direction that the website is taking. I look forward to seeing these sort of new tools integrated. Just a couple of quick thoughts. I think for a formal evaluation, longer term makes sense. There might also be some things we could do lightweight along the way in terms of asking a certain percentage of the people who come to the website.

Did you find what you need? If not, what were you looking for? How would you define yourself? Are you an entrepreneur? Are you a researcher? Are you part of the general public? Just sort of small engagement with collecting some data from that. You can many times learn some very insightful things. That might be an opportunity.

MR. DAVIS: The four C’s survey kind of thing.

MR. CROWLEY: Something like that. Do it in a way that is a little more engaging. A lot of people see that as just the four C survey. Flips it off and says, right. If you say, hey, help improve the website, or did you find what you need because a lot of people want to get engaged. That way, if they feel they are contributing to the community, they are more sort of apt to participate.

It sounds like, too, you are using some natural processes in which you are deploying some things and testing things as you go. If there are some questions that come up around the specific format or the structure of certain things, it may even be worse deploying both. A certain set of web users gets one format. Another set gets another format.

You can actually track what that led to in terms of utilization of that tool or sort of engagement with the site. In one structure, they went away from the site after 20 seconds. If not, it took them down a path and found certain things. Those are my only comments.

MR. DAVIS: I can check with the contractor to see if that is something that they are able to accomplish.

DR. VAUGHAN: Have you at any point in the process, actually focus grouped the website? It would be interesting to do that from the point of usability, user experience. Growing that out, that could be something more straightforward as students.

Another group of large public health users would be coming up at the NCHS data users conference, which is always a very vibrant community. It is coming up, I guess, in a couple of months here in town.

MR. DAVIS: That sounds like a great idea. I was just on the phone with the new SEPIC team right before coming in here. They were very interested in having those kinds of inputs from some user groups. Outside of yourselves, that would be really good to take advantage of some of those, too. I think that is a great idea. I would love to have your guidance as to how to set one up and collect the input, and sort of parse it into actionable things.

DR. VAUGHN: I would be delighted to. I have done a bunch of them around women’s health issues. I would be happy to.

DR. MAYS: Just so you know, it is here in DC, and the data is August, I think 24th, 25th. What you need to do because they are closing very quickly for space and stuff like that, is you really should contact Jennifer Madans and ask about setting it up.

MR. DAVIS: Who can I follow up with to do?

DR. MAYS: Just e-mail me. If it is on behalf of the workgroup, then Walter and I can actually talk about it because that is what we are going to do is to go back to them to say what it is that we are going to do for the meeting. If you want us, we can then attempt to negotiate that.

MR. DAVIS: That would be cool. Yes, that sounds really cool. Anything else? All right. Well, that is it for me for right now.

DR. STEAD: Thank you. Thank you for jiggling things so we could fit together because I think this conversation and the things you just asked Damon, I hope will prove to be possible points of convergence. What I want to do this afternoon is just briefly introduce the NCVHS framework project because we need the workgroup’s input about how you see it fitting with data access and use to help us prioritize this work.

The premise motivating the framework project is that the benefits of the explosion and liberation of health data can only be realized if the systems that we have for making sense of the data keep pace with the burgeoning volume and complexity. Otherwise, we run the risk of being unable to efficiently analyze the resources and to even compare them with one another just because of the sheer quantity and variety.

The framework project seeks to address this risk by facilitating dataset classification use and analysis. We have been drafting two knowledge resources, a data structure and a methods taxonomy to offer a systematic approach to thinking, talking and acting with respect to data and as a possible source of metadata to tag datasets to support reuse.

The target audience for these knowledge resources includes statistical and analytic experts, researchers, data suppliers, intermediaries and application or system developers. These experts, and the tools they develop, would then use the classification resources to help others work with data.

That is the brief two-minute sound byte on what the project is. What we would propose doing, if the workgroup was comfortable with that, would be to distribute a short white paper that I think is now about 25 pages to let people read that in advance. Then Vickie said that we could coordinate a conference call where we would ask for two types of advice.

The first would be from your perch, what is the value proposition. What are entrepreneurial ideas for how to exploit the data structure and methods taxonomy to develop projects or anything that you could see could accelerate or change the game in what you are already trying to do. Would these kind of resources be helpful?

The second ask would be any thoughts you have about how to go about elaborating and curating these kind of resources. The National Committee can focus on the need for something. It can try to describe what the something might look like.

The National Committee is not actually resourced or structured to really create such a thing. If there is a real value proposition from your person, the next question is what is the scalable approach to elaborate in curating this kind of resource going forward, so it would be available for us to collectively use. That is just a brief intro to what we would be talking about.

DR. MAYS: Let me just give a little bit of background, so that I can make sure I catch up the other people who weren’t here for all of the meeting. I realize suddenly that this is kind of coming in the middle of something.

The populations workgroup has, I guess for the past year, year and a half, been working on a framework. In working on that framework, part of what it is attempting to do, and of course, Bill, you can add or subtract from what I am saying, in terms of our space, it would be to help much more on point about how we can increase access and use by actually providing much more detail and in a very systematic way to how datasets are put up. That is a goal that we have.

What Bill has been doing in the Pop group is actually trying to structure this in general. The question that I think is there for us is whether or not this is something that we would want to take on, in terms of looking at do we want this item to go further? Whether or not the framework, as it has been developed, is kind of fine at this point and we say, thank you very much. We kind of put it to rest.

Does this framework, in some ways, overlap in a productive way with some of the goals that we are attempting to achieve? That is kind of the background. You all haven’t read it yet. What I said to him is that I don’t think this is the point at which we can make decisions. Instead, what we want to do is share the work that has gone on in the full committee with you. Have you think about it from the standpoint of what the workgroup’s mission is, which is to make sure that more people more efficiently use the data that HHS has.

What I suggested is that, to give you a sense of this. Then what we could do at a later point is schedule a conference call after you have had a chance to read it. Then we will bring on the members of Pop, like Bill and whoever else he wants from Pop, to be on the call. That is kind of what we are thinking about.

Damon, just some response to whether this seems to be a good next step in terms of from the IDEA Lab?

MR. DAVIS: I think I read a piece you sent before, right? Can you just repeat for me who the target audiences are for this?

DR. STEAD: The target audience are, in essence, experts. Those experts can be application or system developers, or they can be statistical experts, or they can be data intermediaries. They are people helping people access and use data in either manual or automated ways. In the end game, the data structure and the taxonomy are, in essence, metadata structures.

I think the closest analogy to what we are trying to do here is a little bit like the unified medical language system in terms of it is a knowledge resource. It is a knowledge resource that can be used to tag datasets with characteristics and to tag where they sit within the continuum of public health to individuals or health care quality. In essence, it gives a way of externalizing key information about the datasets that would give you a systematic way of either grabbing them in an informatics-enabled more than a human-enabled way. Or even automating comparison of readiness to use for a purpose.

MR. DAVIS: So a couple more questions, if you don’t mind. I realize we don’t all have the benefit of having read the document. Can you just say a word or two about who would be the curator of the framework?

DR. STEAD: That is one of our questions. It could be the National Library of Medicine. It could be something related to HealthData.gov and Data.gov, if it was a useful structure to systematically manage the metadata.

MR. DAVIS: It makes sense and it sounds right. I am just trying to think through all of the different pieces of it.

DR. STEAD: Or does it turn out to be something that is curated through a Wiki. I can see any spectrum. I think that whatever is done needs something to allow it to be systematic. I think that is what has been missing. I think that is what the framework is trying to bring to the conversation.

MR. DAVIS: So in a scenario, I am imagining one of our operating divisions has a dataset that is basically deemed valuable, but doesn’t have a solid taxonomy. It is not usable. It is something that is deemed to be very valuable, and it should be made into a machine-readable format.

I guess what I am trying to get to is if the operating division discovers that their dataset has not got a solid taxonomy that is sort of usable by those experts, and it doesn’t have good documentation about the data fields and all of those things, what then are they to do with that information? They may already, in fact, be challenged to make this data available, let alone machine-readable and potentially at this advanced level of a good taxonomy infrastructure.

DR. STEAD: The framework, as we have been developing it to date, and it can go any direction, is actually at a level above, if you will, the internal data structures or, if you will, the data model that deals with the internals of the dataset. It would, one, say, what is this data set measuring in, if you will, the space of individual data to small set community data to state-level data to national data. Where does it fit in that continuum?

Where does it fit in whether it is measuring an intervention, an intermediate outcome, a summative outcome? Where does it fit in terms of the progression of health determinants from physiologic and genetic factors, out to policy factors? Presumably, it would have coordinates that would actually say where it sat in that data space that would let you understand how it fits into that universe. That would be what the data structure does.

What the methods taxonomy adds are things like, what purpose was the dataset collected for. What voice is represented in it? Is it representing the individual’s voice? It is representing a clinical worker’s voice? So you begin to at least have a high-level view of the biases that are in the data.

It can have an indication of timeliness. How, if you will, mature is it? Because played back to the previous letter that went from the workgroup to the secretary, for the maturity that it needs for different purposes is going to be quite different. How do we actually tag the data set with what it has? Then I can come back, and if you will, tag my proposed secondary use with the level of maturity it needs and see if the two match. It would basically let you externalize it. Whatever part of that information you knew, into metadata that could be then tagged to it and used as people search for things and use them.

DR. MAYS: Let me bring up some of the discussion that has occurred relative to the workgroup. I think part of what we have been discussing is some of those issues, but kind of the complexity of doing it.

I think where you are starting and talking about, the framework, is at a higher level. I am going to take your example. You have a dataset. Then individuals who have put that dataset out, for them it is fine. They think that they are really interested probably in a very specific group of users. They are kind of happening with their dataset.

We come back and we say, well, we think you can get better usage of this. We think that things like even in terms of tagging, that in the tagging, you have this perspective, but the community of users is actually thinking about this quite differently.

Part of the issue also here will be who does this work and for whom the work is done. It is the for whom that is the title wave of we need to get input in order to make it even broader. Or else what happens is if you just ask the user, see, we have to worry about that in terms of the guidance we are talking about putting out. We could just put it out.

If what happens is that we just say, these are some nice things that we think you should do to increase use of your data. But they don’t reach beyond their usual users to make sure that they then are expanding further. What happens is that it still stays very narrow. I think particularly with some of the NCHS datasets, that is the problem. They have a built-in group. We are very clear about the NHIS users.

Yet, the community would like to use it. When they look to say, what has been done with this, it is a language that doesn’t intersect well with what their community needs are. I think this is where rubber will meet the road in the sense of you are at this level. We are at the next level. It really is what would be required for us to do what both of us are asking.

DR. FULCHER: That is a great reframing of that. Bill, I heard what you were saying. As I thought about this, who is the ultimate consumer? I think of the analogy of Home Depot. I walk in, and you can have really an efficient way of all of these different parts and pieces, but I don’t know how to build a darn shed.

The idea is who is the ultimate consumer? Through the intermediaries, the researchers, et cetera, they are at one level that is really working on the store itself. The ultimate consumer is the communities or the stakeholders working in communities. We found that through digital storytelling, it is really powerful. We actually embed the data or the interactive mapping tools. The story is about Chattanooga, and you can zoom into Chattanooga, click on a map. You want to see that same data over in Memphis.

It is an anchor for people who are not versed in metadata and taxonomy and all that to say, hey, I really need that data. What other data am I missing? It engages that end consumer with the intermediary researcher, et cetera. In many ways, that researcher, the intermediaries or the technical assistance folks are relying on that ultimate end consumer anyway on asking the relevant questions. It is the digital storytelling component that really engages around the question, which informs in part the types of data that is needed. I just wanted to build on that.

DR. MAYS: One of the things that we talked about was whether or not there was a way, that when people came to a dataset website, that you could see by using analytics, who goes away. It is like the researcher comes, and they are steadfast about saying. They find their way through what they need because they are kind of use to it.

Then someone else comes to the website. What happens is that it is too hard. Is there a way, like if people signed on or something. Then what you are learning is who it is you are not serving well. That is kind of one of the things. That is who you want to try and gear some of your improvements to. That was part of what we talked about.

DR. TANG: I hear what you are saying about exposing a lot of the attributes of a dataset, so that it could be useful. Are we also talking about a tool to do the matchmaking?

Let me give an example. Let’s say either a community or a researcher or a health care organization wants to know, gosh, how are my patients doing, i.e. patient outcome or health outcome. Now, the question that person might have, not knowing either that much about databases or datasets, or the data sources, they might ask, is this information I am interested, i.e. PROs and EHRs? Is it in some NCHS survey? Can I find it in the Cap survey? Is it answerable by promise questions? Do PHRs capture and answer these questions?

I am imaging that if I just come out saying, hey, I would really like to know the outcomes of joint replacement patients, where do I turn? Is there a tool that could do the matchmaking between my questions and what datasets to turn to?

I will give one more example. If you look at it from a much more informatics question, this came up just yesterday saying, hey, look, we are worried about the clinical documentation and the ways the progress is constructed in an EHR. We think that there may be the mistake associated with it. I would like to know, is there a dataset that can help me answer the question of what the prevalence of potential harms or risks coming out of progress notes that are constructed using templates.

You might add, okay, does the FDA have such a thing? Does some researcher that is done in the RCT have such a thing? Do vendors store this? Does the ONC have datasets like that? That is the kind of questions I might have searching for datasets that could be relevant to my query. It sounds like what you are describing would expose attributes that could potentially answer. Then what about the interconnection? Who translates between my need for data and whether there is an appropriate resource available?

DR. STEAD: You are dead on one of the ways that you could use the data structure, that I would hope downstream the data structure would be used. I would sort of try to connect what you are saying and what Chris was saying. In essence, you would be able to use the data structure to zoom in and out. You could see if I have a database about drugs somebody might be taking, you could then say, okay, I now need to see side effects. It would, in essence, provide a coordinate system in a multi-dimensional space that, in essence, would let you use what we think of as zooming in and out with Google Maps, et cetera, to be able to locate datasets.

I think my gut is this will work at the dataset level, not at the datum level. Once you know a dataset, then you would need to, if you will, open its characteristics in the taxonomy to know how it could be used.

It seems to me I think ultimately this would be a lot like UMLS. Relative to UMLS, it is 1984. We are going to have to have some initial steps. I thought initially, it actually could be used manually simply as support structured conversations between an analyst and somebody trying to solve a problem on one hand.

I think second, even today, the minute the taxonomy was made available in a computerized form, it could be a knowledge resource that could enable automated algorithms for generating the tagging for certain datasets. I am sure once you have a structured knowledge source, then NLP-like algorithms could in fact be used to automate a lot of the tagging, at least believed to an initial degree.

DR. TANG: The question I guess I have is I wonder if we should also be proposing the tool. Let me go to your Google map analogy. It is one thing to put the GPS data, the satellite data on the screen. That is wonderful, but I literally would have to, as an informed human, zoom in to see where there is something of interest.

It is also sort of a browsing function. Rather than the Google Map application that says, hey, describe I am at the Marriott in Arlington. I would like to know if there is a seafood restaurant around. That is the level of knowledge that I have about datasets.

Then Google Map, the app, turns that into, I will find it for you. This is the human address to go. Don’t we need to actually, at the same time, propose that tool set and not just the zoomable map to make it accessible by communities. Some of the people don’t have the data analysts under their wing.

DR. STEAD: The think that is the kind of thing that the entrepreneur community would do.

MR. DAVIS: I am sensing, Bill, that what you are proposing is — we are working backwards, from what it is we want to get to. We are working backwards to how we would get to what you are proposing, I think. I think you are spot-on, Paul, in terms of sort of recognizing that we don’t have the platform or the technology in mind to do it. Bill is sort of painting the picture. Now we have to pick up the tools to actually figure out how to replicate that picture or build up to it or whatever.

DR. MAYS: Let me make a suggestion because I know that the two of you need to get out. Let me take the question, the tents that I see up, and then just do a little bit of a wrap-up, how I think we can work with this. Let’s take Bruce and then Walter.

DR. COHEN: I am going to build, Paul, on your thread and Damon and Bill’s responses with the real world example. The real world example is violence in the streets of Baltimore. There is a data-reporting system called the National Violent Death Reporting system. There is mortality data generated by NCHS. The end user is the policymakers in the city of Boston, maybe the police department and the mayor’s office, maybe the Department of Justice. Those are the end users for the information. Somebody needs to evaluate what data are available to address the information. The fact of death and the demographics of the people who died is in the NCHS mortality files. The circumstances and nature of the occurrence and the incident are in the National Violent Death Reporting System that supplements the mortality data with incident information gathered through survey and combing a variety of other data sources. The structures that we are proposing ultimately would be used by the policymakers. Right now, it is somebody who wants to investigate violence in Baltimore can use this framework structure to see what data are there and what the data look like, and what the data tell you. That is the kind of package that I see us trying to build in this framework.

DR. MAYS: I think I gave somewhat of a similar example to Charlie Rothwell because one of the Congressional staffers came to me, wanting information about violent deaths in African-American men. They found out that states have some national and felt that they didn’t know how to do this. If you think about it, there are real needs for how people can find this stuff, that the tagging is making it very difficult.

DR. SUAREZ: To build on the last couple of words that you just mentioned, the tagging, I think part of your orchestrating your infrastructure concept seems to be very dependent on the ability to have very well defined standardized metadata about the datasets. You were saying you are going to have to go to do some of these queries into what is in the dataset and even drill inside the what is in the data set.

Just about one way to do it is building a very good, robust metadata structure for datasets. I am wondering if that is something that HealthData.gov or some of the work that HHS has done would help. To what extent that needs to be advanced even harder in order to really get to a more granular as possible, standardized metadata of datasets.

MR. DAVIS: I have been trying my best, as I am sitting here listening to this, to get it out of my mind that this is an idea that is before its time, to be honest with you, because of that very fact. There are so many things that need to be built up. The quality needs to be improved at a very basic level, probably before we could get to something as robust as you are talking about.

There are probably some pockets where we could build this out, show the example, and therefore, be able to shine a light on what it is that is achievable with really solid quality metadata, proper tagging, taxonomies and things along those lines.

DR. STEAD: I think you are right. This is what UMLS 1984-1985 was. We are at the early stages of this. That said, if we get the right general structure, and we have the right high levels in the taxonomy, it can be used early on.

I think what you UMLS has taught is that you can automate the vast majority of both the development of the resource and the use of the resource, once you understand its basic structure. We had very different technology in 1984. My gut, which may be wrong, is that this could be robustly used in the way we are talking about within max three years, if we elected to go down this road. I think today, it is not 28 years. It is not six months either.

DR. MAYS: What I am going to suggest as a procedure to deal with this is that we need to get your framework information out to the workgroup. Second, what I would like the workgroup to do is to send a set of questions, just like today.

For example, I think it would be good for you in the framework to tell us and give us some ideas about some of the questions we raise. Who would be the curator of the framework? Who is the stakeholder? I think answering these questions before we have a discussion would be great, so that we are doing straw man on the phone in essence of really trying to go through and look at some of the details.

The reason I am saying that is because what you are talking about has been the discussion that the workgroup has had. It is kind of in a different space. As you talk about automating, we are talking about reaching out further to find how to get the voices in that haven’t been there before.

It may be that we need to find the sweet spot. We are also realizing that when we give this guidance to HHS, we can’t give them guidance that is going to cost a million dollars per dataset. We are also trying to be very realistic.

I think if we pose some questions to you, for you to comment on, then set up a conference call, that we will have a very productive discussion. I think what will be useful in that discussion is to find out the space for us in the workgroup to keep discussing and the space for you in populations to be influenced by what we are doing to discuss the populations.

Are you willing to receive some questions from us and prepare a little bit before we have a conference call?

PARTICIPANT: I would be grateful.

DR. MAYS: Let me do a wrap-up for you of some of the things that we have discussed and see where we want to go with them before you go. I think it is for us to figure out, as we are putting our list together of things to do, whether or not we can have some further discussions about some kind of evaluation or data metrics that would be helpful.

I think the other thing that we want to do is to figure out for you how to get some feedback to you about the website. I have to find that because Jim said he wanted us to look at some websites. I don’t know if this is the one he is talking about, or if he also has some others in mind. We will see how big this website task is.

MS. BEBEE: He wants more than just this website. The exact list of them, I don’t have, but other federal websites.

DR. MAYS: So it seems like we are going to have a website evaluation task that will be before us, which will include yours and some others.

DR. SUAREZ: We could use some framework. In addition to that, we could maybe structure a little bit more the assessment that each of us should do. I think it would be valuable to really have a set of core elements that we all should be looking at. We can then add each of us. Say, I went to this side and found these other things. At least, have a common set of, here, talk about commonalities and standardization, a little bit of standardized questions about what to look into.

DR. MAYS: I agree. I think what we are doing now is seeing is what it is that we are being asked to do by whom, so that then we can prioritize it. I agree that before we start looking at the website, what we need to do is get a sense of what the end goal is and how to do it, so that we are beginning to collect some data around what seems to be kind of general issues in terms of HHS websites. Then what are specifics that different for unique situations. We are talking about the things we are going to put on our agenda to consider for the work plan.

Then when we reach out to do our development of the, we don’t know yet, I guess a session, at the data users meeting, we will talk to them about whether or not there is a place and a space to also do a focus group.

MR. DAVIS: Can I just ask real quickly? I don’t mean to hijack the agenda. Do you know if Jim is coming, Susan? Do you have two sentences that you want to say about this website review thing? I am interested.

MS. BEBEE: I don’t know any more. I just know he is interested in bettering the use of the websites. Everything that you have said, he wants other websites within HHS looked at, just not HealthData.gov.

MR. DAVIS: From a use of their data perspective, the use and access, too?

DR. MAYS: I know it is use and access. Beyond that, I am not sure what. I wasn’t sure what you want. He already kind of commented on, and I chatted with him just very briefly. I don’t know much more than what Suzie is saying.

MS. BEBEE: One thing I would say to you about having other users look at HealthData.gov, one of the things that Josh did for me was to have, and I guess maybe he has boot camps in the summer time, where he has a group of usually graduate students. I was thinking that might be valuable to you if he has that availability.

MR. DAVIS: Do they do a user focus group, or did they like hack on your site kind of thing?

MS. BEBEE: They hacked on the site in a structured way with about 10 questions that we offer to them. It was one of the websites that I oversee, and it was very helpful. It is down and dirty. It is quick.

DR. MAYS: We will make sure we get Josh into this. Then the other thing is, I think the great place to think about what to do would be at the data user’s meeting. It just is really the right set of people. They are the ones that are using your datasets.

Had we kind of done it in advance, you might be able to do it at Datapalooza, but you all have been too busy getting it organized. I think the data users meeting actually is probably a good target for you.

DR. MAYS: All right. Before we let Damon go to do his Datapalooza stuff, oh, we do one more thing, time. Part of what we would like Walter to be able to do in the co-moderation session, since you are giving him some time, is to actually talk about the privacy toolkit.

We have developed a few slides, thanks to Lily, which we will discuss a little bit later. One, I think slide decks are due kind of today. Do we still have time in the next couple of days to get you a slide deck?

MR. DAVIS: I need it today because I am going to be compiling them all into one deck for presenting at the Palooza.

DR. MAYS: We are going to try to work on this then because we actually have the slides. We have comments and what have you. Can he get a couple of extra minutes or what is his —

MR. DAVIS: Walter and I are moderating and not necessarily presenters. You are going to get time, and I think you will be able to sort of judiciously go through what it is that you want to present. The intent is for these to be really sort of high-energy, high-level kind of stuff, not the deep down and dirty. Therefore, I think you will be able to put some key bullet points out, without going into the meat of what it is.

DR. SUAREZ: Just really highlight some of the critical products, particularly our toolkit for community using health data. Certainly, I think I want to mention because I am going to have to say, well, you all don’t know what the NCVHS is. I will say a few words about NCVHS, and then I will just focus on a couple of our products.

The two that I was going to highlight were the work that is being done by population health and community health engagement, and by privacy and security on community health stewardship. Those were the two. Refer everybody to our website.

MR. DAVIS: You will probably get about 10. I had somebody pull out recently.

DR. SUAREZ: It will be helpful if we have a couple of slides. I can show them, or they can download them, too, and have that available.

DR. MAYS: Anything else that you want to ask us?

MR. DAVIS: No. Thank you guys for your time today. I really appreciate it. It is nice to see everybody. Sorry I have got to run.

DR. MAYS: Okay, Lily. I forgot you are on mute. Is there anything before we go? Let’s try and go back to the agenda and reorganize ourselves around how we want to proceed. We have Damon’s update, that is fine. We do need to talk about Datapalooza. I don’t know if you have one of these. I just wanted to make sure.

I want to spend some time in terms of Datapalooza, talking about the slides that we have. Lily created these, and they are for privacy. What was sent around was the ones in which they have your comments on it. What I want to do is talk about the slides, and then talk about, I think, because what Walter said is very interesting, which is he doesn’t necessarily have to flash them all. It could be that the slide decks can only have five. He doesn’t have to use all five, but he can leave five in the slide deck for people to have. That is what eventually, I think, if Damon puts them online, will be what people will be able to access. We want to do that.

Then I want to get Mo to give us any other updates, any other things that he thinks that while people are there, they should see or any things that you think we should watch later or something like that. I want Mo to do that.

Then what we are going to do is go back to our convergence and begin to talk about the sub-committees and their intersection with us. Jim is not coming back, right? Okay. We are good then.

Then I think the last thing we want to talk about is our work plan. Let me just ask Jeannine whether or not she thinks that she can — never mind. I was going to say we are going to put up the work plan, but we are going to have to find our work plan different. I don’t want right now to disrupt the screen kind of thing. Can you send a work plan to Paul, Lily and Mark?

Mark, are you on yet? Okay, how about introducing yourself? I knew 2:00 was your time.

PARTICIPANT: Mark (?) with the National Partnership for Women and Families.

DR. MAYS: Thank you for being on with us. If you could send the work plan, that would be great, just so they have it. We can see. Lily may even think about if she wants to put it up on Google Hangout or not. No, let’s not because it may just disrupt this. Right now, we are in great harmony, so let’s not. We will talk about it. Let’s not push the limits. This is our first baby step in technology, so we won’t push it any further.

We will talk about the work plan. Then as usual, we will kind of end up with people making any announcements, any things that they want to extend invitations to, and then we will end. I am going to try to follow in the footsteps of our new chair and see if I can end this before 5:00.

Let’s start with Datapalooza. Let’s start first with Mo giving us a sense of a big overview about Datapalooza. Anything he thinks that we should know. Then we will go very specifically to talking about the privacy issues. Thank you for coming, Mo.

DR. KAUSHAL: Like previous years, it is going to be divided by a number of certain tracks. I think it is impossible to see everything, so you have to choose what you are interested in on the tracks. I will double-check your question around is it going to be streamed lived or recorded. In previous years, I believe all sessions were able to be reviewed at a later date.

There is impact, there is data signs, there is clinical care and the innovative track. Rather than going through each item under that sub bucket, it is on the web, so you should check that out. What I did want to bring up because there is a lot of extra work that is being put into a set of extra workshops, which is again slightly different and going very deep into a certain number of topics.

The five big topics, and I think probably very relevant for this group, number one, is a privacy and security boot camp, which I think is probably one of the most relevant to the work that we are doing here. If people are going, I think that is the one that I would love to get feedback on because it is something that we are thinking about long and hard in this group. The discussion points we have come up with should be covered there in some detail.

DR. MAYS: Can you just tell us specifically when that one is? I think I saw something where is that the one that is over two days?

DR. KAUSHAL: It has changed around a bit, but Wednesday, June the 3rd from 3:00 to 5:00. I can send around the link. That is the most relevant. Then there is one on Blue Button, as well, which is essentially the standardization of data release. Again, very topical to some of our discussions here. Those two are the most relevant.

The other ones, which I think are of interest, but probably less relevant to this group, are the role of social media data in health care, and then understanding different health data user types, which of course is very relevant to some of the discussions we have been having. We need to double check if things are recorded. If they are, I think focusing on the workshops would probably be the best use of time.

DR. SUAREZ: You are reviewing the agenda for the Datapalooza, the sessions that are sort of more relevant? Is that what you are doing? Did you mention the one about public health care? The public health tract, I am sorry. That was Monday. We have the DataLab from 1:00 to 3:30. Then at 3:20 actually is this other one on public health. That is Monday.

DR. KAUSHAL: It is three days, and then the sessions for each of the tracts are spaced throughout. Again, my best advice would be going to the website, looking at the agenda. The mistake I made in my first Datapalooza, I didn’t look at the agenda before. It gets complicated to go to the sessions that you want to. My biggest piece of advice, pre-look at the agenda. Circle the things that you want to go and see. Many of them conflict unfortunately, so you have to prioritize.

Then stuff I haven’t done, which I would love to do, is the yoga and all the other stuff, which people seem to love.

DR. MAYS: Let me ask a question. Walter, do you have specific things that you are going to? Potentially can we influence you to go to the privacy one?

DR. SUAREZ: You can certainly, yes. I don’t have pre-selected ones, except for the ones that relate to not so much my job, but the topic for NCVHS. If you want me to go to specific sessions, I would be happy to try to accommodate that. I know Monday morning, I am not going to be there because I am going to be flying into DC that morning and trying to get to the 1:00 p.m. session that I am co-moderating with Damon.

Then in the afternoon, I was going to go to that public health one later after our session. Then on Tuesday and Thursday, I don’t have yet specifically any of the sessions. I know I am flying out Thursday.

DR. MAYS: Can you go through one more time the ones that you think are, and just give us a little more description? Then maybe we can see?

DR. KAUSHAL: The workshops are two-hour intense sessions, but they are all at the same time. We have to choose. They are Wednesday, June the 3rd. Maybe one of us could go to the privacy, one of us could go to the role of social media, one of us go through different health data user types. Those three are pretty important. I am guessing Blue Button just may be under.

DR. MAYS: Do you know which one you are going to go to?

DR. KAUSHAL: I was going to go to Blue Button, but I am flexible depending on people’s wishes.

DR. SUAREZ: Are you saying that we have to pre-select the tract that we are going?

DR. KAUSHAL: No, these are workshops on Wednesday, June the 3rd. The other tracts, you don’t need to pre-select. You can just go to what you wish to. Healthdatapalooza.org is the main website. Healthdatapalooza.org/workshops will outline these sessions.

DR. MAYS: I would say, based on the description that the three very useful ones for us would be Blue Button, social media and privacy. I know Walter is going to be there, you are going to be there, anybody else?

DR. FRANCIS: I am not going to be there, but if anything is recorded, I will happily listen to it afterwards.

DR. MAYS: If you can find that out, and then I think we can decide on assignments.

DR. FRANCIS: I am on my way to privacy law scholars.

DR. MAYS: So what we will do is if you can find that out, then what we can do is either make assignments for later or during. I just don’t know if the workshops are actually going to be recorded. If so, that might be great. Lily, are you going to Datapalooza? Okay.

DR. KAUSHAL: Josh should be going because he is presenting a panel.

DR. MAYS: I am sure Josh is still going to go. That is a week later. I am sure Josh will still go. We will do that online, once we hear from you. We will try and see if we can kind of divide and conquer the work in terms of the workshops. I think there are three that are very relevant to what we are doing. Okay, great.

All right. What I would like to do at this point is turn to the slides that you have. Lily, you may want to unmute for this, so we can get you in and out in terms of discussing the slides for the toolkit.

The copy that you have are actually some comments that came from Leslie. I asked her to look at these first, so that we would get some sense of how privacy was thinking about these slides. I would suggest that, Walter, you really don’t need to do all five. Oh, seven, so we have got to delete two.

DR. FRANCIS: I have a suggestion for how to do the whole thing. My suggestion was, I love the first slide, but a little bit reorganized on the bullets. Do you have that, Lily? My comments were about the second and third bullet, which was that I thought they were somewhat redundant.

If the first bullet was the toolkit briefly introduces each important principle of data stewardship for communities using health data, I thought that was great. Then we could say it provides both broad background information and descriptions of stewardship principles. That is kind of like the background.

Then it provides helpful information about implementing stewardship principles and checklists for data users. That is the more practical bullet.

DR. MAYS: What we can do is scan this and send it to you, Lily.

MS. BEBEE: Send me an e-mail and then I will just forward it back to you. I will scan this and send it to you.

DR. FRANCIS: I love the second slide, giving the principles in the toolkit. Then what I thought, you can take out the next slide, examples of what the toolkit covers. We could actually take the data life cycle, and we could put that up in the first slide. It provides both broad background information.

DR. MAYS: I am confused. We got rid of examples of what it covers.

DR. FRANCIS: We get rid of that slide. Then the slide about stewardship and the data life cycle, we put the data life cycle point in the very first slide. We say the second bullet could read it provides both broad background information and descriptions of stewardship principles over the data life cycle.

DR. MAYS: So you are saying take this very first one that is the toolkit that she has up there, and just make the print smaller and then move the picture into it, as well?

PARTICIPANT: No. I think we drop the life cycle picture completely.

DR. FRANCIS: Take the thought about life cycle and put it in that bullet. I am trying to get it to go down. Then we eliminate that slide because we have got to get it down to five, right? Then we just take the title of the next slide and link that explicitly to openness, transparency and choice. Maybe the title of that bullet is stewardship principle: openness, transparency and choice.

DR. MAYS: Then do you leave the picture in?

DR. FRANCIS: Yes. We are explicit about how the slide links to the stewardship principle. Transparency and choice because that is what that slide is about. Then stewardship principle security, we change that title.

DR. MAYS: The next one, ways to improve data security?

DR. FRANCIS: Yes. Instead of saying ways to improve data security, we say stewardship principle data security. Then we have what is in here. The content of the slide can stay the same. I am just thinking about framing each of those slides in terms of the stewardship principle. We are just giving illustrations.

MS. BRADLEY: My thought here was that I mostly obviously went through and just picked out any pictures there were. Then I thought the fact that this data security slide is more of a checklist could give them an idea of like, look, we don’t just provide you with frameworks. We are also maybe providing you with kind of a tool or an explicit list of ideas, just to give them an idea, a teaser of, hey, come and look at our tool kit.

DR. FRANCIS: No, I love that. I just thought the title bullet should link it. Then you could do the same thing to de-identification. Then you are good.

DR. MAYS: That one is stewardship principles of de-identification?

DR. FRANCIS: Yes, stewardship, principle, de-identification.

DR. MAYS: I am going to make a suggestion then. I picked up on the same thing that Walter was saying. Maybe here we should do seven principles of data stewardship. Put the seven there, so that people are clear that, as we are giving them examples, that it is not all of them.

DR. FRANCIS: Before I realized we were limited to five slides, my other suggestion was going to be we have one slide for each of the stewardship principles. I guess we can’t do that because we are limited to five slides.

DR. SUAREZ: If we have already slides for each of the principles, we can put those. The limit on five slides is more about presenting five slides. I don’t think I am going to be able to. I need to do this while I am there. I need to say a minute about NCVHS, a minute about the work that we do across the various committees. Then a minute about population health and the key product there. Then a minute or two about the data toolkit.

In terms of presenting actually slides, I am not going to present seven or eight or nine slides. But we can, in the packet, send more slides. I am sure he is not going to be worried about. He and I talk about the moderating, we were worried about people just presenting 25 slides. I think we can send materials to attach into the packet that is going to be able to be downloaded by people. We don’t want to have like 15 slides.

DR. MAYS: Mo, can I ask you a question? I am just trying to understand how it works. We can send something that is like for posting or distribution versus the slide deck itself?

DR. KAUSHAL: There is a general exhibition hall, whether it is posted. I am not sure if we have got a space for that. Okay, so no. Can we give handouts? Absolutely.

DR. MAYS: Can we send to Damon, and wherever this stuff gets parked on the website? I am trying to make a distinction between sending him a slide deck that he is lining up for the presentation, versus what later people click and get. Is there a way for the click and get?

DR. FRANCIS: To follow up on the first of those suggestions, I think there should be, for the more in-depth slide deck, one slide in the order of each of the principles. If what we are doing is having one for each one, we don’t even need to have the titles say principles of data stewardship. We could just say accountability, openness, transparency.

DR. SUAREZ: After the first slide that said seven principles, then the next one would say accountability.

DR. FRANCIS: My other thought is that the first slide should have a fourth bullet which says, tool kit will be available where and approximately when.

DR. MAYS: That needs to be, I think, also in the one that you are going to use. We need to get these done by the close of business, so that we can send them. That is the five. I am going to assume we have a little more wiggle room to get whatever the in-depth is. You want them both done by the end of the meeting? Okay. All right.

DR. FULCHER: A question about Datapalooza. I presented there twice. As a moderator, it is not a presentation itself when you are co-moderating. I heard what Damon was saying. I am just wondering is that the case, or are there just fewer presenters, and this is an opportunity for the moderators to also. Each presenter only has five slides max.

DR. SUAREZ: We already talked about this. The session is about two and a half hours. It is a very long session. We have about nine or ten presenters. What the session is trying to do, this is the DataLab, so it is trying to examples of the type of data that is being used and the source of the data. This pairing, someone talked about the source, the curator of the data, and then someone is doing the user. We have a sequence of presenters, source and user, source and user, source and user.

Right now in the website actually, the description of that session includes about nine people. I think Damon had the latest version. That is the sequence. We are going to go through the first pair, and then have people ask questions. Some of the data topics are very different. All of them are very different.

The way we are going to start is Damon is going to start and describe some background with the DataLab. Then he is going to pass to me as a co-moderator. I will be able to say about five minutes worth or seven minutes worth of NCVHS.

Then I will introduce the first pair, and he will do the second, I will do the third, and that is a sequence. We could use and we could have slides on my introduction. I am trying to avoid having that. I can have no slides about NCVHS, about one minute of NCVHS. Then have a few slides and say I do want to highlight specifically one of our upcoming tools, which is this tool, and then say a couple of words about this slide. That is as much as I think we can do.

DR. MAYS: Okay. Let me tell you what we are going to do. Leslie is going to actually go out and talk to Lily. They are sending Lily this information. Let them make the seven slides. Then we will wrap back around before she leaves at 3:30 and look at the seventh, and decide the five for the deck. They are actually scanning to you the ones with all of the changes on them. Suzie is going to scan and send you that. Then Leslie is actually calling you. If you could help us produce the seven slides —

MS. BRADLEY: I will go ahead and mute and rejoin the hangout.

DR. MAYS: So if we can now talk about the subcommittees and how you see your work in some way intersecting with ours, if you have questions, things that you want us to put on our agenda, I think that would be great. Walter, can we start with Standards? We don’t know if there are any standard issues or not. Can we start there and ask you if you think there is anything? Or if, for example, the metrics that were coming up might be a part of something that your group would want, or the metrics goes to pot.

Let me just start. I want to make sure standards was there. So far, we haven’t seen anything, but we don’t know because we have never asked the question.

DR. SUAREZ: Thank you. There is something true about Standards is there are always issues. The focus of our Subcommittee on Standards has been quite really centered around the administrative transactions, the code sets, the identifiers. There has been a number of discussions about public health standards in the sense of the messaging standards that are used for things like public health reporting, E-health, E-vitals, those kind of things.

There hasn’t been much talk about the elements, the standards type of thing that more directly relate to the Community Health Initiatives and to the Data Access and Use Initiative. In particular, for example, the metadata standards for data sets. I don’t think that has been really the focus or the role of that Standard Subcommittee.

My concern is that agenda for that group is so full and so significantly — the Standards Subcommittee agenda has been quite full. The Standards Subcommittee agenda and the Standards has not really gone into these other areas of things like metadata standards for datasets or the standards for measurement of evaluating, for example, datasets and websites and things like that.

I don’t think there is a lot of crossover or areas where I think there is going to be significant partnership. There are certainly a lot of, I guess, collaboration opportunities. There have been already some of them done, with respect again to public health and public health messaging standards.

In terms of the type of things that we talk about in standards within the context of the data access and use workgroup, I don’t think there are too many overlaps. Again, I would prefer not to really deviate the attention of that subcommittee with all the work they have to do into some of these other areas.

DR. MAYS: I heard about being careful about putting more work on their plate. The question would be, as the workgroup tries to develop this issue of metadata, whether or not on the fly, we will put it that way, there may be times when maybe what we might want to do is to ask if the standards group might not stay or participate at the very beginning of the workgroup to give us advice, that you act the way that we do to HHS, which is we are not giving you a big workload, but we want you to respond to what we are talking about.

DR. SUAREZ: I think they will. The two co-chairs are members of this workgroup on data access and use. Both of them, Alex and Ob, are supposed to have been here, but they couldn’t.

DR. MAYS: Usually there is only one. They were both here because they were presenting.

DR. SUAREZ: Ob is the person who is representing. He just couldn’t be here because he had to catch a flight also. He will be engaged, and he will be participating in the data access and use workgroup. The deliberations on this, as data standards or standards, even those around metadata, come about, Ob can be the person to bring the perspectives from the data standards.

DR. MAYS: Maybe that is what we will do is ask for some help in terms of the metadata standards for datasets. If we get to this point of also just broadly doing standards for evaluating datasets, that the point person would be Ob, and making sure that when we do that, he is in attendance. I think we are good. Okay. Great. All right. Bruce, you want to talk to us about Pop?

DR. COHEN: So I think the activity of the population health subcommittee really dovetails quite nicely with planning for future activities for the data workgroup. There are three major activities, and Bill already discussed one of them today, which is the framework. The second activity that we have been working on, and the data workgroup has been, too, has to do with community data engagement.

This has been a journey for us. We started out with the reports around communities as learning health systems. We had a roundtable and a workshop. This morning, we finalized a letter going to the secretary, which you have copies of, which I think is really —

DR. MAYS: Remember, they weren’t here. Can you give him a —

DR. COHEN: So this is sort of a milestone on our journey, but a seminal event for us. It is essentially a statement of where we would like to see the secretary and the department go, with respect to community data engagement. A lot of the themes that are in this letter resonate with the work that the data workgroup has been doing since its inception.

You can read this at your leisure or wait for the movie. I will just go through. There are 13 recommendations that we put into three buckets. I will just go through these recommendations. Maybe if you want to comment about how you see them working, the convergence between these and directions we will be taking, that is fine. Or as we move forward, developing our work plan, we can discuss that more.

The first set of recommendations is around alignment and coordination. Those start on page three. The first recommendation is creating a virtual home for community-facing data work within HHS. The notion is, and this has always been an underlying theme, there are lots of folks all over the place doing this stuff, but there is no kind of coordination and alignment. I think that is the substance of the first three recommendations.

The second recommendation is to have the data count while to establish health data coordinating committee focusing on community health data. I could see the data workgroup members being involved in that. The third recommendation is developing a strategic community health data plan for HHS. Certainly, we can use your expertise in helping.

I guess my hope is that the secretary will endorse this and establish a process to create this strategic community health data plan. I hope NCVHS and, in particular, the population’s health subcommittee and the data workgroup will be actively involved in helping develop the strategic plan.

The fifth is really the recognition that health is much broader than the traditional data. Community health data is much broader than the traditional areas that HHS focuses on. We really need to bring in other federal collaborators. The sixth point is we also need to engage non-governmental organizations.

One thing that we really didn’t focus on, that Chris mentioned to me last night in a conversation, is we really haven’t’ integrated the educational institutions in this journey very well. I would like for us to think more about capitalizing on universities and academic centers to bolster our efforts to get data, and particularly federal data, to communities. I think some communities have established relationships with universities with respect to data support and research. Certainly a lot more can be done in that area.

DR. MAYS: So this is number six, you are saying? Add educational institutions to it?

DR. COHEN: For our purposes, I think that would be great. That is alignment and coordination around community engagement. The second bucket of recommendations is around providing technical assistance. Some of you were at the last roundtable. You heard it clearly, communities have an enormous desire to not only get access to the data, but to get training and assistance in using the data.

Recommendation seven is around developing online technical assistance. Number eight really is a little chopped up here in the red line version that you have. It is really around developing and using tools by communities and supporting those activities. Certainly where existing tools help fill the needs for communities, use those. We want the federal government to stimulate entrepreneurs and businesses to develop data tools, as well. We want the federal government, where data tools don’t exist, and communities need them, to help support and develop those.

The example I always use, and I have always been a huge proponent of web-based query systems. There are lots of data online. Some communities have sophisticated capacities to use these web-based query systems or download data and do analytic work. Others don’t. I think we could make a huge difference on focusing on tools that turn data into information to help made decisions at the community level. This, I see number eight as a real point of convergence between the population health subcommittee and this data workgroup.

Number nine, we have never really capitalized, I think, on data dissemination through the regional HHS offices or thought about a way to get more boots on the data ground. We have talked about the agricultural extension service model to help get data expertise in the communities. Others have talked about using the CDC epidemiology intelligence service model to place data intelligence officers, and embed them in local government agencies to help support community efforts.

Just to step back, these recommendations are a combination, I think, of broad overviews that need to be fleshed out, as well as some specific examples that I think we want to push the secretary and HHS to develop more. Item ten is really using learning networks better. We have talked about that some. That came up at the roundtable, how to increase communication and conversations to educate everyone around data access and data use at the community level.

I should have really started with the introduction to this letter was HHS has drunk the Kool Aid. It is time to get the data out there. The piece that they haven’t really focused on yet is, once the data are out there, how can we help folks who don’t know how to use it access it and use it. That is the gap that this letter is trying to address.

The final bucket, which we have talked on briefly before, is from the feedback we have gotten, there is just no awareness of the stewardship requirements as data become more readily available at the community level. We have three recommendations around best practices using the stewardship toolkit and increasing community data literacy around the issues related to stewardship.

DR. MAYS: So for that one, you are going to do 11, 12 and 13?

DR. COHEN: Yes, that is 11, 12 and 13. Susan Kanaan did a phenomenal job of drafting this. I hope it sends a very powerful message to the secretary. It is great that we are opening up the data through a lot of open data initiatives, but the federal government has a huge role in supporting community efforts and partnering with non-government organizations to make sure these data can be used for decision-making. That is what this letter is about.

These themes are certainly consistent with what the data workgroup has been doing and where the data workgroup wants to go. That is the second major effort that the population health subcommittee is engaged in.

DR. MAYS: Bruce, she has to hold at 3:30, Leslie does. Can we quickly do the slides? If we can do the slides and then get right back to you, then we will be good.

DR. FRANCIS: So what Lily and I did was we just took language straight out of the tool kit. Mia has already had a quick look at them. She said there is a lot of info on them. We agree, but because these are background, we think it is okay.

DR. MAYS: You can tell me the five that you want in his slide deck?

DR. FRANCIS: That is a good question. Obviously the first one and the second one. There are two different slide decks. One is for presentation, and the other is for background.

DR. MAYS: He is not actually going to present all five. He is just going to have five in there. Then the seven is what you want to give.

DR. SUAREZ: Actually, there are nine slides.

DR. FRANCIS: I am happy to have people pick out whatever slides they would like as illustrations of the stewardship principles. What there is there are these first two slides. There are two introductory slides, and then there is one slide for each stewardship principle.

DR. SUAREZ: The one slide that is missing is the very last slide that says you should go to here.

DR. FRANCIS: So one last slide says again more information will be available and when. We have got that on the first slide, and we should have it on the last slide.

DR. MAYS: How about we pick what we had before because those were the ones you saw as being kind of really the good highlights that you wanted out of the seven principles. Let’s pick the five because remember, Damon says close of business today.

DR. SUAREZ: Slide number one and slide number two are critical.

DR. FRANCIS: If you are online, we actually have it. We have sent it to you. If you want to look, you can. Go ahead. Just time wise, I want to make sure we can get this done. Slide one is definitely a go.

MS. BERNSTEIN: There are seven principles. Are you asking which of my children I should sacrifice?

DR. MAYS: I sure am. We have been told we have got to sacrifice. Remember, Walter is probably not going to present all of these anyway. You want to at least give him what you think are the important principles that, if he is presenting, that he should talk about.

DR. FRANCIS: One has a diagram, we could do the one that is advancing openness, transparency and choice.

DR. MAYS: Okay. This is three. Two more.

MS. BRADLEY: One thing I really like about this slide, Walter, is just that it explains what notice and consent are. It may sound very typical to you, but to me, this is super helpful. One is just, hey, this is what we are going to do, and hey, this is what we are asking for. I think that would entice me to come and look at the full deck.

DR. SUAREZ: That is very good.

DR. FRANCIS: I really like the community and individual engagement and participation. I think communities will like that one, too.

DR. MAYS: So we have our five then. Oh, I am sorry. That is four.

DR. FRANCIS: Why not do the security one?

DR. COHEN: Actually, if you make those bullets sort of boxes, and maybe put a couple of checks, that might encourage people to think about it as a checklist.

MS. BRADLEY: I will try to do that. I couldn’t find the checkboxes last night.

DR. MAYS: So we have got the five R, the intro one. The principles. What is the third? That is the third one. Your fourth is the community one. Okay. I think these are good. I really do.

DR. FRANCIS: And then the last one is the security checklist.

DR. SUAREZ: I think putting checklists on all of them would be helpful, like Bruce suggested.

MS. BERNSTEIN: I with the second slide, you have the whole list. You can just say, I am picking out three to talk to you about. They are all important, but here are the three that we are going to pay attention to.

DR. MAYS: I think that we are efficient here. As I said, I like using technology. Great. You have the five that we can put in for the slide deck. Then we have the nine that we want to ask him to park up on the website. Who wants to send it? Lily, at the very end, I will talk to you about getting them to Damon.

MS. BRADLEY: Walter, at the very end, you said something about adding the checklist. Did you mean this, just changing these bullets into checks?

DR. COHEN: Just that one slide, Lily, security checklist.

MS. BRADLEY: These are only ideas. You don’t have to do all of them, I don’t think.

DR. MAYS: Thank you very much to privacy. Thank you very much to Lily for pulling this together. Very appreciated. I think we will have a presence. You wanted a presence on the privacy, so we got it. All right. Thank you everybody. Good job. Okay, and we got Lily back.

Paul, are you still on?

DR. TANG: I am.

DR. MAYS: We want to make sure you get a copy of it, as well. Mark, are you still on? We want to make sure you all got those, as well. Lily, when you finish it, if you will send it back out to the list that you had, then they can all see it. If there is any last minute things, then we will make sure that we fix them before we send it off. Okay, everybody, thanks. Great. Okay.

Can we now go back to Bruce?

DR. COHEN: I just want to wrap up with a couple of quick items. Mark and Paul, I am glad you are still on the phone. Anybody in this room, I have given you a high-level overview of the letter to the secretary. This is the letter that is going to be sent, but I would love for us to brainstorm around both the population health subcommittee and agenda, and this data workgroup agenda about what we can be doing to promote these ideas.

DR. MAYS: Bruce, I actually have a strategy for how we will work that.

DR. COHEN: My final thing is the third activity that we are going to engage in on the population health subcommittee is around core metrics. The Institute of Medicine came out with an omnibus report reviewing past efforts at developing core metrics to be used at all levels. They came up with a suggested set of core metrics, one primary set and a secondary set.

We had a discussion earlier about having a fall roundtable hearing or a workshop. We haven’t quite focused on exactly what it would be, but it is time now for the National Committee, I think, to step into this arena to talk about implementing strategies related to core metrics.

I think the focus is going to be inviting government agencies and other data providers. I am being scooped by the visuals here. At some point, as we go through the work plan, I would really like to keep everybody involved and aware of what we are going to be doing around core metrics in the fall. It would be great if the data workgroup can be a partner in this effort.

DR. MAYS: Here is what I am going to suggest. We have got a way in which we want to work on the framework. I am going to suggest something similar in terms of the community data engagement letter. Bruce, I just need you to make sure that I have captured the ones that you wanted us to focus on. It is number two. These are in the recommendations, page three of the letter. For those of you on the phone, it is the letter that you were sent that is called subject recommendation of supporting community data engagement by increasing alignment and coordination technical assistance and data stewardship education.

In that, you pointed out number two, number three, number five, number six, eight, nine, eleven, twelve, thirteen.

DR. COHEN: That was my opinion of what I thought, but I am open to having the data workgroup members.

DR. MAYS: Here is what I would suggest, which is I think it would help us for you to make an ask to us, that is specific to our mission, which is on data access and use. If you can go through those that you picked out as your high priority, and ask us very specifically what it is that you see we can be helpful on. At the same time, we are going to come up with a time in which we are going to go through the list of the recommendations. We will also opine on what we think.

We want to be in the space where, for the workgroup, that we consult to you. Tell us what your specific ask is. Then we will kind of come up with a response to it.

DR. COHEN: I am happy to give you what my priorities are. Certainly, they are flexible if there are places where we think the data workgroup —

DR. MAYS: We are going to do that separately. We will go through them on our own and offer our opinion. At the same time, we want you to tell us exactly how we can be helpful. Remember, the workgroup is working a little different than the committee. We kind of don’t do exactly the same thing.

We want you to tell us what you want comments on. We will answer that. Then, what we will also do is see if there any other additional things that we think because we are just in a different space, we should just share with you. As opposed to just mimicking the workgroup.

DR. COHEN: One last thing I forgot to mention, the planned convening activity will be November 17th around the core metrics. That is the day before our November full meeting. It is probably two days before the data workgroup meeting will be scheduled, so November 17th.

DR. MAYS: What we will do in our work plan is figure out date and space and time that we can do this, and then get feedback to you. At this point, this isn’t like urgent for you, right? The letter has gone off to the secretary, so okay. We are good.

Lily, I will catch you up, so that you can get the notes in terms of what we are talking about in terms of one and two. I think you probably came back on when we started talking about the core metrics.

Again, I think with the core metrics, we will wait until you are further along, and then ask us for how we can be helpful to us once you have those. We will kind of come up with the space that is our space. Sound good? Okay, great.

All right. Jim is here, so I want to take advantage of Jim being here. To go back to this issue of what kinds of things, in terms of HHS projects in general. I know you mentioned this issue about websites. It would help us to get a better sense of kind of what you are thinking about and if there are any other things that you want the workgroup to do. In our charge, our real customer really is HHS. We want to make sure that we take care of HHS.

MR. SCANLON: I think the main focus that HHS would like to use the workgroup would be to give us advice on the data that we have or make available through websites. I think you have already helped us a bit with that.

One thing I think you started is, for many of our major data sets, we have HealthData.gov, we have Data.gov, which kind of harvests all the datasets and tools that appear on HealthData.gov and other places. Those are largely link forms. They basically have a link to data or tools and so on. I think we are continuing to grow that. We have over 1000 datasets and tools on HealthData.gov now.

That, in a way, is the tip of the iceberg in terms of the data that we have. What we started to do, asked Vickie and the workgroup to do, would be to look at some of our flagship surveys and how they make their data available.

In addition, one of our agencies, SAMHSA, which is mental health and substance abuse, it is the same sort of thing. They have some of their data resources up on the web. It is in the form of a, they call it a portal. They have another name for it.

Then we have already started asking you if you could give us advice on where we are going to be revamping our health systems measurement project, which is the website that contains five measures for each of 10 domains in a health system. It is things like health insurance coverage, access to care, preventive care, affordability, things like that.

The number of indicators are purposely kept small. We are going to be revamping that in terms of a new platform and in terms of better display and visualization. Again, I think any feedback and advice you could give us in terms of how to improve that. The nice thing is you can go to those websites when you have a free moment and see how it is acting now.

My own assessment is many of these dataset websites have been built for researchers and aficionados. People know all the surveys. They know their research area. It is like knowing the literature in your field, knowing the surveys that helped you. You have to be an aficionado.

We are trying to reach a broader audience, which you sort of know how the developer and the technology community, what would they be looking for? How would they even know about this if they are not aficionados? How could we get the best? This is the multiplier idea.

The research community, they tell us what they are happy about, what they are not happy about. We have a lot of direct connections to them. They always say it is too slow to get the data, too hard to get the data, so we are working on that.

You have the perspective of the folks who can almost get this data into a much broader ecosystem in terms of applications and use. It may not be presented in a way that would be salient to them. I think that is my concern. If you think there are tagging possibilities or ways, we try to make all of our data available now in some sort of computable form. In other words, if a report contains tables, the policy is that we make that available in an Excel or the other formats. Any ideas you could give us along those lines would be very helpful.

I think you took one look at how the National Health Interview Survey, which is kind of the flagship survey, how they make their data available. It is everything from reports online to table, make your own table, to a search through much more restricted data in a research data center. There is a whole spectrum of how the data is available.

They also have public use files, which are available. You have to go to the website. You have to make an arrangement to download the data. You have to agree not to de-identify anyone. We are trying to do that agreement part sort of virtually, an agreement online. Things like that, any ideas like that, that you could help us to basically get the broader view of the technology community to make this more useful to them.

We have a lot of social media activities at HHS, but it is not really on the data part of it. It is on HealthData.gov, which is now going to be a mobile application, as well. This is on the data side particularly, I think we could use your help.

Then there are some other surveys and datasets that I think we would like to start queuing these up. Just get an assessment to try one or two, and see what the ideas are, what suggestions would come from the group. Then give them back to us and keep it going that way.

We are also down the road going to be revamping the ASPE website, but we are just not there yet. It is a way of how do we get the reports and the issue briefs and everything else that has been done. It is not so much data as it is policy research. It is not so much quantitative. Are there ways to make it more user friendly?

We will be using standard approaches for usability. That is not quite enough to reach the communities. That is one of the things that I think we would like to get. I can answer questions, but I think that kind of feedback from that workgroup would really help us a lot. We really don’t get feedback like that. We don’t really have that perspective. Hearing it from the workgroup would be very helpful to us. It is the sort of thing where we tell you the website we are working on, what we would like and send us your feedback. Something like that. SAMHSA will probably the second one.

DR. MAYS: I think it is the next. Part of what we talked about is that we need some standardized criteria, like when we look at them, like a set of things that we are going to ask everybody to do. We kind of start at that, and we ended up a little stalled out. I think the first thing that we need to do to be helpful to you is to say, what is a set of questions that would be helpful to the examination of the website.

The second thing that would be helpful to us because Damon talked about this is some people have these cycles for their kind of changing things. It would be helpful to us to maybe get from you kind of an order that you would like us to proceed. SAMHSA, I think, is probably the next one for us.

MR. SCANLON: I would like that to have that one be the next one. We are actually going to be looking at mental health opportunities for improvement in mental health data and behavioral health and substance abuse. We may have an opportunity coming to add some new measures to the surveys. Then we will have an opportunity to make those available.

I would like to do something in the area of — our agencies, one of the ways they make the data available, and again, keep in mind the sort of continuum of freely available on the web to some sort of a data use agreement or restricted access to very restricted access in a data center where nobody gets the actual files, you get the analysis, to no access at all. We used to have a distinction between de-identified and everything else. Now, it is not all that clear.

Some of the agencies, what used to be classically de-identified in the old days is now viewed as research identifiable because there is just enough detail there to make it useful. There are folks who make a living writing articles about how they can re-identify. Which means there is a threat there, even if you don’t do it, it is really a threat there. We have to guard against that.

To be honest, when our agencies are evaluating and putting together their datasets and making available, they think about that. Not only is the set internally de-identified, but in view of what may be available, in terms of geography, ZIP Codes, so on, are they revealing anything else that looks like it is de-identified, but may, for someone who has the resources and access to other datasets that are identified, be able to put things together.

We actually had a one-day meeting here at HHS for the statistics agencies on this mosaic effect. In one sense, it was reassuring to hear that. Most of the statistics agencies, now again, this is their business. They have to get this right. One big mistake here, and you have lost all credibility.

They already were aware of this concept of we not only look at vital statistics, datasets or HIS, but we think of what is available during our de-identification and disclosure review. At any rate, I probably would like the workgroup to do something, but I don’t know how we would do it yet, for the public use files that we have.

When the agency does have public use files, is there some way to make those more — again, you have to sort of know where to look. There is no one directory. Is there some way? What would be a way to make those more readily available again to the technology community, but really all of the data user community. Any ideas along those lines? They are already there. In other words, if you go on the websites for CDC and NIH and AHRQ and CMS, they are usually in the section of the web page that deals with that survey. We try to think of can we pull them all together somewhere. They are awfully hard to find. That would be a nice resource. Then if we could think about how to do that, I am not sure quite how to do that.

DR. MAYS: Isn’t that what you want is for us to do things to make people out there more aware of the datasets? Or is it giving advice about how to make the datasets easier to use? Both, okay.

MR. SCANLON: Particularly, the technology community. Making it more available there, the same ideas will apply to make it more visible to the research and the public health communities and so on. There is such a large ecosystem, which you all know. Whatever we think is new that we are doing, boy, it would be great if we had standards for this or that, or indicators for this or that. Somebody has already done it. They are all over the place. Nobody knows what somebody else has done.

When this was part of the open data, the whole idea of making this more visible and transparent would be helpful. There is almost nothing that we have talked about the NCVHS that somebody hasn’t tried already. It is sort of unknown, or it is here or it is there.

Having one website is not the answer. Whenever we think that is the answer, we do it, and we get 10 users. We get an initial bump, and then nobody is going there anymore. They enlisted a lot of work to post some of their hospital discharge files. They called them slim files. They made them really de-identified. We got very few. A lot of excitement, very few downloads.

DR. FRANCIS: Jim, this is Leslie. From a privacy perspective, as data get released and there are all of these new types of data, you were referring to the mosaic and other kinds of issues. Privacy is particularly interested in the questions of new forms of data, including wearables.

Also, Linda wanted me to throw on the table financial information and data interaction. As these things develop, one of the things I think that NCVHS can do in interaction with the working group would help to look at some of those kinds of privacy questions, so that we can make sure the data are used in ways that are appropriate and responsible.

MR. SCANLON: Again, I think our focus is we have data, we have it in different forms available in a spectrum of ways. If you are not an aficionado, how to make it more broadly. At least it looks available. Certainly, to the community, beyond the research community. This is what they do for a living. They know the literature. They know the data. That is not the way it is with others. Some of it would be useful to developers and applications and public health, and some of it won’t be.

This is really the goal of any of our open data. Suzie, I mentioned that when we get a little further with HSMP, we will probably ask them to give us advice on that, as well.

MS. BEBEE: It is in the middle of a redesign, so it is a little bit early. I want to get the redesign done before you would look at it. It is going to change and morph and look differently. The old platform, it is going to a new one.

Could I interrupt here? I sent emails out, which is why I was gone. Can I ask? It is critical about getting the slide deck. Lily, I sent you an email, so you need to look for it, if you would, please.

PARTICIPANT: I am forwarding it to you. You weren’t on the distribution list, sorry.

DR. MAYS: She has actually redone them and sent them back out. Let’s try to have a bit of discussion. Then we can let Jim move on. A couple of things here. One, I think we need to think about the two different sides of this.

One is, and this is what Bruce brings up all the time, is the side of how to get beyond the usual users, how to get to that community that he often is talking about. I think part of what one of the questions has to be is in the context of how the data is, how far can we push out for use.

Then on the other side is the context of is there something different to be done that makes it be able to be utilized differently. I think those are kind of like stepwise. We can start in the beginning with the visibility question the way it is. I think that is something that we could all kind of opine of different groups, how to blast, how to reach into transportation or agriculture or something for them to know about this.

I think it is a different question about the utilization. I think that is the question that probably will take us more time of are you willing to change? I will give you an example. Certain institutes at NIH have decided that when a paper is published, that they also want infographs to go with it, so that you have the scientific article, and then they put on the website these things that they think Middle America will be able to use and understand the science better.

Things like that we might start thinking about how to then make it go beyond the usual groups and push it further out, so then people can say, oh, well, I might want to use that dataset and how do I do that. Then we also have to say, is there a set of tools that will make it easy for them to be able to use it.

MR. SCANLON: Again Vickie, the thing the workgroup can help us specifically with is that technology audience. I think is the main purpose of open data, but if we could do that – otherwise, I think we are sort of throwing it over the transom and hoping somebody picks it up. I don’t think that is going to happen by chance.

DR. MAYS: Let me see if there are questions. Let me start with the people online. Paul, Lily and Mark, any questions for them, or any comments or suggestions about how to go forward on this?

DR. TANG: I completely agree and resonate with what Jim was describing in terms of researchers can access and do access the data. Yet, there is so much untapped, both need and opportunity for communities to explore the data we have talked about in the past. Bill gave a wonderful description of the framework. I think that provides the infrastructure for understanding the data we do have.

I think it calls on us to create this application that connects and interprets people’s needs in a way to bring to them the relevant data. I think you probably weren’t there when we had that earlier discussion. At any rate, I think it is very consistent with what you just presented.

DR. MAYS: In terms of pushing specifically out to the technology group, any thoughts about that?

DR. TANG: We talked about this maybe a year ago. Even the people who know how to work with data don’t necessarily know what the problems are. This application can help them actually even understand what end users, like community users, can need. In some sense, communities describe what the questions they are asking of themselves. This tool almost translates it to what kind of data source in what way can answer those questions.

Then the entrepreneurs can look at essentially this, oh, now I understand better what you need. And I understand where the data sources and the data sets are. Then I, the app developer, the tool developer, can create the connection that renders the infographics that are helpful for you to answer your question.

I think there is this translation service. We talked about the data consultants in the past, that translate even for the technology suppliers what the meaning of the question. Then in turn, the meaning of the datasets that are available. I think there is this interpretation that is needed.

There are three parties, the people who really have needs, the people who have data, the people who have technology and the fourth go-between, I think, is going to be important just as Jim was saying. Nobody can throw either questions or data over the wall and expect people to just find it, pick it up and know what to do with it. I think we are still missing, even if we had the questions and the answers, the data developers need help in understanding the true underlying meaning of the questions and the data. I don’t know whether that was clear, but I think that is an important part of our job.

MR. SCANLON: I am not suggesting that we don’t have dissemination programs because the agencies work a lot on that. The folks we get are the researchers and the public health, the statisticians, which is a world we know. That is not the other world.

I think, for example, the workgroup earlier on gave us some ideas about tagging that are very simple. For someone who is a developer and is interested in health and health care, has a lot of knowledge in the technology area, how would you even know where to look. You go to Google, all right. Even Google has ways of searching and finding data and datasets. There are sub tools and everything.

There are ways of making it more. The tagging was one way. Remember, the PCAST report, even for EHRs, thought that was sort of the internet tagging approach rather than the standards and EHR approach was the way they were recommending to go. It is sort of an internet cloud versus boiling the oceans. We will end up somewhere in the middle and sort it out. I think that would be very helpful.

The other thing is, as Paul said, once you start this, then you learn a lot more. You have more questions about more data. You say, oh, now I see how that could apply to health care or my community or to my state or the question I have about my own health or something like that. We are still, in many ways, with the whole open data initiative, we are still, if I think of it as life stages, we are probably entering adolescents.

The kinds of applications you see at Datapalooza and other places, they are beyond the student application. Some of them are serious, heavy-duty applications. It is not adulthood yet. That is the kind. You need the data, you need the information to be able to do that.

DR. TANG: Can I make a comment to follow up on the example Jim raised of Google? What is interesting is that Google has the tag about the data. Google also knows the questions being asked. I think that makes all the difference. I wonder if there is something for us to learn here.

One of the ways that is translated, we used the example earlier of Google Maps, Bill did. The map, you can zoom down and try to find things. In a sense you are browsing, but Google also knows that people ask about restaurants around my hotel. They make that information available.

In a sense, that is sort of this other role. In addition to Google, one, finding out the datasets, and two, finding the tags associated with them, they also know the questions that people ask. That is the whole basis.

DR. MAYS: Any other questions around the table? Anything else, comments?

DR. FULCHER: A comment about getting a critical mass in terms of people accessing data. Community comments now has over 20,000 registered users. We are finding when people are doing searches in the search engine and articles that are part of the community comments hub, it gets to the top. Really, it is not just dealing about data, but a strategy to really drive traffic to those data sources. It really bumps it up in these different search engines. That is one comment.

The other is we talk about data. You mentioned, Jim, about broadening the group of users. It is typically the usual suspects now with researchers and people using. It is just how data is manifested. It is this idea of what is data. I don’t want to get too philosophical. It is Friday. What is data? It is the zeros and ones. It is the bits and bytes. That is data at its core. Then it is turned to information when you have members. To other people, that is data. Then to other people, that is information. Then it cycles back.

It depends on again who the audience is when you talk about data, and around mapping and data visualization or data narratives. That is where I talk a little bit about digital storytelling. Depending on the broader audience, they may want to see how that data is manifested in visualizations or other types of ways. That is just a comment as we think about how we want to reach a broader audience to attract people, not to just core data in our relational data base systems with metadata that is not going to resonate. But how do we drive them to that point through these lenses of data visualization and digital storytelling?

DR. MAYS: Anyone else?

MR. SCANLON: I think Damon would have something to say.

DR. MAYS: He was here long enough. He gave us some very specific things, as well as us asking him about things like data metrics and kind of how he is evaluating what he is doing, and kind of even the input that we are making. I think in terms of giving him feedback and back and forth, I think it will be good to even begin to get a sense of him being able to tell us, well, that strategy worked for this group.

Again, it is kind of like what Paul is saying. As we start to learn more, then we are also more effective at trying to look at particular datasets and say, if you have this unique feature, this seems to work. If you don’t have that feature, then this is probably not worth you wasting your time doing X and Y. I think if we can also work with him on that, that would also help us as we go along, as well. Anything else? Okay, you are free to stay with us.

Okay, folks, I think we are doing good. We are getting a big work plan here, so that is why we are going to have to, at some point in time, spend some time really figuring out what else we can do.

We have heard from standards. Debbie is reminding me. We should have a break. How about a 10-minute break? How about we make it five? We just need to talk a little bit about the work plan. That is really all we have. Let’s take about a five-minute break. Then we will come back.

(Brief recess.)

Agenda Item: Next Steps, Work Group Member Announcements

DR. MAYS: Let’s think about what we have heard. I am going to try and do a little bit of putting out the task list. I am going to tell you what I think the first priority is and then have us after I get the notes to work through the plan because I think it is a little too hard because everything is all over the place to make sure that we can figure out over time how to do it and also we have lost so many people. I want to make sure we do this as a group.

Lily, you help me out. If there anything I missed that you remember or anybody else, just put it out there. Task wise. Here are the things that have come before that we have been asked to do. Then we are going to go back and prioritize them. But I want to make sure I have everything.

We are going to think about the issue of data metrics. This is specifically in working with Damon. Here, what we are talking about in terms of data metrics is really starting to ask the question of what is making a difference. Damon comes to us. We give feedback. We give him ideas. It is like throwing the spaghetti on the wall and seeing if it is done. I think we are at a place now where what we want to do is concretize these things and start to figure out what seems to be effective for whom.

In that task, what I think we want to do is low-hanging fruits. I think that was a great suggestion where I think it was Kenyon that was saying aren’t there some things that we can do. He talked about the four C’s. We brought it up again. Can you figure out who came and left? There are analytics that are in the background that we might take a look at and say how about collecting some of that and letting us see what the results are.

There is a more complex level and that is actually developing some metrics and trying to figure out what those should be. I want to think about the metrics questions as there is low hanging fruit. There is much more detailed approach to that as well.

The website evaluation. You could hear what Jim had said, what Jim wants and what I think Damon is asking. They are not exactly the same. I think that there is — in the Venn diagram, there is a great deal of overlap, but they actually each have some specifics. I think with Jim what he is asking is the two-pronged question of how do I make these data sets more visible, but he is also asking the utilization question. I think I commented before. I think each of those has a very different set of demands. I think of how to make it visible, we probably can again low hanging fruit come up with some ideas about that.

The other I think in terms of how to get people to use the data, that may require us to make comments about the data being in different forms, the data having different kinds of information that goes with it, the data even being presented in a different way. I think that that is part of the work that he is asking for.

I think on the side of Damon when he is asking for the review, he really wants to know what do you think of this website for HHS. He is not asking us to go to the further out users. I think he is asking in general how functional is this website, how appealing is this website, and how easy is this website to use. My sense was that Damon’s questions were at that level.

The review is going to require that we develop some kind of protocol so that as we do it, we are all doing it in the same way and that again we can even evaluate that.

One of the things I think we need to do so that we don’t stay in the same space unnecessarily long is to evaluate ourselves. I want to try and build in when we do an activity to look at it and let it run on its own with something in the background, evaluating how well this is occurring. And then we either pull the plug or we make some changes. That is part of what I am also going to say about our projects.

The framework discussion. I think we have laid that out. I think that that requires everyone on the committee to be familiar enough with it. It requires that we come up with a set of questions and we will follow that up with receiving some answers from Bill and then having a call. I think that is where we are with that.

I think as we move to the rest of the subcommittee, I think with standards, we want them — they have a very full plate. I don’t quite know how they are going to do it. Walter now has this other stuff to do. Alix is great. Just give her time and she is going to be there in terms of moving it along because I can see that already.

I think with them what we want to do is we want to take advantage of their expertise on things like the meta-data standards. That is going to come up within the context of the work that we are doing around the guidance. And I think it is also going to come up within the context of intersecting with populations with Bill.

DR. FRANCIS: It is also going to come up with respect to meta-data for data segmentation and privacy issues.

DR. MAYS: That is coming up for pop as well then. I think that that is great. Kind of nimble working with them and trying to move that piece ahead. We will know then when to consult with them. I think we will put that on hold as to we need them when we reach that point and then we can just make sure planning wise we let them know ahead of time.

I think the other piece in terms of pop is the community data engagement. What I see as the work in terms of the community data engagement is again — a part of this is really how to get the information out. It is a marketing, using the tools that we know. I think that is part of the question that we are being asked.

But at another level, we are being asked how do they do it differently. I think it is very much the Jim question.

The approach for this will be pretty much to ask him because we have a lot of — the 13 recommendations is to ask him to really tell us in our space specifically how he thinks we can help. But because they don’t our space as well, I think we look at those recommendations and bring them back to him as well with some questions. I think that is where we are and then we will see where we go.

I think Jim was real clear as to what he wants us to do which is we need to get back to where we were on the development of guidance. We really do need to come up with a set of questions that we want to ask people when they come in to present their data. We want to figure out how we respond to them, whether or not we give them comments right then and there or whether we prepare something. In the workgroup what we had talked about before is sending them something before they come so that we have again some standard of what we are looking at as opposed to just relying on them to talk about it because it depends on how sophisticated that particular group of data conveners are around these issues. We need to push the envelope and decide what it is that we want to ask them to pull together to give us before their presentation.

I think this is where there is a lot of work to it in terms of it, but at the same time, it is clear that this is where Jim wants to see us going and that was the original development of the workgroup. I do think that for me that becomes a bigger priority to start getting back on track and pulling those things together.

Those are the major ones. Is there anything that I missed that people see as task that we should be thinking about?

MS. BRADLEY: Just one. Maybe you already covered it, but there is the user conference in August.

DR. MAYS: That one I think we do that in conjunction with you.

DR. SUAREZ: That would be more the National Committee.

DR. MAYS: I think we can give more that over to the Full Committee. But I think what we also want to do is to talk with NCHS about the focus groups. They actually may be able to do them. They are used to doing focus groups. They have individuals who do this quite a bit for them in terms of both their cognitive testing unit as well as just focus groups in general to get feedback. This may be a suggestion and we can see whether or not they can pick it up.

MS. BRADLEY: Isn’t there overlap between what we are doing in terms of creating a standardized set of questions and what a focus group would do? Perhaps I don’t fully understand what the focus groups are.

DR. MAYS: My understanding of what Damon wanted for the focus group is that he would like some of the individuals at the meeting in August who are — remember, they are going to be very experienced data users to pretty much just give him some feedback off the cuff about the websites. I don’t even see where necessarily we are going to be doing a lot of that. I think that for us it is to get him the data users. It is like just connecting him. I think it is for the IDEA lab to decide what he wants to ask of those users.

MS. BRADLEY: If they have a standardized set of questions that the folks who run the focus groups could maybe look at them and borrow them.

DR. MAYS: Again, this is expertise that is at NCHS. They usually are not asking about a website. They are typically asking about a specific question. They are asking to turn the question into a researchable very specific question. Often, they are going from you have pain to how am I going to ask you specifically about this pain so that I can keep everybody with the same metric of thinking about pain on a scale of one to ten or something like that. There is variability between men and women or young and old or something like that. It is a little different than I think what he is looking for.

But a question would be if you could put in our notes is whether or not Damon has a specific set of questions that he would like to see answered so that he can have those ready for the focus groups.

DR. FRANCIS: It is hard to figure out how to put this into a particular task. One thing to be sure we think about across the board is the application of stewardship principles. I would just have it a mega-question in terms of having in the background. We have the capability to issue spot for healthdata.gov about whether particular types of data or releases or possibilities raised especially noteworthy stewardship questions. That should just always be a question we are asking in the background. That is not like a particular —

DR. MAYS: Can you give me the specific — I did not get it in terms of — what would be the question that is in the background?

DR. FRANCIS: The whole stewardship toolkit. Are there — I will just use the identification and the question. Are there any worries about particularly data sets given what other data are available that we now — or even interactions between HHS data sets?

Another one could be downstream uses of data, making sure that even though the data that get released are community data sets. Are there appropriate safeguards in place to make sure that people don’t re-identify de-identified data?

DR. MAYS: I get it. This is what would be useful, Leslie. I know you are moving in space now. You are not some place where you can actually type these up. But to take your seven principles and generate the questions and those are questions that we could then make sure get passed on. For example, those are part of the questions that we could ask our customers. Those are part of the questions we could say to Damon here is some of what we are going to worry about. You are worrying about the website, but do you also want people to comment on these things in terms of finding these answers on your website. If you think that that is significant enough as to why someone would or would not want to use the data.

DR. FRANCIS: I will be back. I am still on the phone, but obviously, I am going to be on mute because my airplane takes off in about 20 minutes. I will work online and send you all of that tonight. I am going to back on mute and keep listening.

DR. MAYS: Thank you. Anything else that you think that we have left out? Here is how I am going to suggest that we work. After we get our minutes and stuff like that, I think that some of this we should do in smaller work groups because you have very different expertise. I don’t think that we want to have you working on things given your time on things that you could contribute, but you know that your time is best used better over here. I am going to come up with here are these buckets like I am going to use as an example. Chris really has built in expertise to go and examine a website. We might say we are going to have a work group and this is what we are going to do in this work group and Chris will be in that work group. That makes sense given what he could do without it becoming a burden for him. This is where my remembering part of what you all do as your skills. I am going to try and come back to you. If I give you something and you say that is really boring for me or that is just not me, it won’t hurt my feelings. I want you to tell me because I want you to do the task, number one. But number two, I want you to do with expertise and efficiency as opposed to just being a part of the group.

We may reach out a little bit beyond this. One of the things I might tell you if you are in a particular group is can you get a few other people to join you and give you some feedback. Can you shoot this off to some other people because I don’t think there is anything I have to worry about yet that it is so confidential that it should not go beyond the work group yet? That is how I am going to approach it because you have networks in which you will bring some of the best and the brightest to that network. I think that would be the best use of your time in terms of making assignments.

There is one other thing that is not on here that I looked at Kenyon and remembered. We talked about it when I was presenting to the entire committee. And that is once we get this guidance done on data use and access, we want to do something with it, which is like put an article out there. You know how your trade publications often look at these things. That is what their read or we are going to do a scientific publication or something that also gives our guidance some credibility beyond that it is just posted on a website. But we want to put it out there somewhere where either it gets read and then you see people blogging about it and discussing it and agreeing and disagreeing with us or you want to put it in a scientific publication where it raises the bar in general so that it is not just that we set it. But it is also that because it has been peer reviewed, people begin to think about it as having that credibility to adopt it. That is the only other thing I think in our workspace. Again, that may be in Kenyon’s wheelhouse. Unless you are opposed to that, what we have talked about, that is kind of how I think I will try and approach this.

If you have any other things and you don’t want to just openly in the mike say, do that or something, email me. But otherwise, I think I am going to have you be pods and you are going to get your buds to join your pods. We are going to multiply how many people we have looking at something that are experts as opposed to the whole of us are looking at something and I am going to slow you down because I am ask you ten questions. If she would just let me go, I could get this finished. I prefer to let you run with what you are strengths are. How about comments on approaching the work that way?

DR. FULCHER: I think that is great, the way you suggested that and taking a discrete piece so I have a sense of what I need to do, timeline, when I report back. I think that would really help because if it is a long list, you don’t know how much to be engaged or not engaged in the different parts. I like that approach.

MR. CROWLEY: That makes sense to me. I think as part of this work plan if we can define what those discrete tasks are, put some timelines around it, put the people to the projects and then go through some sort of exercise to make sure that those expectations meet with the folks’ expectations. Then if we can define those projects and hopefully be ready to roll.

Building off some of the guidance documents that we had from the last meeting where we brought together ideas from the committee over several discussions, from some peer reviewed publications around open data access and some other resources from the NCVHS — existing community data infrastructure. All of this is coming together to frame some guidelines and some recommendations on open data. It might be one of those next steps to take that and put it into a format that is digestible and usable by different data stakeholders at HHS or beyond.

The only other comment I will make and whether it is for this cycle or future cycles. I think a lot of what is discussed around understanding stakeholder needs or the demand within the community or generating interest involves putting in place mechanisms to have that bidirectional flow of information with the community and with HHS. Whether that is a community data meet up place or some other method, but that allows people when they have questions about data, they can put their questions forward with those that might have technical solutions or other types of expertise related to those questions or the data that is out there that they are participating and providing some input there and just making that easy to use. Once it is there, there will need to be some sort of promotion of it. Whether that is a small marketing budget to make sure ads are going to places where they need to be or conferences where it needs to be. These things need to work hand in hand.

DR. MAYS: I want to pick up because we talked a little bit about this in the work group in terms of the guidance that we give. In that guidance, you would want people to have FAQs or blogs or something so that as people come to NHIS or NHANES or something like that that community starts to populate solutions, et cetera.

What you are raising — we have to think about a way for the committee to get this bidirectional information because we keep talking community, community, community. I am often one of the ones of who are you talking about. Even though we have public comment, we may think about one of the feedbacks that we want to give to some of these projects is that they have a way to have some input. For example, for the pop one, it may be that that is what they need to think about. But they have to really think about it because when they open up for comments, I forgot what the last issue was. It is the review. How many people, for example, have decided in this review that we are going to be doing relative to ACA that they want us to hear them.

I think it is something that is a recommendation from the work group back to the full committee. I think the full committee needs to struggle with that. I think it is an important issue, but I don’t think that we have a good sense of how to do it without being overwhelmed.

DR. FULCHER: That is a great comment. I am wondering though when people make public comments, do we know who they are in the room? Do they identify themselves?

MS. JACKSON: They have to be identified. They are included in the record. The information is in the minutes and the records. We have a clear understanding of who they are. Most of the time we know the people. They are the usual players, but we want to make sure everything is up. The communication that often that they raise might be something people are thinking about who are out on the web, which is as communication and response back to them.

DR. FULCHER: Thank you. That is the struggle we have had for a number of years going back and forth in terms of our websites of having a user name and password. You have an identity of who the person is or just having it completely open where people can provide anonymous comments. That is something that has worked against us is not just having — you should have the same level of identity on the web as you do in person in terms of public comment. That is something that dovetails into the websites and how you offer comment online or blogging to comments and getting that input back to the committee. I think that is something that needs to be thought through. I don’t know where we are at in terms of the new health data.gov, what is required, what is not.

MS. JACKSON: In terms of the National Committee, the privacy toolkit is really our first fore into aggressive marketing of really getting the toolkit out electronically to all kinds of neat LISTSERVs. We at one point had a LISTSERV that did not really — stakeholders. When we transitioned to gov delivery then those are people who have more alignment with what we are trying to do.

A second layer of what we are anticipating and what we want to do with the toolkit is to get feedback like you are saying, a two-way communication. We are not at that stage now, but that is definitely on target. The fact that we can do something like this and bringing Lily and others on, the fact that we had a roundtable that brought in so many people just last fall that had a very different set up than what the committee usually does. They are small steps, but this is definitely on our target.

That is what I wanted to bring to your attention. The aggressive marketing or aggressive way of getting our materials out. If I had a chance to mention in the full committee, we were going so fast with so much amazing productivity. This is the hardest work of the committee that no one really knows about. That is kind of what people have called the committee. They look at the materials and say who are you. Where did this wonderful stuff come from? We are looking at all kinds of ways to get passed that hurdle so that people will be aware of our products, the toolkit, and what the work group is doing will I think thrust us forward in this year and on.

DR. MAYS: Let’s give last comments to those online because we are going to end early so that I can work with Lily to get this stuff off to Damon. We are going to make sure that close of business is five. Paul, Mark, Leslie, any other comments you want to make before we end about the approach?

DR. FRANCIS: No.

DR. MAYS: Going once, going twice. How about a proposal to adjourn? Can I get a second? Thank you very much for your time and thanks to those online.

(Whereupon, at 4:40 p.m., meeting adjourned.)