[This Transcript Is Unedited]

DEPARTMENT OF HEALTH AND HUMAN SERVICES

National Committee on Vital and Health Statistics

Work Group on Data Access and Use

November 19, 2015

National Center for Health Statistics
Auditorium
3311 Toledo Road
Hyattsville, MD 20782

TABLE OF CONTENTS

P R O C E E D I N G S (1:05 p.m.)

Agenda Item: Welcome

DR. MAYS: Good afternoon. Welcome to everyone. This is the workgroup meeting, the NCVHS workgroup meeting. We have people online and we have people here in the room, and we should have several people who are going to pop up on the screen. I hear there are some shy people who didn’t want to pop up on the screen, but we are going to see who is who.

I would like to welcome you to the meeting. Part of what the mission of the workgroup is to do, which is the Workgroup on Data Access and Use, which is to come up with innovative ways to help HHS utilize its data. It’s to increase ways for people to access that data. So we have a variety of individuals who work with us often in the role of either as staff or consultants to help us achieve that goal.

I’m going to ask people to go around and introduce themselves, and we put some time. So given that this is the first meeting for several people, tell us what you do, the kind of space that you like working in in terms of areas that excite you and that expertise that you have, because it will help us to get to know you and it will also help others to be able to connect with you.

And just so that everybody who is new knows, these are live microphones that are actually, you’re being recorded, and in addition, we have an audience so the public is actually also out there as well as some of our members. So we are actually a public meeting and so if you have any particular trade things that you are concerned about, it’s public. So just remember that, because I know many of you can be in the midst of development and stuff, so I just wanted to tell you that before you share certain things.

Okay, let’s start here.

DR. RIPPEN: Helga Rippen, president of Health Sciences of South Carolina, member of the full committee and now member of this working group, also supporting population, population health subcommittee. Key passion, appropriate use of technology to improve health, and transforming data into actionable knowledge.

MR. GERSHMAN: My name is Greg Gershman. I’m the CEO of a company called Ad Hoc LLC. We are a small software development consulting firm. We do a lot of work with government agencies in particular. We work currently on the new version of healthcare.gov with CMS and very involved with a lot of the efforts there. I was in the past Presidential Innovation Fellow, have done a lot of work mostly in taking open data and making it useful in some way or another. I have been doing that for about five years, the past five years, in various different contexts in the public sector.

DR. MAYS: When you say open data, give an idea of what kind of data.

MR. GERSHMAN: One of my first jobs in this sort of stint in my career when I have been working with government was with GSA, and we built a recall search engine. So we took recalls from FDA, from Department of Transportation, a bunch of different sources and aggregated them all together and built a search engine so that people could access them and just search through them very quickly and find recalls and things like that.

And then when I was at PIF, worked with a bunch of people on open data stuff and have been involved to some extent with open data around the marketplace with healthcare.gov.

DR. MAYS: Great. The reason I’ll ask you sometimes these questions is so that some of the committee members will get ideas about how some of what you have done might be helpful to other things we are doing on the full committee. So some of us understand, and some of us it will be great for them to have the details in order to see, wow, maybe my project might be able to fit into that kind of category. So I’m not grilling, but I’m just making sure we have examples for people.

DR. COHEN: And I have an ask for you already. My name is Bruce Cohen. I am on the full NCVHS committee and cochair of population health. I’m from Massachusetts, mostly retired from the Massachusetts Department of Public Health. I am very interested in getting data to communities to support community decision-making. We had a wonderful conference on Tuesday where lots of community folks said there are data sources out there that they are frustrated; they know they exist, but they can’t figure out how to use them, and they are not in an easy to use way.

For instance, ACS, the American Community Survey, has an enormous amount of neighborhood level data, community level data, Census track data, but folks in communities just don’t know how to use it. So there are examples that I hope that will engage this data workgroup in helping identify models or ways to provide data in a more useful form to less sophisticated data users.

DR. MAYS: Did you have two data sources you were talking about? ACS, and what was the other one?

DR. COHEN: The other one was vital statistics. Not all vital statistics are readily available births and deaths, but as they become more available at lower geographic levels, figuring out how to create a portal or portals or products that will allow people to manipulate the data fairly easily.

DR. STEAD: I am Bill Stead from Vanderbilt University, and I’m a member of the full committee and Bruce’s sidekick as cochair of pop health. I have a spectrum of interests that go from my day job to my role on the committee in that in my day job I am very interested in personalized care and pop health, and have spent an increasing amount of time trying to figure out how we know enough about social and behavioral determinants, in addition to genetics, to link people in a two-way fashion to the right resources to help them address their health.

In the work we are doing with the pop health subcommittee, we are really interested in how we enable communities to be effective learning systems, and these two ideas work together. The former, my day job part, is more about how we help individuals.

My NCVHS hat is more about how we help the community develop capacity and therefore more about groups, but it all ties together, and then part of my dream that I think does relate directly to the data workgroup is that individuals will — we will be able to bring together the rich array of data that is increasingly just part of all the digital stuff we carry around and in our homes to allow, to create the information environments that allow individuals to manage their life, their stress, their health, and connect to healthcare when they need to. So that’s the space I’m interested in.

DR. MAYS: Can you take a moment and say something about the framework, particularly since you won’t be here? I think that will be helpful.

DR. STEAD: Yes. The pop health has been facilitating development by the full committee of a set of data classification resources that we call the framework, and in essence the idea was could we build a data structure on one hand that would let us see the relationships of data captured at different population scales, and could we create an ontology of the attributes of datasets that if you could tag an available dataset with where it fit in the population scale and of specs and with attributes about the type of data it contained which can be, for example, what’s the voice captured in the data. Is it the voice of an individual? Is it a voice of a clinician? Is it a voice of an administrator or whatever?

That if we tag datasets with that information, we would be able to make it more reusable across the different kind of things we are trying to do, and we have given you, I think, the current version of a white paper that’s trying to get this concept down. I have mentioned it briefly in some previous meetings. We would love any thoughts you have about how you might use, if such a thing existed, and if people began to tag datasets with it, would it affect how you went about thinking about building the kind of products and tools you build, because if not, it’s probably not worth worrying about.

If the answer to that question is, well, there are some things, some lightbulbs that go off, and those would be things we might — then the other thing you could help us with is what are your thoughts about how we could either automate or crowdsource the development of such an ontology so that it was, in essence, done as part of a networked activity, not so much as part of a top down kind of activity. Those are the two things we thought you might could help us with when we have time.

DR. MAYS: Okay, so that will be on our agenda a little bit later, and we know that you won’t be able to be here. So that’s why I wanted you to take a few minutes.

MR. CROWLEY: Good afternoon. I’m Kenyon Crowley. I’m a member of the working group. My fulltime job is as deputy director of the Center for Health Information Decision Systems. We are a health IT research center at the Smith School of Business at the University of Maryland. So broadly speaking, our work is focused on how we can support the design, effective use of information systems into healthcare, and more specifically some of the key areas of interest in recent years are around healthcare analytics, how we can harness data from different parts of the healthcare system to make sense of that and help with decisions for both organizations and clinicians but as well as empowering patients with their own ability to make decisions in their health literacy.

So do a good bit of work in sort of digital health technologies and mobile health. So we have some ventures in the diabetes mobile gaming space that have shown some interesting results in terms of efficacy, do a good bit of work in community health improvement initiatives in terms of helping community organizations better harness the data in their communities and work across different partnerships to embed some of the social determinants of health into the overall sort of system of public health. I’m also a doctoral scholar at the University of Maryland iSchool where my focus is primarily on precision behavioral health interventions.

So how do we customize the interactions for different population groups on a range of individual differences, from clinical to behavioral as well as sort of psychological in usage and information needs and gaps they may have across time. And I teach some classes as well in health IT strategy as well as mobile health. I’m fortunate to work with a great team of folks. Even though we are in the business school, for many of these projects we are typically partnering with computer science or public health or pharmacy or for medicine and do a lot of work with startup companies as well to help validate their products and sort of help them in their journey as well.

MS. BRADLEY: I am Lily Bradley. I used to work at ASPE as an innovation fellow and help to support the data access work group. I’m now back out in California and excited to be continuing to work with you guys. You probably know more about our HHS datasets than anybody else, possibly in the world.

MR. COUSSOULE: I am Nick Coussoule. I’m the senior vice president and chief information officer for BlueCross BlueShield of Tennessee. I’m a new member of the committee and a recently recruited member on a couple of the subcommittees, particularly the data access and use in security and privacy.

MR. LANDEN: I am Rich Landen. I am also a brand-new member of the National Committee on Vital and Health Statistics. My day job is with Quadramed, which is a small medium-sized software developer in the HIT space with a range of products. My data background includes health insurance plan related standard claim forms, back to the dark ages paper standards, and the last couple of decades that morphed into electronic data interchange and what we know as the HIPAA administrative simplification transaction sets.

MS. BEBEE: I am Suzie Bebee, and I work in Jim’s shop at ASPE and in the data policy piece. So anything data is of interest to me. I work a lot with NCHS. Some of the stuff you heard that Charlie talked about, I’m engaged in that, ICD-10 standards, how they are actually going from ICD-9 to 10, also how the surveys at NCHS are using standards like HL7 and the CDA. We have some pilots going on there out in California, and I’m staff to the subcommittee on standards and have been for 16 years.

DR. MAYS: Let’s see who is online. Damon?

MR. DAVIS: Good afternoon. I am Damon Davis. I work in the HHS IDEA Lab, commonly known as the chief technology officer’s office. I serve as the director of the Health Data Initiative, and in essence what that means is I am a liaison out to the rest of the department in terms of trying to catalog datasets that we make available on healthdata.gov. So I’m frequently in search of datasets and trying to disseminate some of the policy information from the White House OSTP, OMB, GSA, about our open data strategy to my colleagues across the agencies. So just generally on a mission for open data sharing and trying to make sure that it has multiple uses.

DR. MAYS: Great. Who else do we have online?

DR. KAUSHAL: I am Mohit Kaushal. Good afternoon, everyone. Nice to meet you all. So I was part of the advisory board group last year and continue to remain so, and so my background quickly, physician by training and practice in the ER, and the mid part of my life investing in venture capital. I then joined the Obama administration early in the first term, was in the White House and then the FCC, heavily involved in the technology reform. After leaving my main day job, now I’m back in the investment world focusing on data and analytic companies within healthcare. Two other parts of my life, I’m also a fellow at Brookings, so heading up a whole piece of technology enabled care, so what are the policy levers that we can implement to enable this whole space. Interoperability as you can imagine is a key part of it.

And then recently joined the faculty staff where we are starting a whole new group within bioinformatics focusing on AI to help get data. It’s nice to be here.

DR. MAYS: Thanks. Who else is online? Leslie? Chris?

MR. DAVIS: Not sure what happened. This is Damon. At one point I just got knocked off and jumped back in. So it’s possible that the same thing happened to the other folks.

DR. MAYS: Okay, I think what we are going to do is let them actually get them up again. Damon, can you hear us? Damon, let’s get started with your presentation, and then we will loop back in with the other individuals that we have.

Let me just explain what this part of the meeting is. Damon, because of the IDEA Lab, is actually one of the customers for this workgroup. So what happens is that Damon and Jim Scanlon will tell us a bit about the things that are going on in HHS. They will also make requests of us as to ways in which we can be helpful to them.

So do we have them? Either of them? Damon, why don’t you go ahead, because we can hear you on the audio.

MR. DAMON: Jim was planning to go first.

Agenda Item: Requests to WG, Request for Comment on WG Priority Setting, WG Web Review Approaches

MR. SCANLON: This is Jim Scanlon. I am here in the Office of Planning and Evaluation for HHS, and we are the Secretary of Strategic Planning Evaluation data policy and sort of internal think-tank for the Secretary and HHS leadership, and we try to coordinate data, statistical policy, data collection policy, across HHS, and we have an internal data council which has representatives from all of the HHS agencies that produce the major programs that produce data and statistics.

So just in terms of the origin of the workgroup, I think it was two years ago the leadership of HHS, including Damon’s office and the Chief Technology Officer and others, as part of our open data initiatives asked if we could create a working group of the NCVHS that could provide advice to HHS on really that represented the technology development side of the data ecosystem.

So we had a number of folks that were public health researchers and clinical researchers and statisticians and epidemiologists and so on, but we, I think it was realized, correctly so, that we really didn’t have in HHS at that time a way to obtain systematic advice from the developer community, and so as a result of working with Damon’s office and the Chief Technology Officer and NCVHS, we found a way to create this working group that is — and you will see the broad range of expertise represented on the workgroup.

Again, we tried to use the workgroup to react or give us advice on how HHS is doing or how we could improve matters, relating largely to our web presence, our data and digital presence, with respect to how we release data, the availability of our data, how easy it is to use on the web and so on, and Damon will tell you later, we have created back as part of the open data plan, we created a part of data.gov website where we have tried to put the HHS datasets and data resources and links on data.gov. We also have healthdata.gov, where we try to put together that same data for the health area.

But actually our agencies disseminate — and we try to identify datasets and resources and to make them available through those venues, and again, that’s intended somewhat for the developer community as well. The idea is that can we, with the data that we have and can make available, can we besides the research community and the public health community, can we enlist the aid as a multiplier of the technology community, the developer community, to take that data and either promote applications of themselves or advise us on how we could make that happen through any means.

I think one of the early sets of recommendations from this workgroup was how HHS could better tag information that would comport with search engines so that when people were looking for data, if you weren’t already an aficionado and knew exactly what you were looking for, that this would be the way the developer community would look at it and what they would be looking for, and how they would look at different datasets and how they relate, and I think that original set was very helpful.

Then we asked the committee to give us some feedback on some other ideas, too. Then we began asking the workgroup to — I certainly was doing this. Ultimately this comes down to individual websites and pages. How we were doing with some of the data portals for data or with some of the websites for some of the —

In this case we were looking at basically the public health survey and statistical data, and I think the workgroup took a look at a couple. The National Health Interview Survey website and I think if I’m remembering, Vickie, we asked them to look, and maybe that is still going on, at the SAMHSA data portal as well.

What we are really looking for is are there ways that the developer community looks at how this information is made available and gives us ways that we can sort of maximize and multiply the availability of the data to the broader health system and human services system, and so we try to always make our data available in machine-readable form. I will tell you a little bit about the spectrum of how we make data available, but it’s clear that we’re not — again, in many ways you have to be an aficionado. It’s getting better at being able to search and find what’s available, but in many cases you almost have to be an aficionado and know exactly what these surveys are and what they contain, and we are trying to make it a little easier to do that.

Usability, at least web availability, usability is another area that we have asked for advice. So that’s why we created the workgroup. We have some good advice and ideas from the workgroup already, and I think from my point of view, I think Damon will have some other requests. I would really like to have the committee, the working group, continue to look at some of our data portals, web data portals, and how our agencies release data, public use files particularly, in terms of any ideas they could give us about how to make this more organized or more prominent or more usable to the — particularly from the technology developer community.

So I will probably be asking the workgroup which of those websites I would like, the HHS would like them to look at next, and I think this is how just quickly I’ll summarize. So for all of our agencies that release data, generally they don’t release it in identifiable form, because they are either governed by the Privacy Act or by HIPAA or by some other confidentiality statutes or recommendations.

So we can make data available, number one, through actual reports on the web, and cases where we do that, we try to — if the report contains like the enrollment data for the Affordable Care Act, the monthly and quarterly, we try to accompany the report with a link for the data, a machine-readable file that contains the data as a link or as an attachment so that other people could look at what’s in those tables and do their own analysis and presentation.

So but that’s basically making it available publicly in an analytical form. The second area is kind of like the most broadly available public use. This is where the agency takes the data it has. We have this from many of our agencies, everywhere from CMS to the National Center for Health Statistics and NIH and AHRQ and others. Here’s an example would be the National Health Interview Survey, where the agency puts together public use files. They are de-identified. The data is somewhat edited to protect against the ability to re-identify or identify anyone in these surveys. So that means that some of the geographic detail is not available there publicly.

But basically the link is available online. There is a fairly simple agreement that you find the data where it is and you pick which of the files you like, and then you can download it. You sign electronically an agreement that you won’t try to re-identify anyone. So that’s another way, which is relatively unrestricted.

And the third way is through a research data center at, for example, we have this at NCHS, at AHRQ, and a couple of other places have similar kinds of arrangements, and there it’s more restricted access. Normally you have to go to the agency research data center website. You have to fill out a request for research data. That goes through a review process. It doesn’t have to take long, and sort of a feasibility process, and then the agency will — this is done electronically or through email at any rate, the agency will, if it makes sense and it’s feasible and there’s not any disclosure risk, the agency will do the analysis for you, maybe provide you statistical tables or regression or whatever other technique you were using and email it back to you. Create a workspace there and they’ll keep it for you for a while. Then you could publish or analyze or whatever you would normally want to do.

And then the most restrictive way is where you literally have to come — you are interested in very detailed information, possibly even identifiable information, and so there you have to engage in a data use agreement with one of our agencies, and there you have to probably indicate what data you need, what is it for, how you will protect the data, how you will secure the data, and it’s fairly serious. It’s not something that an individual — normally it’s an institutional kind of a responsibility as well, and when you are finished with the data it would be destroyed and so on.

So there are a number of those written agreements that Medicare data, which has been very helpful for decades and to some extent the Medicaid data is in that category now, and now they have a virtual data research center way of getting that. But in essence, you are getting access to micro-record data and linked data and in some cases it’s even identifiable data you have to agree not to disclose obviously, because that’s covered by HIPAA.

So that’s kind of the range of — and then there’s another that’s really data that’s narrative or it’s directory type information or it’s qualitative information. You know, it might be advice to consumers. It might be health advice, all those things, immunization recommendations. A lot of things that we make available that are — it’s basically information. It’s public health information or human services information. That can be available publicly. Most of it now is available on the websites.

So again, I think it’s fairly — for those well-steeped in the research or public health research areas, I think they sort of know how to do this. So some of these take a little while. But I think we are really trying to find a way to — if people can’t find the data we are trying to make available, we kind of — obviously they can’t use it, and if it’s so hard to use and understand, then again we have sort of failed in making it broadly available.

So again, we are not looking for statistical advice necessarily. We are looking for advice from the expertise that you folks around the table represent in terms of from the point of view of the web and applications and development, how would you when you’re looking for this kind of data or when you just want to see what’s available for potential applications or analysis, how would you look at it and how could we make it better here at HHS? So why don’t I stop there, Vickie, and answer questions?

DR. MAYS: Let me ask a couple of questions. In terms of the three or four types of formats that the data is available in, are you asking us to think about some additional formats? You’ve come up with about four different formats that the data is available in. Are you asking the workgroup to think about some additional formats for the data to be available? That sounded like one ask, and then the other ask would be for us to really look at these websites and figure out kind of if we were to do use cases kind of who can navigate them and if we can come up with improved ways to know that you have this data and secondly to kind of navigate.

MR. SCANLON: Exactly. For example, you might look at NHANES, which I think is fairly — you know, it’s not so gigantic that it would put people off, but it’s an example, but let me think about what would be the first. But there are probably 12, probably 12 major surveys that we have that really comprise most of the statistical capability at any rate, to monitor the health and the human services, and so again, if we are not, if it’s almost impossible or difficult to find and use the data that we think we are making available, then we really need advice about how to make — I wouldn’t, as a first step, Vickie, go into the four, the spectrum of how data is available, because some of that is still governed by law and regulations. We are not going to be able to do much for a while anyway.

But I think if we can ask the work group to focus on one or two maybe of these examples, and they can give us some advice; one would be a public use file from, you know, maybe NHANES. Then the others are kind of — it’s a little trickier. It’s really data.gov or healthdata.gov.

But I don’t think we can change most of those modes of — well, think of that, the data release, the public data release spectrum, and we sort of mixed and matched. We had a workshop about a year ago that looked at the potential risk of disclosure or redisclosure in these various meetings, and we found one or two in between modes but those are pretty much, it looks like what — so it would be within — I think we take that as a given for now, but particularly the public use files, which is what people really like to get, you know, that way you don’t have to get anybody’s permission. You just agree not to try to re-identify somebody, and then you have it, and you have to protect it obviously, but you can really do the analysis you want.

So that would be my preference to start, and I’ll probably give the group maybe — well, NHANES has — the thing is each one of these surveys probably has a couple dozen, you know, specific files, but I think the workgroup could probably tell us, you know, how would you look at it and how could it be made a little more accessible.

You might recommend that we put together a central guide to the public use data sets. I mean, maybe when you search they come up. I don’t even know if they do. If you type in the National Health Interview Survey or the MEPS, it will probably take you to that page and then you would have to find out where are the public use files there or the reports or the link to the research data centers.

And I don’t want to have the committee, the workgroup, looking at too many. I think it would be good to try how does this work with some of the major most popular datasets or potentially the most useful. Again, how do you — was it easy to find them? Is it easy to find related data, public use data files? What would be — is there a better way to label? Is there some way to pull together — the problem is these things get generated all the time.

It’s a little hard to keep an up-to-date — we know what those major surveys are. We keep a directory of those, but the datasets and the reports are — you know, they happen almost every week, updates and so on, and so it’s a little harder to tie down a catalog or something like that.

Does that make sense? I think that would be very helpful if — I mean, the research and others, they want to make it easier, and the easiest thing for them is public use files, but I think for everyone else who doesn’t necessarily know these datasets, we really need to find a way to make them a little more transparent.

DR. MAYS: Let me take some questions from the workgroup and see if anybody has questions and understands kind of what we have been asked to do. I think that we can come up with a couple of surveys we want to do. We are going to talk a little bit later about a rubric which may give us some information on kind of the design and usability and then we may need among ourselves to also talk about some additional pieces in terms of how people access those things and how we can push the data out.

Are there questions for Jim while we have him on the phone, because he actually is going to go to the data council.

(Discussion of video issues.)

MR. DAVIS: This is Damon. I was told in the chat to log out of the session and log back in, and you’ll get your full functionality basically by being kicked out.

DR. ROSENTHAL: Hey, Jim, just a quick question to follow up on Vickie’s question. Are you limited to these four means of distribution, or would you potentially like to know more about other means of distribution, such as putting data in a data explorer, Google public data, Tableau Public, et cetera? The idea being instead of just posting data out for users to find it on your website, putting it out into the ether, letting 1,000 flowers bloom, et cetera. Last year we talked a little bit about that. I showed you some examples of how your data is actually already in those locations and some things we could do to expedite that and then walk through an example, which was done by ReadWrite web, where they took a number of your different data sources and had an open contest where they had half a million people doing this and walking through it. So the idea of a public data explorer, are you guys open to that?

MR. SCANLON: They seem very interested in that. I think Damon and I, I’m going to see if you’re okay with that. I don’t think our agencies know that their data is out there.

MR. DAVIS: That’s fine. I think that is a great approach to making sure that the data are more easily discovered and generally better disseminated, and I’ll get into a little bit of that when —

MR. SCANLON: As I remember from the earlier discussion, you could look at how many visits there are to the NCVHS websites or the NCHS websites, but when you look at Google or the other, this portion of Google or other sites like it, it’s an order of magnitude higher. But I think we would like to learn about that, and I think if we could learn a little more better and if there are examples that seem to work well — the only issue would be the confidentiality protection, but from what I remember from the earlier case study, that was not an issue. But that would be one of the things we would like. I would very much like, and I think Damon as well would be very interested in — so I just don’t think our statistical agencies and others necessarily know about that.

MR. DAVIS: There is a great example of that actually out of NASA I believe. They index their metadata on their datasets, and it just added an exponential discovery of their data. It’s something I’ve been meaning to follow up with. I was given a contact from NIH to NASA, and I would very much like to follow up on that to implement something along those lines.

DR. ROSENTHAL: Briefly let me just ever so briefly just sketch out two different trains. One is how do you get indexing and Google to work, indexing metadata is great. And then this other thing is a data explorer, which basically you put your data out. If you go to just browse. Pull up Google data explorer right now, and you will see CDC data; you’ll see a variety of different datasets, and it’s not the datasets you are looking at. It looks — it’s in a visualization environment.

So your data is already linked and mashed with a variety of other different datasets throughout the world, and all the metadata is done for you. All the taxonomy is done for you; all the tags are done for you; there is no standards review of that. It’s actually done — it’s an interesting model to look at.

DR. RIPPEN: In addition to that, there also needs to be the ability to be able to download the data, because again, if people are using it to help with communities or I would say neighborhoods is the correct term, then being able to maybe have people who need to actually do healthcare delivery be able to match it with other data. They would still need to be able to download it. So I think that there are two major themes. One is open source with the app, and the other is the data.

MR. DAVIS: Again, guidance like that in terms of principles would really be helpful to us.

DR. MAYS: If you look at the screen, I think Lily has brought up some examples for you. Jim, here’s a question. Do you think that this — if we have time at the February meeting that we might see if we can actually do a presentation about this, if we can get Josh in a little bit earlier? We will see. To do a presentation before the full committee?

MR. SCANLON: If we can do it with some HHS datasets particularly. I’m assuming that Census Bureau has — they might be there as well, some Census Bureau data. I think so, at least what the concept is. This is a whole other, I think as Josh says, this is a whole other way of getting data, liberating data, and reaching other bodies of potential end users.

DR. ROSENTHAL: Just one quick note, a follow-up on that. The idea is that if it’s in a visualization environment, the woman who won, the young woman who won the comorbidities for diabetes contest using a variety of HHS datasets, you don’t need any coding, you don’t need any data capability. It’s literally doing analysis in a visualization environment. So it opens up HHS data to whole worlds full of people who don’t have any coding or technology skills.

PARTICIPANT: It’s a whole new dimension of people using the data, yes.

DR. MAYS: Okay, I think we will look at this as one of the things we want to pursue.

DR. RIPPEN: Again, I just want to emphasize that there’s three or four different uses of the data and liberating quote, unquote, the data, and I think we just have to make sure that all are met, not just necessarily one.

DR. MAYS: Okay, sounds great. Damon, are you still with us?

MR. DAVIS: Yes, I am.

DR. MAYS: Thank you, Jim, because I also know that you are going to data council. So I think that you gave us a very exciting task that is quite doable for us in terms of bringing it to the committee. So I really enjoy that task for the group, because I think it will let us explore some different things. Anything else that you want to share or ask us before Damon does his presentation?

MR. SCANLON: No, I think again we appreciate everybody willing to serve and we are trying to make it a virtual meeting. So we don’t have to necessarily force everybody to travel. So any way you think we can improve matters, let Vickie know and we will try to do it.

DR. MAYS: Great. Thanks, Jim.

DR. FRANCIS: One improvement would be if people identify themselves before they spoke.

DR. MAYS: Oh, okay. I think that’s good. Go ahead, Damon. What’s your ask of the workgroup and any updates that you have for us?

Agenda Item: Requests from HHS IDEA Lab and Follow up on WG Suggestions

MR. DAVIS: Sure, so I will start with the update first and then move into ask, I think. The first thing for those who are unfamiliar, we recently updated healthdata.gov, our public open data platform that was established, as Jim alluded to, to catalogue all the datasets that we have and really try to provide solid information for the public as to what’s available, a little bit of documentation about what the sort of the voice of that data, what its opportunities and limitations are.

So healthdata.gov has gone under a revitalization. We recently relaunched it in the summer. We are moving it out of beta form right now, and some of the things we are trying to accomplish are making sure that the data are more easily sortable and searchable. We are hoping to implement some functionality that will allow sort of that, you know, the amazon.com experience of data exploration where people who viewed this data were also interested in this data, and that’s a suggestion that came out of this workgroup. I remember seeing slides very early on from Lily and supporting the comments from Josh along those lines.

We are trying to implement some of the things that were established as recommendations out of the workgroup for better discoverability and documentation of the metadata on healthdata.gov and just to keep in mind, I think most people probably know this, but the data that appear on healthdata.gov are federated to data.gov. So in the health domain on data.gov, you will see a large variety of HHS datasets, as well as many others from various entities.

We have USDA data. We will soon have some data that was ported over from Department of Education with regard to some elderly populations that they have looked at, and that data was moved over to our Administration for Community Living. So you are seeing wider and wider blocks of data catalogued. We are continuing our search across the department to find the data that may be valuable to people.

But along the lines, finding the dataset may be of value to people and only part of what we need to be doing and recently or within this year, we implemented a pilot program called demand-driven open data, and I can’t remember, Vickie, if David Portnoy had been brought into the workgroup to sort of discuss what DDOD was and the opportunity to present it.

DR. MAYS: No. Oh, wait, Damon, he did come, very early on, but it was just to introduce himself as your fellow, as the fellow. So he didn’t present to us. He just attended.

MR. DAVIS: I do remember that. So I think one thing we should probably do is bring David back and ask him to do a bit of a presentation as to what transpires in demand-driven open data pilot which we — the government; we love acronyms — affectionately refer to as DDOD. But demand-driven open data has been a pilot where we are trying to move away from a push model of delivering data to the public to being more conversant with the public about what their data needs are, where some of the challenges are with our data, and how we can make those data better. So demand-driven open data takes an approach of allowing folks to engage with the department through a wiki knowledge base and a GitHub platform that allows people to start to enter what we are calling use cases for the data, and those use cases fall along the lines of, quite literally, I don’t see the data I’m looking for on healthdata.gov to the more granular sort of I see the data I am looking for and it’s really great except I need these following updates to be made for it to be valuable to me in my entrepreneurial research, media, whatever endeavor it is I’m engaged in.

So I think you would be pretty interested and excited to see what DDOD has to offer. We have been currently sort of taking a step back and trying to evaluate the process for demand-driven open data, because we recognize that it can be a great asset to the public in terms of speaking with the department about what they need. The challenge that we are then faced with is what value does this then provide back to the department so that we can make sure that this is mutually beneficial.

So I think if someone would make a note, we should definitely invite David Portnoy to come and make a presentation about DDOD, because I think it has a lot of opportunity to integrate with healthdata.gov in order to create both a platform of discovery as well as a feedback loop that is better supported by a large portion of the department in terms of discovering data and chatting about it, et cetera.

DR. MAYS: So what you want from us in terms of bringing him in is for him to present the project and for us to comment on it, or is there something else that you want us to do?

MR. DAVIS: Yes, I think it would be valuable for you guys, for us to have your expert opinion on what you see as the opportunity here. We get a lot of requests for data and quite literally it goes direct to the email address of the program owner.

So we get a lot of one-off requests, and the idea is to make sure that those requests are beneficial to a larger community of people. So there may be two, three, four, five folks that are asking for some of the same stuff, and if those requests are documented in the knowledge base, basically the operating division is going to be in a position to point people towards that resource as a means by which people can understand what the data opportunities and limitations are.

So DDOD is the pilot that we are working on. It would be great to get this workgroup to comment on that.

I wanted to go back to something that Jim said before we get started in terms of sort of examining our data portal and identifying where we have opportunities and challenges. I think we need to be really smart about how we deliver what we find, because the challenge from the department is obviously budgetary. We can make all the recommendations in the world, but if we are not actually allocated the dollars to making the improvement to the portal that we see, unfortunately it’s quite possible that those recommendations could fall on deaf ears. So we are eager here — or handcuffs may be a better way to describe it.

So I think I just want to make sure that as we are doing the examination of our data portals and our data resources, that we are conscious of probably delivering a scale of what is most important to work on versus some nice to have so that in the event of a budgetary opportunity, someone has a target list of things that they could aim for.

So I just wanted to pause there and see if that would be, how that resonated with the group in terms of just developing not just a problem list, but a strategically focused list that would be the must-have versus nice to have.

DR. MAYS: I think the question is in terms of — I understand what you are saying, and I think we had talked once before about this issue of giving people feedback about their websites and whether or not they really would have the resources and whether or not they would have just finished a cycle of doing it, and I think the issue probably is going to be for us that we are going to try to come up with some general things and give people their feedback and then develop some general guidance which is out there for them whenever they are on a cycle where they can make the changes; then they would probably refer to that guidance, but I hear from you that there is a sensitivity that we should have.

So I think that as we take websites to do feedback it will be that they have volunteered or that they want us to do it at this point so that we avoid some of, I think, what you are concerned about. That might take care of that.

MR. DAVIS: Excellent. I am sure Josh would second this; folks who are leading edge like CMS and CDC who have just an incredible vision for what it is that they want in terms of data products, I think it could be helpful in the recommendations to provide some guidance to some of our smaller operating divisions as to what they might implement from this, the package of success. So what are the best practices that have been adopted, and how can we disseminate that across the department. Those types of recommendations can be really valuable as well.

I am hearing a lot of really cool things about what CMS is about to do. We had a meeting with them yesterday, and one of our big challenges is disseminating those best practices not just across that agency but quite literally to some of the other agencies in the department, the other operating divisions. So to the extent that we can fold in some of the things that you are seeing as crosscutting, I think we have already alluded to that. That could be really cool as well.

Moving on, just a couple of sort of quick things that I think would be interesting for this group to discuss, and you’ll forgive me if you have already covered some of these things, like I know that privacy of data comes up quite a bit in various conversations, and I just wanted to underscore that HHS has really strong leadership in the Privacy Offices, both at OMB and OCR, our civil rights agency, and Devin McGraw who has been (audio drop) in the department is now our, is the chief in OCR, and Lucia Savage as privacy officer at OMB, to the extent that there are looming items that this group has had questions about or had suggestions for the department about, it might be good to invite a conversation with Devin and Lucia at the same time, just to sort of expound upon what the department’s privacy policies are and how we are doing enforcement, what kinds of suggestions and recommendations we are making to the public, et cetera.

DR. MAYS: Damon, we have just had Lucia Savage here earlier, right before. We just had Lucia Savage and Rachel Seeger here. So they were sharing with us some of the excitement and the ways in which they are actually working with the developer community. Part of what we are hoping to get from them is kind of a case of how they were able to engage the developers to develop these apps that are helping with some of the work that they are interested in, because I think that would be something we would want to share broader as we do these guidances.

MR. DAVIS: Excellent. I thought I saw Lucia’s name on something previously, but I couldn’t remember and I didn’t have a chance to go back and investigate. So I am glad to hear that.

Have you already extended an invitation to Devin McGraw?

DR. MAYS: No, I think what happened is that the chair of our privacy committee actually met with Devin McGraw and so we were able to get some input from the privacy committee for Devin. So we are doing good on that.

MR. DAVIS: Another area is the sort of you have a lot of apps at HHS that are focused on consumer engagement, you know, trying to disseminate information about HHS, be it public health information. We have more and more apps and mobile oriented software that we are trying to disseminate, and to the extent possible it would be really interesting to see how this group views the communications and coordination and dissemination of those apps, as many of them are in fact using the data that we are providing as an agency, turning it into a mobile sort of consumer facing format. I didn’t know if that was something that this group is at all sort of interested in or engaged in in terms of the overarching sort of broader dissemination of the data.

DR. MAYS: Wendy Nelson is not here today, but I’m sure if you say the word mHealth within the spectrum of NIH and now NSF, Wendy’s name comes up. Wendy is going to be staff with us starting with the next meeting, and I think that that is a space where Wendy probably can respond on our behalf, because she does most of the training in this area and knows quite a bit. So I think we might get from her a presentation on where we are in terms of app development for consumers and see what she has to say. So I think that is a good suggestion. So we will put that on the agenda as kind of far down for us to also look at.

I would like to combine that with kind of thinking in terms of the agencies and whether or not there is a way to think about how to do this with particular agencies that want to get their information out.

MR. DAVIS: So the final thing is in the development of this framework, it’s a really interesting and very comprehensive document. I can’t help but think that this would be something that the health data leads who are my liaisons across the department would find valuable in discussing because they will very much be at the forefront of implementing anything remotely resembling what’s in that document. So it would probably be valuable maybe to have you, Vickie, or someone else attend a health data leads meeting, which happens quarterly. I just had one on November 4. But to talk through some of the things that workers may work on, and specifically this framework. I think that would probably be a good use of the leads time as well as a good opportunity for you to present this and bounce it off some of the folks across HHS.

DR. MAYS: So let’s follow up with that. I probably will see if Bill can actually do a virtual presentation to the leads group and kind of get some feedback back and forth. Would that be reasonable?

MR. DAVIS: That would be reasonable.

DR. MAYS: Okay. We will look at that for — it’s December 4? I’ll talk with him about it.

MR. DAVIS: We just had one on November 4.

DR. MAYS: Oh, November 4. Oh, okay. When is your next one?

MR. DAVIS: I think it’s going to be early February.

DR. MAYS: Oh, okay. We will see if it is going to happen around the time we are actually here for that NCVHS meeting, and then you might be able to get one of us in person. Sounds good. Anything else? This is great, Damon. I think that we have some good asks from you. Anything else?

MR. DAVIS: No, I think that is about it. I had a conversation with Greg yesterday about the Beaumont Foundation and public health engagement with our open data and perhaps inviting somebody from Beaumont to come and interact with the group with regard to public health apps. Not sure how you all feel about that, or is that a topic area that’s been discussed already, but I thought I would at least put it here on the table.

DR. MAYS: I was at the public health meeting, so I actually met with the people at the Beaumont Foundation about something else, but at our community metrics meeting on Tuesday, there was a representative here from the Beaumont Foundation. So there is an intention to keep working with them. So I think as we move along, if there are some ideas that may be coming from the community metrics side here or you may have some ideas in terms of things that you are asking us to do, whether going to the Beaumont Foundation to get them done might be a useful kind of project. So let’s keep those in mind and see kind of where we end up with next steps.

MR. DAVIS: Okay, sounds very good.

DR. MAYS: Can you say on? I want to make sure that — we are going to also talk about the blogs and we are going to talk about at HHS as well as kind of some individual blogs, but just before I do that, I want to make sure that Leslie, who we can also see on the webcam, can introduce herself.

DR. FRANCIS: So I am a former member of the community and former cochair of the Privacy, Confidentiality and Security subcommittee of NCVHS. I am a law professor and a philosophy professor at the University of Utah, and my specific interests are — I think the fairest way to put it would be making sure that we have the kind of —

DR. MAYS: Leslie, you may want to face your camera so that you are facing us. We see you looking into it, but it’s up to you.

DR. FRANCIS: I’m facing my cellphone. Anyway, I’m interested in making sure that we can really use data consistently with appropriate privacy protection, and I’m a privacy expert.

DR. MAYS: Great. We have anyone else on? Anyone else on? Dr. Vaughn, I hear you are on the line? Would you introduce yourself?

(No response.)

You might be on mute, because we haven’t heard anything. Okay, we are going to move on. These are people who are technologically skilled.

Agenda Item: Blog for users/HHS approved blog

Okay. Here is what we want to do in this next part. Thank you, Leslie. Here is what we want to do in this next part, which is we want to get information out. You heard all the things that Jim is talking about, all the things that Damon is talking about. We keep coming up with a ton of ideas, but they kind of sit, and so we talked about this issue of potentially doing a blog, a blog in which it’s individuals giving their individual opinions. Walter also raised about the fact that HHS can host blogs. So Lily, I’m going to actually need you to come and be here, because you are going to talk a little bit about the background information.

So there are two ideas here, and they, as far as I can tell, should both be considered. An individual blog, Josh has put something together as Josh’s blog, and we can begin to add things to it and build on it and every time that we come up with something that we think is useful in terms of HH data, in terms of access and use, we can put it there. What it means is that for people who want to use this data, they start to find this go-to place, not a big formal long drawn out thing, but something as simple as I found that if you do X, it allows you to get to the data, or you may have had difficulty finding the data dictionary, and here is, you know, it’s kind of hidden from first seeing it, but here is where you find it. So it’s things like that.

The HHS blog, as far as we can tell, is usually longer treatises on things. So it’s kind of a long article in which a person talks about some issues as opposed to like little fixes.

I’m going to let Lily talk about it, because she did some background on the different blogs, and then I’m going to turn it over to Josh.

MS. BRADLEY: Part of the research I looked into was just about what processes there are available for starting blogs, and there really no explicit processes in place if you want to launch an HHS blog. HHS operating divisions and staff divisions, which is how we are sort of organized, they have their own internal rules about it. So they have processes for like clearances. Federal advisory committees do not have guidance about how to go about setting up a blog or what rules would govern that.

So there’s kind of no set rules of the road, nothing precluding you from kind of going ahead or not, and there are these other blogs that you could potentially create some sort of partnership or something to provide it on. So I have actually pulled up a few of these websites, and I’m just going to actually broadcast them.

DR. MAYS: So while she does that, Josh, can you talk a little bit about as we seem to have talked forever, but you could talk a little bit, because it’s going to go, so talk a little bit about what you have showed us last time?

DR. ROSENTHAL: So I put together a little bit of a personal blog, which is open to anyone, and I should probably migrate it to the right spelling of the NCVHS, instead of NCHVS, which was kind of done tongue-in-cheek. But the point of the story is that it’s just a place where I am keeping notes where we have these various ideas. If you are asking for a bit of information, instead of emailing it to someone onesie twosies, we can have that all out there, and the beautiful part about that is that I get asked probably 30 or 40 times a day at some point where is this, how did I find this, are you running into the same issue, and the point of the story is that we can keep notes up there, examples of our work, and answers to commonly asked questions. It’s free. It takes five minutes, and it’s available to anyone interested. Is that what you are after?

DR. MAYS: Yes, exactly. Can you lift — I don’t know, because this gets very complicated because they are now on Adobe Connect as to whether or not you could share it with us?

MS. BRADLEY: Okay, so now we are all looking at what I’m looking at. So I was just going to show one of the blogs that had come up was ONC’s blog. Their federal advisory committee does not have a blog, but you can kind of see — actually a lot of these are more like announcements.

DR. MAYS: This is what Walter was referring to in the sense of different people writing things, and that was — I think it’s, quote, approved by ONC, right?

MS. BRADLEY: Right, so the way the system kind of works at HHS, depending on which group you are with, there are certain clearance processes. I’m not super familiar with ONC’s, but there is a process through which they would work with public affairs to have things approved, and there are lots of signoffs. CDC has a very long one.

DR. MAYS: So I think what we want to do is to say what’s the purpose of this blog? I think the purpose of the kind of details that they are putting up is a discussion of an issue. The other that I think we are talking about with Josh is here are fixes, here are finds, and here is problem solving. It really isn’t about details on an issue.

And I am wondering if there is a way —

DR. ROSENTHAL: Go to NCHVSDG.wordpress.com. So this was just something I threw up in an afternoon while I was messing around, and so if you click on that, that’s a mobile menu up there. So it works nicely on a mobile phone, and I also did this sort of to illustrate a little bit about what Jim and Damon were talking about, tags; one of the ways you organize content, whether it’s data sources or posts, is by tags that are done top down through taxonomy and tags that are done bottom up by user through folksonomy, and so if you go there, you’ll see categories, which are kind of top down tags, and tags which are bottom up, and so hopefully this gives us a little bit of a sense of how you can use different means of navigating through things.

So if you go to healthdata.gov, for instance, you’ll see tags, but the only tags — so there are tags. There are a couple of pages, and then there are categories. So you can think of each little post as a dataset, if you will, and they can operate vertically and horizontally. This is typically what you find outside of healthcare when you are looking through data sources.

DR. MAYS: Damon, can you see this?

MR. DAVIS: Yes, I can.

DR. MAYS: Okay, just wanted to make sure. Let’s settle on one, and we will use it as an illustration.

DR. ROSENTHAL: Just click on the home header for a second, and just click on the menu, too, the menu off at the right-hand side and just leave it there.

So essentially this is a blog which means you put a little bit of text with some links or resources. I use it for meeting notes as well as various issues, and the most recent stuff is at the top, and then you can create — so it’s rolling, it’s a weblog, right? Then for each one of those little blurbs, you can either assign it a category, which is typically what people think about when you think about organizing data, or you can assign it a tag, and tags in this case are often done with users, and I have it set up so that a user can assign any tag. So as individual users assigned tags, those get bigger in the tag cloud. So you can see specific things.

So if you were to click on one of those tags, it would pull up all the associated bits of content, which are often links to different resources.

DR. MAYS: So click on a tag.

DR. ROSENTHAL: If you click on examples, it will pull up all the posts about examples. If you click on slides, it will pull up all the posts with slides, et cetera, and if you click on one of these read more, there’s the full post there.

DR. MAYS: Try your hand with government data, let’s click on that.

DR. ROSENTHAL: So if you click on that, that will take you to a link, which shows you a pdf. So one of the things we are always talking about or one of the suggestions that gets made is publishing an ERD, an entity relationship diagram. That’s what tells a developer or someone who is familiar with data what went into this thing how to interpret it, how to even think about it. So if you click on this, on the pdf, you will see an example using a public use file. Click on the forward button on the slide.

Actually just let me show you one little thing. So here’s links, click on one more. Sorry, one more.

So here is a file as I find it, and I have to have some questions before I can use it. So this is illustrating taxonomy. How do I solve for this? So click on the next button. What are these things? What do they mean, et cetera? So there are ideas of entities versus attributes. This is basic taxonomy 101.

Someone at HHS has made this master diagram. They have chosen not to share it with the public, which means it takes me a hundred times as long to be able to figure it out. So I walk through these are entities. These are relationships. There is a plan. There is a contract. There is an org. There is a parent org. That is a hierarchy that HHS uses but chooses not to share with the public for some reason.

Click on the next slide forward. Those entities have attributes. What is a contract? So I basically walk through what is taxonomy, what is this thing I’m talking about all the time that all the developers are asking for. That has no privacy implications whatsoever. It’s just the schema that you use to make sense of it instead of asking people to figure it out every single time.

DR. MAYS: Let me get some comments and questions. Let’s start with you.

DR. RIPPEN: You know I was just going to ask how did you organize it, because as you know, blogs can become pretty vicious and hard to find, so it’s interesting that you are doing both the folksonomy but also doing the regular hierarchical kind of here’s — well, you are doing tagging. You are not really hierarchical, right?

DR. ROSENTHAL: There is some hierarchy in there, too. This is sort of an example of what a blog can be and how you can do it, and if I were to do it kind of the right way, I would look through every data set and I would have a post corresponding with it, but it’s also sort of a heuristic or a little teaching model to say here’s how we can think about tags and here’s how we can think about hierarchical categories.

If you go to healthdata.gov, you do see tags by type. You see it by JSON and xml and et cetera.

DR. RIPPEN: Do you allow people for comments, because again, there is the — you were asking for feedback. So how do you get exchange? I know there was do you like it or not. I just didn’t know how much of a —

DR. ROSENTHAL: This is just a little thing I put together. So it is open to comments, but you have to register. So all of the stuff is configurable that you want to build a blog. You can have anyone leave a comment or anyone leave a tag without permission, without registering. You can open up to a few people. I think the default I set it on is anyone can do it, but you have to identify yourself and have an email, but all of that stuff is configurable.

PARTICIPANT: What’s it based on?

DR. ROSENTHAL: WordPress, which is what the NCVHS site is actually based on.

DR. MAYS: So we have not publicly released it, but you put it together to give us an idea that each meeting we come up with stuff, but it’s just not going out fast enough, because it is kind of waiting for the guidance to come back through the full committee.

DR. RIPPEN: Well, the dilemma with any kind of information is what is the value and how do you get the value and how do you get it in the most effective way. So I think there is one thing with regards to keeping information so people know from a historical context what happened and some of the themes that were discussed, and also when we get as young as we are, you know, did we already cover the topic? For those of us that are new, what has been covered already? So I think that is an important thing.

But I do also think we have to think about what is of use and where is the gap and what do people really need, and that is also true from some of the earlier things with regards to the data. There are different types of uses for data that have different values for different sectors, and then on top of it, the whole concept of granularity or I should say as geographically defined as possible to enable actual use at a neighborhood level.

The other nuance then is, well, how do you deal with some of the privacy components, and I’m just saying that in the Internal Revenue Service, I believe, or it could have been Census, but I do think it’s IRS, I actually did come up with algorithms, because you can actually statistically mask data in a way that is still statistically sound but can’t be recreated. So you kind of make it a little blurry.

So again, as we — and I’m assuming that this may be the case already, but again, as we start asking for people to share information, that we actually don’t have to reinvent the wheel and everyone leverages the best statistical methods to do that. So we can get more use and that it’s as granular as possible.

DR. ROSENTHAL: If you go to Google.com and search for this in this workgroup, you’ll find a bunch of slides on that, including standards and comp science that apply to this specifically, some quotes from folks, from friends that I have in comp science talking about how to do that, which would be great to solving all the stuff that we talked about consistently. It’s been done before in other verticals. So we have some of that; if Jim or Damon are interested in that, we can pull that back out.

DR. RIPPEN: Because that way we can get to the data and we don’t have to go through — and if everyone does it, we are like free the data, because otherwise we can’t ever get it, or we get it at such a high level it doesn’t have the applicability.

DR. MAYS: What I would like to do is go around the table and get a sense of your thoughts about this. Suzie, let’s start on your side then, since you had your card up.

MS. BEBEE: I had more questions, I think, than comments. Is this supposed to be an informational blog? Is it supposed to be internal?

DR. MAYS: The blog as we were thinking about it would be something where it’s for external people. It allows them when they have questions about HHS datasets that they might look and see if somebody has solved that problem, and it really is a kind of — it is going to be probably most useful at the higher level. It is probably going to be data entrepreneurs that we want to drive there so that they will think about developing things and kind of being in that, okay, I can use this; I could probably design an app from this.

So it is probably that group or the researchers. It is going to be people who use already, but it’s that sometimes they don’t know about some data. They work in doing health insurance. So they know these. But they didn’t realize there were other datasets. So it really is to drive, I think, the kind of sophisticated user to be able to say if I wanted to use federal data and I have some questions about the actual using it, whether or not they would come to this blog and see if it answers it, and if not, they might learn something about other datasets. So that is the way we are thinking about it.

MS. BEBEE: So I was thinking as you were talking about a place to come to see if problems have been solved about using the data, but then also about what Jim Scanlon just talked about, and how we are doing in the government with access to the data. So it makes me wonder is this blog going to identify the data sources and then blog about each of those data sources? So if there is a public use file and someone wants to use it, that that could be the header, the topic?

So I’m thinking a little bit higher about — so there are problems that might be addressed and solved like we saw in a presentation from OCR where they talked about the different questions, the popular questions, so this would be the popular problems. But higher than that, it’s those problems are based on what data? So addressing what coverage of the data that we are talking about, which datasets, which — and then what are the common problems that we could solve and address on the blog?

DR. MAYS: I think we have to think about that, because I think — and I see exactly what you are saying. It would be nice if when we were coming up with I solved this problem, if we could drive you to somewhere and it says here’s the dataset, here is the information about it. It may be that we don’t always have to blog about it. It may be that we can drive somebody to it, because I’m going to hope that what you have is places in which, say, it’s about NHANES, that we can drive you to someplace and then you can learn; oh, I didn’t know about that dataset.

And then you can go and learn so that we are not doing kind of the basic here’s NHANES, here’s what happens on NHANES, but instead we can find a lot, and if we don’t find a lot, maybe the suggestion is to say to NHANES can you do a YouTube, can you write a blog, and we will list it, drive somebody to it, but that we may not want to be the here’s the dataset and here’s what it’s about and here’s how many variables there are.

MS. BEBEE: I agree, and so a particular website that I oversee at HHS has a component where if you want to go to that — I’m looking at a measure. I’m looking at one measure, and if the user wants to explore that data further, it then sends them out to where the source of that data is.

DR. MAYS: See, and that is why we would want to then go back to you, because you have already created that. So we want to be the come-to for how to solve a problem, so that we can give you great hope and get you to use and think about developing something, increasing those kinds of things. But in terms of just the pure informational about the dataset, I would prefer if we could find ways to link people to that.

So the first time we mention NHANES, we should have things that send it off to here’s where it is or even to you or something, to the website that you have. So that was the way I was thinking about it.

Okay, Rich? Anything you want to comment about?

MR. LANDEN: Again, wearing my newbie hat. Not a data professional. The conversation I have heard is really talking about three different types of websites. We get the one we have just been seeing an example of is more for the professional, the data professional or maybe not professional, but somebody that knows their way around. The other two I have heard about are how to help the general public, ones with no training skills, education, and just how do we help them get what they need even when they don’t know what they need, and the third type is more just kind of a narrative summary of the highlights of what the workgroup is doing, and when I think of that I think in terms of — I don’t know how many have seen John Halamka’s blog, Geek Doctor at BlogSpot? John, after every ONC FACA meeting — My Life as a CIO.

One of the things in his blog routinely is he will do a six or eight paragraph summary of the FACA meeting. Very down to earth, just hit the pithy stuff, very easy to read. It’s both an information rich and an easy and fun to read blog at the same time. So those are the three things that I have been thinking of as I’ve been hearing conversation about different blogs.

DR. MAYS: Let me just say, one of the things that we talked about much earlier in this group was use case. So we were trying to think of, when we deal with people with datasets and looking at, you know, kind of what they are producing that we would think about, is this at the level of a data entrepreneur person? Is this at the level of a researcher? Is this at the level of the community? Is this at the level of the consumer? And to kind of give them feedback as to who what they have set up seems to work for kind of thing. So I think that I appreciate what you are saying about the different groups we are talking about.

Nicholas?

MR. COUSSOULE: I guess I have a few different thoughts. I think the discussions today have been both about almost like a core R&D, right? Here’s some potential kinds of answers. You know, let’s see if we can associate them with problems. Or we have a well-defined problem that we then say here are some potential answers to. So I think it’s actually very useful to do both of those things. We were talking about the blog that the gentleman next to me was showing; it wasn’t to be explicit to say this solves this problem. It was to be here’s a way to think about solving some of these problems. So I think that’s — to me, I look at both sides of that as being quite useful. Obviously if they intersect that is even better.

DR. FRANCIS: I’ve had my hand up for a while. Are you seeing the thing?

DR. MAYS: Yes, I’m going to run the table and then come to the screen. We can see you.

MR. CROWLEY: With regard to the blog, I think it fits nicely when we talk about sort of dissemination of the information, because one of the challenges that HHS data faces is a lot of people don’t know it’s there, don’t know what is happening with it, or don’t have a way to understand, but through a blog, you know, there are ways to take that link and put it into a lot of different communities and environments, whether that’s through LinkedIn groups that have tens of thousands of members that are interested in this information, through the different Twitter hashtags and groups which are interested in this information, through different conferences that are on this subject, which you could sort of push that into that information sphere of those folks.

And then also even from the meetings themselves, sort of live-blogging during a meeting, and you might sort of pick up the amount of people that are participating in this. I mean, I do think — we talked about this point earlier — it’s important to have the purpose and understand what the value, what are the targets you want to hit with the blog and what is the audience and the purpose, but I think we have — we have talked — I have been here for a bit with Josh.

So we have talked about these audience groups in the past, and I think we have a sense of where they sort of live in the digital world and across the marketplace. So having ways that we could push an information asset that could help — for example, I thought it was great this demand-driven open data initiative that Damon talked about today. That is very consistent with some of the principles we have talked about for a while, which is how do we have some type of learning health system.

For a learning health system, you really need to create mechanisms for people to get information out of a cycle, to get that information back across the community and reuse it. So I think the blog is consistent with these different aims, and also given the fact that it’s a relatively low resource effort. I mean, if we break apart the posting and have some general goals and —

PARTICIPANT: And making sure we constantly add to it. That’s the big thing.

MR. CROWLEY: Well, that’s the big thing, and that is one of the challenges. I mean, people are bombarded with content these days. So if we do a blog, I think we need to be very — think it — how can we make it really interesting? Josh has fun ways of doing things. How can we make it fun and target these? But it’s a way — why not, what is the downside of trying to see and pushing?

DR. ROSENTHAL: You could if you wanted to, I mean, if you have interns or I might do this with one of my interns is just over the summer basically say go through healthdata.gov, and I want a post on every single dataset, how to use it, what it means. You can, this doesn’t have to be either or. If you want meeting notes, you do it in a category. If you want problem solutions, you do it in a category. If you want whatever, you do it in a category.

Like, were I doing this in private, in the private world, what I would do is set up a blog linked to HHS.gov and basically have a post just have interns go through and like literally set up the indexing right and just use those pieces as content shells for your content, since HHS has never developed any shell of content around it, and just own that whole thing before somebody does that. It might be good if we start doing that.

MR. CROWLEY: And again, it’s a way to be more active about getting the information out and having it coming to us.

MR. GERSHMAN: I think a blog could be a great idea if — I think it’s very important to keep it updated and have it update regularly. If you feel like you want to get a message out and perhaps there isn’t enough volume to fill up its own blog, to represent its own blog, you could always use some other blog. Like maybe healthdata.gov has a blog or, you know, the HHS blog or something like that.

But it’s really just about putting out your message and getting that to the community that is important, and it is really marketing in a lot of ways for the datasets and, you know, the products that you are putting out there that you want people to use.

So that could all be very useful if it’s maintained. There are other avenues that you could also pursue. I was just looking actually at that data-driven stuff that Damon mentioned. It’s really interesting.

So doing something where you can have a community kind of form around something. It could be something like a GitHub repository where people could add comments, things like that. It could be a Facebook group or something like UserVoice or something like that. It doesn’t have to necessarily take the form of the blog.

The thing about a blog is that it is sort of like publish, you know, you’re publishing stuff out. So if you really want engagement and discussion and things like that, there might be some other medium that is better suited to that. So I think it is just identifying like what the goals are and then trying to find something that matches that. A blog could work, but there could be other options as well.

DR. RIPPEN: I would second that. I think it goes back to what is the purpose. One is to get a message out and crafting it. That’s good for a blog, because you control it. If you want the community to solve problems, because you only have so many FTEs in the world and you want people to kind of begin to coalesce about the topic and get a user group, then that is a little different, and then you are looking for other people to help other people out, and it could be, you know, it might be some other government agencies who manage that data that could help, too, but it may not be a requirement.

So again, I think it might be not just one; it could be multiple. I think the key is just understanding what the goal is, how you are going to measure that you have really done what it is that you said you wanted to do, and that you have the resources to make sure that you do it well, because otherwise it’s just yet another thing.

DR. MAYS: These are great. I think that we are going to the online, because I definitely want to hear what they have to say. So Leslie, then Damon.

DR. FRANCIS: So a couple of these comments are, I think, the time for making them is past, but I’ll start with the goal point, which is that we always have these interesting ideas at the meeting of the working group, but there’s no follow through, and one of the things that the blog could do is it could be a way of generating follow-through to a user group. So I’ll start with Josh’s example of taxonomies. We could say the goal between now and the next meeting of this group is to get good useful taxonomies of some of the most important datasets, whether we do that through FTE at the department or whether we do it through Josh’s interns or do it through crowdsourcing.

But it doesn’t make really good sense to have a neat idea bubble up and then dissipate even if it gets referred to with something like — I love Geek Doctor too. It’s tremendously informative. But it’s not — it just tells you what happens if ONC is, like what he thinks about it. It doesn’t tell you what systematically is being done.

So it seems to me that the really important thing for this, whether it’s a true blog or however we do it, for this group to do is to set some — this is the systematic thing that is going to be done so data can be more easily used. Otherwise we have had nice conversations, but we haven’t gone anywhere.

DR. MAYS: I agree; I think that part of what happened in the full committee is, I think, a better understanding of how we can move forward. So that’s one, and two, we are still in the process of pulling staff to this group, and I think that that looks better as well. So I agree.

DR. FRANCIS: Let me just follow up on that. A use for the blog, if we were going to think about it in terms of user generated content, could be a way of augmenting that kind of lack.

DR. MAYS: Okay. Damon?

MS. BRADLEY: Damon had to drop off at 2:30, will rejoin. He did have a couple of questions that he put up on the chat. I guess the first is a comment. He says I think the healthdata.gov blog would be a good resource for the group. We can cross post any of the content that gets posted.

The second is a question. What is the value of having an NCVHS workgroup blog versus having the members blog on some of the example sites?

DR. MAYS: I think the question that is being raised, which we can discuss, is if we are like driving this back to healthdata.gov, and I thought that the goal was to draw other people who aren’t at healthdata.gov and that what we are trying to do is draw other people in. We can then send them to healthdata.gov, but I think if our blogs are staying the same places where what they want is for us to increase it, I’m not sure that necessarily that we are going to get a different audience, but again, that is one of those scientific questions, but I think that that is the issue for us in terms of moving forward.

What I would say, because I am going to move onto a couple of other issues here is that doing it seems to make sense. What to do and how to do, I think needs some time. So rather than just we go out with the blog and start putting things on it, I think we should say what’s the — I think the two things for me are to be clear with the purpose, and the purpose isn’t just our purpose as a workgroup. We want to make sure the purpose fits what Jim and Damon need for HHS.

And then the second is to be clear about our audiences, because we can’t reach all four of the use cases that I have talked about, and it may be that we are not talking about doing a blog, once we settle into what our real use cases are. It may be that we are talking about something else.

I think we have to like decide how much interaction. Do we want a passive pushout, or do we want the ability — I think Helga was saying this very well is in the sense of do you want there to be users that — I mean, I love those sites that you go to and somebody else like has six things. You say, I can’t, I don’t know how to do this, and six people will tell you. So it isn’t us telling them, but instead you have created a group of people who like to be helpful and will do it. I mean, that is really my preference is that we get other people involved so that it stays fresh that way.

Do you want to create — again, I’m going to use your example of like users group. If any of you have used — at least I have — some of the different software. The users group, you know, ATLAS.ti has a user group. SAS has a user group. And then you go to these groups for people that are — you are often looking for people who are better than you and have other ideas, and you begin to share. So it’s like do we want an NHANES user group? So we start thinking about that.

So I think what we need to do is between the meetings is to talk about trying to get clarity, and I’ll get Lily and I to start with some purpose statements which we will check in with Jim and Suzie and Damon, and then see kind of where we can go from there. So let me — Leslie, did you have your hand up. I thought it moved in my eye, but I wasn’t sure. Okay, Suzie?

MS. BEBEE: So two things that I think about after our discussion is are we developing another HHS blog or are we going to leverage what’s already there, and liability. So if we are going to give advice, what liability is that for HHS?

DR. MAYS: Okay. It’s not HHS blog. It would be like Josh is blogging, and I add something to Josh’s blog. So that’s we said there were two different kinds of blogs we could do. We could do an HHS, or we could do as individuals putting something out there like Geek Doctor or something. You just put it out there, but it’s not at all the official speak of the committee.

DR. RIPPEN: I guess I would like to follow up with Suzie’s point. I think that maybe there is in this case potentially a two-pronged attack. Again, this is a working group and, like all working groups, there is a lifespan, and again the importance of actually people being able to leverage the data that is available is important, and I assume will be so for the long haul. If that’s the case, then you have to think about sustainability and where people are really going to go.

If the intent and the charge is really about helping the government and Department of Health actually get people to use their data, I would actually propose that maybe it is helping the data.gov blog to some degree in addition potentially with then private blogs, because there is nothing better than reinforcement, because the opportunity — because again, that is the intent, and that maybe the charge is also to make sure that we tweet about it and we actually add some of the innovative ideas about how do you do an effective blog and how do you organize it, and maybe provide the opportunity to build off of that too.

So we may need to, but I think we do in the long run have to make sure that from a sustainability perspective in meeting the needs, because I would say NCVHS and even Joshua’s fabulous blog, the question goes to who knows — where do most of the eyes go? I am sure it is data.gov. We can create a subculture, which is the ones that actually then help each other.

So I think that if we can maybe do both, and that will help us move things quicker. That’s what I was trying to say. But we still have to answer the question of what is the purpose, how do we do it, how do we link to an external one to get around some of the other things about helping each other out, and maybe helping the blogs that are currently existing.

DR. MAYS: Good points, because for me what I get worried about is sustainability, because I don’t want Joshua to be the blog all the time. So it’s like we really do have to get kind of a —

DR. RIPPEN: And actually answer the questions, going back to Suzie’s perspective, because they own the data.

DR. MAYS: So that does worry me, and that is one of the things I want to explore before we do it. I mean, how many times have I known about things that started and then it’s like — oh, that group left, and so it’s no longer there, or oh, the kids are doing it now, and you go, oh, it’s a totally different blog now. So I agree.

MR. CROWLEY: So what Lily said and blog aren’t mutually exclusive, but I think the whole greater point of sort of how do we foster a conversation around the data and get people engaged in sharing what they are learning or what their questions are, what they are doing with the data, whether it’s the different apps or the different research questions that are being done. I mean, something that with the right amount of support could really become organic and grow in and of itself, and maybe even HHS or some other resource can sort of moderate the discussion and be a part of fostering a rich environment, but the point being whether the model, some of the product forum models we talked about, both for the cores we have looked at, but something that really fosters a community approach to discussing, sharing the data, sharing results, and then within that discussion, I mean, you could link out to healthdata.gov or other HHS data assets directly from there, but in the sense that the community is building upon itself and having the architecture to support that.

DR. MAYS: Chris didn’t get on? We lost Chris Boone? Okay, I don’t know what happened. We have to afterwards figure out, because Chris was on for a minute, but it may have been problematic.

But that’s, I think, an idea, Datapalooza, like can they push it out?

DR. ROSENTHAL: I am on the steering committee for Datapalooza with Niall and Kavita, Niall Brennan and Kavita Patel from Brookings and myself, so I keep this so I can keep track of stuff, because I get asked the same questions when we have the same discussions, rather regularly. So that’s helpful, instead of just emailing it out.

Whatever you guys want to do, my bias would be towards action. Let me just say that. You could do a product forum. You could do all the things you guys are bringing up, have been discussed repeatedly. So however you want to do it.

It tends to work pretty well, as Kenyon said, when you have a hub that you can link out to other things with and you have a center. I’ll just say that.

MR. CROWLEY: I’ll second Josh’s point. I mean, action. By having something out there, you learn, you improve, you tweak.

DR. MAYS: The only thing that I worry about, and it’s not that it will stop it, is that it has to be sustained. I just really feel before we launch something that we have to have commitment. I don’t know if it’s like something in the department, even though we are being outside it, or if it is something where it is a joint strategy or it’s something that is connected with some entity or something like that, but that’s to me the biggest issue. I can’t tell you the number of these type of things that everybody was gung-ho, started it, and then it’s like — and if it doesn’t start fast enough, it really sinks. Really sinks.

So when you start it, it has to in some kind of way, like it’s the day after Datapalooza or something, or it’s the day after you have the greatest finding from a research study and everybody wants to know how they can learn it kind of thing. So it has to have an event, I think, to launch it. You almost have to thunderclap it out there after you launch it, and then you have to be really — you got people who jump on it right away.

MR. CROWLEY: Is there an HHS health data social media manager, you know, who is responsible for pushing things onto social media and managing releases, that sort of thing?

DR. MAYS: Let’s have that question, because we have wanted a social media person for the full committee, and we don’t actually have one.

MS. BRADLEY: Damon knows more of those inside things. I mean, we have ASPA, the Assistant Secretary for Public Affairs, which has a large social media group, but then for each staff, like CDC also has social media and you can keep having more within the group. Like ASPE could have a social media person if it fit in their budget, but it does not, and I doubt will anytime soon.

DR. MAYS: We can talk to Damon about that, though.

MS. BRADLEY: They used to have a comms director, Steve Randazo(phonetic). He has moved on, but they will probably have somebody filling in. That’s for the IDEA Lab.

DR. MAYS: I think again it feels like déjà vu. We have gone through this. We have great ideas, but it has to work. So I think now the next step would be working with Jim and Damon and Suzie and others to see, okay, are you willing to commit to doing this, and here is the architect of how we think it could be done so that it includes what you have already, but it also includes these other things out here, and how can we make this really work. So I think that that is the next conversation.

PARTICIPANT: Actually, Lily, your list of blogs that are already out there, could you post that on our blog?

DR. MAYS: Okay, I think we have some moving forward with that, some work to do and to see if we can really — because I agree; action is where we need to go.

Okay, let’s take a break. I just want to check on a couple things before we get started instead of getting started and having a short amount of time. I want to take a 10-minute break now, and then what I want to do is to come back and talk about our task sheet and our expertise and things like that. That’s a longer discussion. So rather than starting and breaking in the middle, can we take 10 minutes now?

(Brief recess.)

Agenda Item: Review Charge

DR. MAYS: Okay, people, we are back. Okay, here is the next thing we are going to do. Okay, so we have from now until about 3:55, because we have the person who is going to discuss a web review with us coming in at 3:55 online.  So hopefully they will have that up and ready to go.

So here’s the story with this particular document.  This is really our charge.  The seven items that you see on the front page — so on the first page are the seven charges that were given to us, and if you think about this, this is the space that we should be working in. It is a very big charge for the size and the bandwidth of staffing that we have.

So what I wanted to do today was to go through — and I was hoping Damon would be with us — and go through these charges and discuss what we think is the expertise that is needed. If this is a space where you think that you have expertise — now before everybody gets too quiet, if you say you have expertise, it doesn’t mean that you are getting ten assignments.

Instead, here’s the goal for this. We need to come up with what our work plan is for the next year. We need to have a 2015-2016 work plan, and we need to have a 2016-2017 work plan. What we want to do is this is the charge to us to figure out where we are in terms of this charge, in terms of the expertise that we think requires doing this, the expertise we have in the workgroup, and then we need to work with people like Damon and Jim and Suzie and others in that office to say okay, what is it that you want us to do? Because we have to prioritize, and there are some things that we will think are good to do, such as we have already been down the path with this work in terms of whatever we are going to do about blog or whatever it is.

So that we want to continue. We have been asked to also comment on the framework. So we know that those are going to be priorities for us, but we have to decide what else we can do, based on — and Jim gave us some asks. Also, knowing the expertise that is needed and what is in the room, it allows us to have other people in and out. Some of you are very busy, and there are times at which this works for you and at other times which you say I have a big project and I have to kind of not be here. There are expertises that are not here.

So we need to know as we add people, as we add consultants to the group, what it is that we are looking for, and that we have a sense that that is a priority of a task that we know how to undertake and that we have the people to do it or we need to get a person to do it.

So that is the goal of this. So filling out the priority is not something we are going to do. We are going to work with HHS people to do.

Your workgroup expertise, you are going to fill that out, and the expertise that is needed, you are going to fill that out. We are not going to do all of it in the meeting today, but I want to work through a couple of these and then send the rest of this online and have us work through it, and when we get priorities, if those priorities that come from Jim and others, we don’t have the expertise, we may come back to you and say can you suggest some people who can do that?

So let’s take — I think the interesting ones are to take charge 1 and I’ll let — we can pick another one, which would be — I only want to do two of these for today. Wait a minute. I’m trying to find the other one. Oh, charge 4.

So let’s start with charge 1. There’s a lot of work here. Review the portfolio of HHS data resources, and as you can see, they listed lots of data resources: administrative operational survey public health and research data. They want us to look at current policies, mechanisms, approaches for promoting access in innovative use and applications of HHS data to improve health and healthcare.

What we tried to do is to try to figure out all the actions that are listed here, and for this first one, they are all reviews. In terms of areas of those review, it’s the whole portfolio. They want us to do policies, mechanisms, and approaches.

Then you look at the domains. They are really pretty big. So the question is what is the expertise that’s needed to review the portfolio of administrative data.

Helga?

DR. RIPPEN: I think before you get to the expertise to review, I think what you need is what is the framework in which to review them in. So for example, you know, what’s the scope, you know what granularity from a geospatial, what type of information, kind of nuances that kind of go to the tagging of what it is that others brought out, because if we can have kind of a framework of what data — so what is it that a review means, and that we think about it in a way that we can then reuse that framework so that we can actually then leverage it for the future steps.

It becomes really important, because then we will have tagged it as we review it. We will have the information that would help us decide is it conducive for an app for a consumer in a city versus a researcher versus whatever, depending on what it is. So again, I think that is a really hard thing to do.

Then with regards to expertise as it relates to the evaluation, I think that the people who created the databases, you know, if they could just summarize, you know, fill in the form as they say; what you can do is you can disseminate it and one person doesn’t have to do it, the data owners — you can ask the data owners to do it.

Then what you do is it is more review afterwards, but you have to provide a framework. Otherwise it doesn’t matter.

DR. MAYS: See, this is the chicken and egg question, because we started off by talking about developing a guidance for best practices, and in that guidance it would also be when we reviewed somebody’s website or the dataset that we would want them to answer this set of questions, and then we would then respond back to them with what we thought, have them come into a meeting, and respond to them about what it is that we thought were good.

Part of the problem, I think, and I want to kind of answer or respond to what you are saying is that we may need to do priorities first, and from the priorities then say, okay, now tell us more information. The problem with that is we want to eventually get to this can we develop a rubric that can be kind of given to them all so that we have a framework that can fit kind of regardless of what the —

DR. RIPPEN: I think you can have a consolidated summary. We’ll call it instead of a framework, because we have frameworks and best practices, and that is actually not what I’m getting at here. What I am getting at is really what is the metatags that describe —

DR. MAYS: You’re speaking Josh’s language here.

(Laughter.)

DR. RIPPEN: So in a sense that we do actually two or three things at one time. One is having us categorize what it is that we are evaluating, which I don’t think any of us has to do, and then based on that and the website, then we can assess the value or provide feedback. So it is not best practices. So what can we extract generally across every single one of these that —

DR. MAYS: Give me an example of how you see it working, because —

DR. RIPPEN: So let’s say one of the surveys, NHANES. So NHANES there is a question about what is the covered population, what’s the frequency over years, how many datasets, right? How geographically dispersed is it? What is the ZIP level or Census level or whatever it is? What are the privacy concerns? So what do they currently have? What work has to be done with regards to — you know, it goes to the policies as far as the release, kind of these buckets that you have here, because you would — actually for each one of them, current policy — so is there identifiable information, yes or no? Is there a stripped version? Things like that that you can then have in buckets, right?

DR. MAYS: But we design that, right?

DR. RIPPEN: Well, yes, because we are going to — then once you have that, now you have a listing, and you could even suggest when you think about prioritization, who prioritizes, probably Jim or whoever, but there might be ways of using that as a prioritization tool, too.

Then even, you know, having them assess how many people download it, how many times has the site been hit? How many times has the data been downloaded, because they have all that data? Is it going up or down? Where is it trending? Then if they do that, then we go, we already have the facts, and now we can react to it from a broader perspective, and what I’m hoping is that the data that we collect anyway, we can then use in the future to tag the data that isn’t tagged that way to make it more usable. So we are getting two for one.

DR. MAYS: And your tagging would be what?

DR. RIPPEN: The tagging would be, okay, so who is the intended audience? What is the years? What is the geographic granularity? Because if I’m a developer and I want to do something that is at a city level, it doesn’t do me a lot of good if it is only at the state level, or I have to kind of flub it.

DR. ROSENTHAL: Amen. I swear we came up with a schema before, like audience, geography —

DR. MAYS: We have some things we have discussed, but we didn’t come up with a schema yet.

DR. RIPPEN: I don’t mind helping somebody work on it. This is like the core thing, because if we do it right, it’s like nirvana.

DR. ROSENTHAL: I may just have it in my head. I mean, they started doing it by type, which is great.

DR. RIPPEN: I would actually go on data.gov and see how they tag it, too, because why reinvent the wheel?

DR. ROSENTHAL: The only tag they have is format.

DR. MAYS: I was going to say, yeah, that’s a problem. That’s part of why we need to suggest others. I think that what we want to think about, as well, is the algorithms that we can use in the background, because, for example, I want to know kind of something about any information we can get about users and how long they stay on it. That’s where you are finding if it’s helpful, not helpful. I think we want that little thing to pop up. Some people will do where then they ask you to give a little survey about it. If you will take it.

MS. BRADLEY: We can provide you with guidance on the paper burden here, expedited review for information questions. There is HHS guidance on that.

DR. MAYS: I am going to run up against this, because I’m going to suggest it, because I think part of what we want to know for a website, you know, I’m even less interested in who uses it; I am more interested in who isn’t using it, because you can tell who is using it, but it’s like who do they need to work more on to get access to this information?

DR. RIPPEN: I guess I would be careful — well, okay, I don’t really talk about it as the website. I know the website is the entry point to the data. So again, it’s the question, it can either be hits or downloads. I mean, you know downloads when you have it, because time of spending on it might just be more I can’t figure the heck or they went for lunch and left it open. I think the question is what measures are you using and for what purpose. I mean, in the end if it’s data and you want to download it, you are going to press the button and download it, right, or try to figure it out.

So whatever it is, because it could be time for different sites, but again, it is really — this would be actually filled out by the data owners, and then the evaluation —

DR. ROSENTHAL: One of the easiest — we walked through last year this example of other datasets, where you have top down from owners and from bottom up, you can do one of two things. You can have people self-identify their level of expertise through their community. So you know who it is. Then you can have them assign a star rating to the dataset. You can say this dataset gets five stars by experts who are interested in community health and search that, and all of those widgets — it’s not custom code. That’s the stuff you can do in five minutes through anything.

DR. RIPPEN: But again I want to step and make sure I understand what our charge is with number 1. So with number 1 it just catalogues, right? Is that true?

DR. MAYS: Well, okay. So the first one really is to review it, and in terms of the review, let me just read it, because I think it’s only a piece of it. Review the current portfolio of the HHS resources for promoting access and innovative use. So yes, it is a catalogue, but I don’t know that it’s just the catalogue of the thing. It’s the catalogue of use and potential innovation.

DR. RIPPEN: Okay, so actually now I understand the star —

DR. MAYS: Yeah, so even though we have it here, I want to do the other. So review the current portfolio of HHS data resources and approaches for promoting access and innovative use and application of the data to improve health and healthcare.

DR. RIPPEN: Okay, but some you can give a score to. So for example, those that you have to actually do an official request with justification and pay for the data, would be not an easy to use issue. So there are some measures that we can actually take into account.

The question as far as how do you — we could do it based on opinion of the group, right, as far as how useful do we think, but that is a limited set.

DR. MAYS: The group being us or the group being users?

DR. RIPPEN: Well, but do we have the authority to ask users what they think? I thought that — I didn’t think so.

DR. ROSENTHAL: Just to throw this out, if you did want to do a product forum or blog, you literally — so I looked through dataset by dataset, each one becomes a slug, and then I can do users. You could do this whole thing outside if you wanted to.

DR. MAYS: Explain.

DR. ROSENTHAL: Let’s say I create a blog or product forum and I find the top 100 datasets on healthdata.gov, and for each of those I create a little post, and it’s totally outside of HHS. It’s just me doing it. Instead of that as — when we talk about this community or this blog piece, one other thing to keep in mind is if someone could come along and do that independently.

DR. RIPPEN: Yeah, the nuance is that valuable to whom?

DR. ROSENTHAL: No, there’s like if the dataset is in public, you can have users comment and share their opinion on it instead of just leaving a comment in text. You can just like give a star.

PARTICIPANT: And then you said you can track and analyze.

DR. ROSENTHAL: Oh, yeah, if I wanted to, instead of going through HHS, I could do this in an afternoon basically.

DR. FRANCIS: It is not an OMB problem if you have something on the website that allows volunteers, allows users to volunteer whether they found it useful or not. It’s only if you are doing the survey that you sent out you could make — you should check that.

PARTICIPANT: I am going through that.

DR. FRANCIS: (Comments off mic.)

DR. MAYS: I am going to let Suzie comment.

MS. BEBEE: Well, separating HHS versus Josh and his blog, okay?

(Laughter.)

MS. BEBEE: So my comfort level goes to the bottom when I hear about bringing people in and then tracking them in some fashion. I am just going through an enterprise performance lifecycle and authority to operate a website, and one of the issues with that is whether or not you have a low, medium, or high risk dealing with security, and you cannot track — that’s it. You cannot track the — you have to identify — let’s put it this way, you have to identify what personally identifying information you are collecting, email, whatever, and then you have to, based on which level you are going to be, low, medium, or high, security-wise, what you are going to do with that data. That is the process you have to go through in order to operate an HHS website. That is one of the small pieces of the EPLC.

DR. FRANCIS: They are not collecting user information. All they are doing is having a user without giving identifying information say what was useful or not useful, just like when you go on and access the information on a federal government website. It doesn’t ask you for identifying information. It just asks was this website helpful to you or not, yes, no.

DR. RIPPEN: There may be a different way of doing this. I call it reverse engineering. So the challenge with some of these questions is, you know, if we are using it based on who happens to want to complete a survey or who wants to is based on the bias of whoever they are and what they want to do. It may not be a broader community that you are getting input from.

So if instead we had asked people that represented different industries to come and say what would make a dataset useful or not, easy to implement or not, and even the population health group to some degree on what information is valuable from a population health perspective or public health perspective. One could actually provide at least an insight into what the characteristics are that are important to make data useful or not, and an easy example is is it at neighborhood level or not, because if it’s not, then from a community perspective it’s not that useful.

So again, that may be the nuanced way to actually gather attributes that make data inherently useful or not.

DR. MAYS: See, I was going to say I don’t think that we need to make it our job to actually worry about the users themselves. Our job, we can give advice to the owner of the dataset about ways to bring different types of customers in. That was why I was saying I want, if possible, to get information or to suggest they collect information from the back end so that you can get some sense of who is clicking off, et cetera.

But I think our job should be not to worry about the users but to give advice about whether their website or dataset or ways of accessing the data will — what’s the word that I want — can be used by a consumer, can be used at a community level, or would be, not can be, would be, and to me that seems like great feedback and that it’s up to them to then see whether or not they can reengineer to make changes, and I think it’s kind of one of those things where when you look at giving advice or this framework or this guidance, whatever we are going to call it, that is what should help them then to reach those other levels.

Suzie?

DR. BEBEE: So that said, it makes me wonder about giving advice to the data owners, correct? Is that — why they wouldn’t already know who their audience is? So in looking at number one, administrative data, I think is CMS. I look at survey data, I think of NCHS. I look at public health data, maybe that’s CDC or even HRSA, and research data, would that be NIH or — okay, so I think all of those agencies know who they are providing their data resources to.

DR. MAYS: But what I think they don’t know are ways to think about providing it so that the level of who they are providing it to can change. So for example, I don’t know the extent to which CMS has thought about for every study they have released to have an infographic. I don’t know the extent to which, well, NIH is thinking about this or the extent — like the stuff I brought up today with Charlie.

So I think they do know who they go to, but what we are trying to do in data liberation is to come up with ways that they can go further out in that and so, having information about whether those people even come to them, you know, I think is important. When people go to CMS is if they get the wrong number, are they really trying to get something from CMS? I think that is kind of what we want.

Helga, then Josh.

DR. RIPPEN: Again, I just wanted to go back to action, and so I’m willing to work with someone that knows about the different datasets to come up with at least a first cut, because that way we can then actually discuss things with a focus sort of does this make sense or not and move forward. That would be my proposal.

DR. ROSENTHAL: Yeah, I also for my opinion I deny the premise that the HHS folks know actually who their users are. We spent last year hearing one group after another saying, no, that’s not the case. I believe some of the quotes, we can look back through public record, are we had this attitude of leading a horse to water and telling them to drink. Literally they weren’t tracking users until last year we say we have this data; that was a new idea for them. But for one example, you want to take BRFSS for example. You know that’s being used not in research and communities but by public health plans to risk adjust, post the BMJ model, but owners of that had no idea.

The point of the story is there is a lot. We put that session together in Health Datapalooza and put the BRFSS people up there with the payers and providers using it. Those were new communities. They had no idea that these old school sets had massive value of — there was no connection there.

So at least from my experience, I think although the HHS people know a lot and their data owners know who uses it, I think they can know some more, and that can be a massive contribution. That is one of the things we put together with HHS and CMS and RWJ. That’s but one example.

So I think there is massive contribution to be made saying, hey, you have these datasets you are putting out there. You think of this as community and research. You know that this has massive value to the private sector beyond some kid doing some health app and making money that way? Like literally we did that with Wennberg published at BMJ where that set actually out predicts claims for risk adjustment, right? Oh, frick, now there are dozens of health plans using that for their population health risk adjustment. Did CMS have any idea behind that? No. That’s but one example of like massive value in the system that I would expect or at least they told us all last year that they were interested in feedback from the market on.

DR. FRANCIS: And massive questions about it is really important for everybody to know how this data is being used.

DR. MAYS: I think that that is something as well that some people don’t have a sense of how their data is used. That’s why one of the things I thought would be useful for the committee, either just the pop committee or the full committee, is to get a sense of at the county or city level, the way that people take data and do mashups so they can get a sense of the range of ways that the data is used, because I don’t think that the feds even know the way in which you can help an area to better utilize their data in terms of everything from being able to map foreclosures that will tell you where you can go buy the cheaper houses to, you know, being able as I gave the example, being able to reduce the rate of foodborne illnesses.

They did that by an app. They worked with Yelp and they identified — so you would get on Yelp if you had a bad outcome, and so those people once they realized that it was on Yelp, they cleaned it up. San Francisco reduced foodborne illnesses by 33 percent.

So there are ways in which I think that different groups aren’t aware.

Leslie, you want to say something?

DR. FRANCIS: I already did. The comment about — it’s really important to have knowledge.

DR. MAYS: Okay, so explain just so I’m clear and we are all on the same page about what you would see relative to charge 1 to do.

DR. RIPPEN: So charge 1, the first step is kind of put together I guess categories that would be attributes of whatever the dataset or the website would be that would be important for us to actually meet objectives for charge 1, and would also include characteristics that we might want to get. That would include the charges from the different potential user groups, just to make sure that it’s not based — you know, sometimes you drink too much Kool-Aid and you lose kind of perspective. So also make sure that we gain some additional insight from potential user types just to make sure we are not missing anything.

And then so I think that would be the first one, but it would require collaboration with someone that would be intimately familiar with the types of datasets, just to make sure you are not missing some categories so it can be used regardless of what the dataset is. Then vet it, because that’s the hardest part is this kind of infrastructure kind of form.

Then once everyone is comfortable with it, then the second phase would be to ask the data owners, and again, the prioritization, we probably want to test it with a few data owners that might be at a higher priority, and then based on what we get back from them, then go to their site to see did it make sense or not and figure out do we have to adjust things because they are not clear, because we don’t want to waste anyone’s time.

Again, then we can actually do a quick survey and we could probably even develop an algorithm that calculated some score and given the usability is such a big thing, it’s just kind of funny, right, with regards from a user perspective. So I think that actually would help achieve charge 1 in a pretty efficient kind of way. So that’s what I would propose.

DR. MAYS: Okay. Here is what I would suggest, which is that we try to develop this as a little team and a partnership. I think we should ask one of the NCHS people if they can work with us on this. My first go-to person would be Susan Queen, because what you want is in NCHS they know a lot of the datasets. They know them intimately. So what you are getting is someone who is making sure that the questions are the most relevant for that whole bank of datasets.

Then I think, because of the twelve surveys that Jim talked about, a majority of them are under NCHS. So once you get the first cut-through, then I think what we do is bring in someone from CMS who does the more administrative datasets. So I think the way to do it is to kind of in layers, to start with NCHS, because they are the majority of the datasets, and then — when Jim said the twelve surveys, I think that most of those surveys are under NCHS.

DR. RIPPEN: Okay. If that is his priority, that’s fine, because I know there are a lot of datasets at CDC, for example, like a lot, and then also CMS obviously has a lot of data.

DR. MAYS: See, that’s what I’m saying. I think what we do is we bring CMS second. So what I am trying to do is start with who has probably the biggest number of datasets to do the foundation, and then what we can ask is somebody from CMS to look and give some comments that we need to probably get somebody from CDC Atlanta that is doing the public health datasets. We could ask Vicky Booth, for example, to help us. So I think that that might be kind of a team approach. Does that sound reasonable to you?

DR. RIPPEN: Yeah, just I wouldn’t want to do a lot just with one view, because again what you want is one type of — I call it a form, just because it is easier for me to just say form, because what you do want is one form that can be leveraged across regardless of the dataset. So you are absolutely right as far as your approach. If we want to start with one to get first feedback, that’s fine, but before we ask them to really do it, we need to probably get feedback from the others.

DR. MAYS: I agree. That is exactly what I am saying is we are starting there, but I think that we can do exactly what is here. The people who are doing public health, et cetera, to also then weigh in and make any changes so that this form is useful regardless. Sometimes it is just a tweaking of — you know, it’s like when you actually do a survey and you have two ways of saying the same thing so that you make sure that everybody recognizes the question they are being asked.

DR. RIPPEN: In the spirit of what Charlie said, and it won’t be taking an hour.

DR. MAYS: So I am going to say thank you very much to that, that that’s a good start. I am going to see if we can get Susan Queen to help us with an NCHS person, and if there is any commentary, Suzie, that comes in terms of Jim’s office with any input, please, and this is the first small group.

Second small group is going to be Josh in terms of the blog. So we are going to put you over — in terms of the blog. So I just want to make sure nobody gets overloaded in terms of time. But I think if we start with charge 1 that that’s actually a charge that is the priorities that we heard from Jim so that we are really doing pretty good.

Let’s look at one more charge, and then what I want to do is in a few minutes we will have Neely Current on, who works at the University of Missouri in the School of Information Sciences. She has a group in which they do blog development — I’m sorry, website evaluations. So you actually have a rubric that she is suggesting. So I think we want to also make sure — you should have this.

So she will come on, and then I think what we want to do is to understand — the way I kind of see this, what she has, is that it may really help us in terms of those consumers and community groups in terms of the — so let’s consider this relative to what we are actually talking about and see then how it goes.

Okay, so is she on? Is she just going to be on by phone, only by phone? So she is not on video.

Okay, so let’s actually go to her now, because since we just kind of come up with — so can we hear you?

(Discussion of tech difficulties.)

MS. CURRENT: Can you hear me?

DR. MAYS: Yes, we can hear you very well now. What I thought we would do, because we just finished talking about kind of some of our datasets and how we would evaluate them, it would be great to hear you go through and discuss the rubric that you sent to us. It would help if you start by introducing yourself and saying a little bit about your background, just for the group.

Agenda Item: Website Review, Process and Product Discussion

MS. CURRENT: Thank you so much for having me today. As I heard you introduce, I’m from the University of Missouri in the School of Information Science and Learning Technologies. We have a group for the past 13 years who have evaluated websites, software, applications, and over the course of that time, we developed and implemented many methods to do that, and most of it is on user-centered designs, and so that approach means that you put the end users at the forefront in your design process. So you want to hear their voices, their opinions, and their needs as things are designed and you confirm that the sites and the tools that you build match their needs.

So what I have just sent you is just one small part of the whole process that we do, and it could possibly allow you to do kind of standard evaluation if you are looking to evaluate many websites. This is just a basic guideline for a way to evaluate sites.

Generally, though, it will allow you kind of a baseline of evaluation. What I would also recommend is doing user studies, having them give feedback, having them use the sites. Often we record sessions. So we have them navigate through the site in a task-oriented approach. So can they use a particular graph, identifying those end users in the beginning and checking in with them as you design and build. So this tool that I have here is only one small piece of bringing in the users into the design process.

So if I was understanding, your committee was seeking a way to evaluate many websites. Was that your kind of end goal?

DR. MAYS: Yes. What the committee is interested in doing is looking at several of the HHS websites and trying to understand several things. So we are trying to understand, for example, how to make the website more usable to drive it such that individuals who are entrepreneurs will use our website and develop apps or other things that will then push the data out more. We want people who are researchers, for example, even though they know their way to the website, to have better ease. We want to figure out a way that community can use this data as they go about community health planning and thinking about how these national statistics might be a source for them to compare themselves.

Then of course for some things we might even want consumers to be able to go on and navigate their way through. Now not every website is going to be able to satisfy all of those user groups, but some should and some will try.

MS. CURRENT: Okay, so if I’m understanding, you are wanting to make sure that the sites are usable for your constituents and that they can actually use and manipulate the data perhaps for entrepreneurial uses or to build apps based on the data that is available.

DR. MAYS: Or to use it in their community settings, to use it as a researcher, to use it as a consumer. So it really is to push it out and that you come to the website and that, first of all, what you do when you come to the website is you use it and that it is usable to you, and then after you use it that it results in something, depending on who you are, you take the advice, you develop an app, you compare your community. So we want to be able to do this in a way in which there is something that comes of it. So not that they just visit but that they visit and they do something.

So maybe what you can do is then talk us through how your usability of website design kind of works, your thinking about it, and who is your customer, when you use this approach?

MS. CURRENT: So we are a group of master’s and PhD-level students employed at our lab, and then we train them to do these usability methods. So we serve a range of clients from corporations and healthcare to journalist groups, startups, entrepreneurs, and so they come to us at different design stages. They may have nothing built. They may have something, a website, for 10 or 15 years, and want it evaluated.

So we tailor our work to where they are in the development process, and as I mentioned, there are a variety of methods to bring in the user experience. So user centered design incorporates the user in the design process, and one of the main approaches is, as I mentioned, kind of what’s called a task analysis. You identify your core user groups and then you — I’m hearing myself, it sounds so strange to hear myself at the same time — so you identify a user group, and then identify key things that you want them to accomplish on the site.

From there, then we can conduct a session where we ask them — we have the site up on the screen. The computer is recording the session. So we say, okay, you want to create a dataset on literacy rates for this population in this area, and then we watch them try to do that on their site, or health statistics in this case, and then they would speak aloud and describe their experiences, and we also record and make notes.

So we see where they struggle, where they may not be able to use a particular tool, where they like a section. We evaluate kind of based on this rubric. We look at the language, the content, navigation, aesthetics, design.

So at the end of the process you have had sometimes up to an hour interaction with the end user up to, I would say, the starting point is five users per user group that have distinct tasks, and then by the end of that you have a very much better understanding of how people are actually using the site, not just what you think they do. So that can be a very specifically designed session.

In the end you have a report and video recordings of the sessions, to see how people are really using the state and recommendations how to make improvements for your developers. So that is a very common user centered design process that is very useful and contributes to an improved site.

MS. BEBEE: Hi, Neely, this is Suzie Bebee. You and I have exchanged — hey, how are you? So when I saw your framework that you just commented on and the different categories, what we talked about before was looking at the SAMHSA website, and I think you guys had started to do that.

MS. CURRENT: Right. So one of the methods that we call design review, others would say like a heuristic walkthrough, and it involves the listing of those criteria. So we look at the layout, the design, and information architecture, et cetera. So this team of — usually we have three people independently review a website and using usability principles, and then we also provide screenshots and recommendations for how to improve the site. So that would be called the design review, and our team has started informally to review the SAMHSA website. That would be something we volunteered to do and we can provide you so you have a better idea of what usability is and in this case what a design review can offer.

MS. BEBEE: So I’m wondering if there would be one or two more of these categories. You have nine, and if we would develop something that targets what Vickie just described about the data itself; some of these websites, SAMHSA being one, substance abuse, our agency, within HHS, has the data, and we are interested in knowing the usability of that access to the data, and we may have some ideas, some guidance, that could be added to this particular framework that would target some things that we are interested in. Is that a possibility?

MS. CURRENT: Absolutely. Often we would tailor a document similar to this, and we have done one for e-commerce. So we added in specific things that pertained to e-commerce. We did another one that was about library usage. So we added something specific to what were patron needs. I think that is a great idea to add in section that tailors to, in this case, data visualization, data usage. So that would be a good addition to this rubric.

MS. BEBEE: Some of the things that we have talked about earlier, Neely, is the data geek that is coming to the website to get the data and what are the tools across a spectrum of tools that would be used or are they offered and that would be in line with some more of this framework that we would add to.

DR. RIPPEN: I appreciate the fact of adding the component of the data, because usability for complex sets of datasets depends on your hat, and so how do you help people navigate? So it is a usability maybe of how to use.

And as a separate thing to consider, because I don’t know if it is, so there is usability of the site to make it as easy as possible for different levels of skilled users to leverage it, but then there is another component, which is it the fact that people are there and they are not using it, or that they have never even gotten to the site, because then usability is a moot point. So again I think the question of how does one disseminate that information, and I know, Josh, you are bringing it up to some degree with the blog, the whole how do you get the word out. So I think that we have two challenges, potentially.

MS. BEBEE: So what Helga was saying is we are interested in learning about the websites that HHS is responsible for and in this case just the one, SAMHSA, with your gracious offer to do that for us, but we also are thinking, just FYI, of how we would then be able to disseminate that information about the website to the agency, broader across HHS, that kind of thing. So that is out of scope for you, but it is where our thinking is.

MS. CURRENT: Right. Oftentimes we have worked directly with the developers. We have different stakeholders involved. So communicating with them to see what they are already doing. They may have some Google analytics in place for this site. They may have other approaches where they are collecting information about the end user. So I think that would make the most sense is to see what the developers are doing, because they are already — and communicate with them that we are working together to enhance the user experience.

MS. BEBEE: So do you have anything where you are looking at — have you looked at this for other customers that you have had where the improvement of actually using data that is a resource within the website, have you had any of that experience when you do your review?

MS. CURRENT: Can you clarify? I am not understanding the question.

MS. BEBEE: If you are looking at a website with this prism and the framework that you have, and then thinking a little bit beyond that to customize about the data that I referred to and seeing that they offer public use files, let’s say, and maybe tools in order to analyze it, download it, all the things that someone that would want to analyze the data, do you have — have you used any evaluation framework for that specific purpose?

MS. CURRENT: We have used this evaluation framework or one similar to that over the course of 13 years. It has varied as I said over the time. At one point we had over 100 items in the checklist, and really it became more tedious than useful. So we pared it down to this smaller tool. We added one for GIS, one for libraries, one for e-commerce and then what happens, then we recommend we evaluate the site using this and then the developers or the stakeholders make a decision about which changes to make on the website.

DR. ROSENTHAL: I think at least one of the things that I am interested in — I have heard Damon is interested in and is at least two or three of the charges, as I see it, is it is not so much about UI and UX as like close your eyes and imagine there are 500 different data sources; how do I find those pieces of information? People like you use this thing. This thing is similar. I think is outside of the font and navigation and checklist, there’s also an information management, information architecture, similarity index, like kind of basic thing.

Is that basically what you are getting at, Suzie? Like how do people — the issue isn’t is the font big enough? That is definitely very nice classic UI and UX. The issue, at least as I see it, is there are all these data sources. There is no organizational rubric to it. There is no folksonomy tags bottom up. There is no tags and categories top down. If I am going to the site, there’s how do I find out about it basic indexing, but once I’m there, not just tasks and what do I do, but how do I actually — to me, I see the problem as an information organization problem more than a UI/UX problem. So just to clarify what I think people are trying to get at with you.

DR. RIPPEN: I would like to build on that, because I think that is kind of a core thing, and it’s also the question of if I am looking for a particular set of data, how do I quickly find it? Because a lot of times it is imbedded in bigger datasets. So again, not only how do you organize it and get to what you want and how to do that, but then also how do you actually find something very specific, if that is what you need, and quickly determine if its attributes meets your needs. So it is a very complex sort of thing, and I don’t think any of the websites are even close to thinking that through. So there is a future of what we would like versus what is the standard data website right now.

DR. ROSENTHAL: There are definite best practices from e-com you can use. If you think about a dataset like a product basically is one way to think about it.

DR. MAYS: I think the question we are trying to get at is whether any of that can be added onto this. Suzie?

MS. BEBEE: What I was thinking of when Helga was talking was SAMHSA has a dataset, and if we would give you that name and focus, as Josh was saying, on the navigation and the ability to navigate easily or not, using your framework but focused more on thinking of people wanting to find the data on this website and how easy is it to do that, how easy is it to use it, what tools are offered, but from a user and friendly kind of approach. Does that make sense?

MS. CURRENT: Yes, right. I am seeing if you have many sites that you are wanting to kind of evaluate, like I think it was Josh who said you’re not necessarily interested in the UI/UX, which kind of this rubric does focus primarily on that, but more understanding how the dataset serves the end user or the struggles that they might have. I think that can be addressed through some of the user studies that I mentioned. That might be another approach like seeing how people are using the datasets, where they struggle with it. There are different methods that you can use to improve the information architecture to improve the way that the datasets are viewed.

DR. RIPPEN: I guess I want to build on that, because heuristics —

MS. CURRENT: But if you are wanting kind of a systematic approach, the rubric is kind of a baseline for looking at an evaluation of the site. So it’s kind of a starting point, and then collaborating with the designers to address some of the information architecture.

I think Lily mentioned doing that kind of comparative review. That’s another way to find out the way other similar organizations or that have large datasets, how they are managing the data visualization, the extraction of the data. That’s another approach, looking at other groups and entities and how they are displaying it.

DR. RIPPEN: This is Helga Rippen. So going back to heuristics is I have a website; I can tell you what works and what doesn’t work. What I guess it sounds like what we are interested is while we know we have people that want to find information and it’s complex, so how do you help them find something, which is more like, you know, when you do rapid prototyping and you engage kind of your human factors experts early in the process for core principles. So what is it that they need to do and how do they need to do it? So that way you can help guide the construct.

But that’s really very different, and I guess the question is a little bit of scope, and I think that your recommendation about maybe doing a comparative assessment may be kind of that intermediate approach where you can look at best practices would be like a Google which has a different way of approaching it, versus some of the government sites and even shopping sites.

The nuance there is obviously the government has some other higher level restrictions as it relates to visualization and what they are able to do or not do on their websites, which makes it a little bit more complex, too. But that might be the intermediate kind of approach.

DR. MAYS: We have a couple of people online. We have I think Chris is online and Hillary is online. Do either of you have any questions or comments? We are looking at the rubric that we sent you around evaluating a website.

DR. DAVIS: This is Chris. I’m here. I agree a lot with points that were already raised by Josh. So I don’t think I have too much to add to that.

MS. HARTLEY: Hi, this is Hillary.

DR. MAYS: Hi, Hillary. Can you introduce yourself? Many people haven’t met you before. So if you could just take a moment to introduce yourself.

MS. HARTLEY: Yes, sure. I am Hillary Hartley. I work at 18F which is the newish team inside the General Services Administration. We are sort of sister organizations with the USGS and also with the OCSIT, the citizen services and innovative technology team inside GSA. We work very closely with both of those teams and are essentially a client services team. We are a software team and we are also a consulting team. We work with any agencies that can approach us, and have the desire to work on an idea, a project, data, anything.

I believe I missed some of the discussion. So talking about the earlier discussion around — I am kind of super interested in it as the framework of a more typical heuristic review, which we haven’t done too much of. I mean, we are generally starting products from scratch and kind of building it another way with user research and user testing baked in. And of course we are guided by core usability principles and accessibility principles, but sort of seeing it laid out like this and with a score was really interesting to me and something that I would like to sort of think about more with our team as we are faced with projects that are kind of revamped or something along those lines. We approach it very much from a kind of user centric focus on what the users are appropriately telling us they need and then they are able to get out of it versus a more sort of heuristic approach. So this is interesting to go over this morning.

MS. CURRENT: Right, and Hillary, I think that is definitely the way to advocate the agile design process where people — you are getting the user feedback throughout the design process and the group here has suggested what is the way for us to — is there a measureable way or approach to evaluate each of the sites, sort of in an equal weight or to score them. So this is something, a tool we have used, but I do think it’s focusing on that iterative design process and testing it with users and observing, helping navigate the site, it’s a much richer way, richer experience and richer way to give feedback, and the developers will have a much better sense about how the site is used rather than kind of a rubric style.

DR. MAYS: Let me take a comment from Kenyon, and then I think we need to pull ourselves together to determine what the next step will be.

MR. CROWLEY: Thank you, Neely, for joining us. I think it is great work, and I think to figure out how our specific context fits into that is probably one of the key challenges, and I just want to sort of bring up a point to the table, because we talked about some of the specific theme areas that this type of assessment might touch during some of the spring sessions, and I know some folks are new. So let me just share that with the group and see if there is any feedback about how that fits here.

So we talk about eight concepts that might bear relevance in some sort of rubric. We talked about availability and access, findability, usability, usefulness, comprehensibility, linking and combining data, and support from the data provider. We have some more, and I won’t go into each of those now, but I just want to make sure that that’s still sort of on the record, because we had some of those discussions before, and I think it will be important as we move forward to build off that.

DR. MAYS: We don’t intend to drop those. Part of what we were attempting to do is to determine if what can we add onto, I think, this rubric and then we had specific things we wanted. That’s why I have always pushed that before anybody does it, let’s the workgroup be able to have some comments on it, as well.

Leslie?

DR. FRANCIS: This is a question really for Josh in a way to link to the discussion, which is that as I understand it, one of the problems is of course all about finding the dataset you want to use, but typically isn’t what happens that somebody identifies the dataset and then downloads it? Have I got that right, Josh?

DR. ROSENTHAL: Yes, I think so.

DR. FRANCIS: So you are not going to be able to tell by evaluating what people do on the website whether people are encountering the kinds of problems that concern Josh, like whether or not the data comes with the appropriate architecture.

DR. ROSENTHAL: There are three different layers to this. The first is is the stuff indexed and can people find this from external sources, which is what we have talked about a little bit, and then the second layer is it really is this information architecture; can I find it?

The way to think about it is I go on Amazon. I find a pair of shoes. People that bought the shoe or looked at the shoe liked these other shoes. That is literally the way to think about that. That is done on other government data sources. It’s worth probably digging in and finding out is that possible with HHS. I think I have heard from various folks it is. If it is, that is very significant information.

Then the third is actually when you do download it, what is the nature of that data? Is it usable, useful, comprehensive, et cetera? There are kind of three layers to it.

Does the stuff get out there? Which the answer right now is sort of no. When I search in a search engine for something, I am not able to get into the right areas. When I do get into the area, can I find something meaningful? So if I am a community person, and neighborhood grain is important to me, can I pull up all the datasets that have neighborhood grain? No. I can’t do that. That is that information architecture piece we are talking about.

I have pulled up a set that has a neighborhood grain. Show me other sets that have neighborhood grain. Just even in the architecture of the site or the content discovery system, if you want to call it that, and then finally in the delivery system, when I actually download that, what is the nature and utility and usefulness of that actual data? There are like three layers to the thing.

So Leslie, I think you are describing the difference between the second and third layer. So there definitely is the data timely, worthwhile, but even before I get to that, can I find the thing or is it just an alphabetical list of stuff is sort of another issue.

DR. FRANCIS: Yeah, I mean, it seems to me — I really appreciate the three layers, and it seems to me all three layers matter, and at least some of the questions that we were discussing in the earlier part of this meeting had to do with the third layer.

DR. ROSENTHAL: Exactly, and if you get them, they are complementary in the sense if I index my metadata on my data source discovery and I put that into a reasonable format that is discoverable by search, like they all three matter and they all three build upon one another.

DR. MAYS: Here is what I think we should do at this point, which is to try to see how we can roll this out. You have been wonderful about offering to do the SAMHSA website. I think what we would like to try to do is to blend some things into what you have already suggested, if that is going to be workable, in the space that you have. Suzie, I am going to kind of start with you. You see what’s in the rubric. We would like to add some of the things that the workgroup has talked about previously, and we have a list of them and then maybe there might be a couple of things that evolve from today.

So we are interested in things like, you know, you have that how accessible is this, how usable is this? So we have some concepts already that we wanted to do.

Suppose what we did was send those to you and see whether or not you could blend those into the rubric you have, and then talk about the users, how you get to the users of the website. So would it be — it would be your graduate students that become the users of the website, and remember, there are two things for us. There is going in the website, but for us it’s also getting to the data. So for NCVHS, we are very interested in people using our data.

So SAMHSA stands as the website that you go through to get to a particular dataset. I am sure it’s someplace else, but we want to kind of do it by looking at the federal entry through their website. Is that feasible for you?

MS. CURRENT: Yes, it definitely is, but I think what Josh was describing is looking at those three different layers. Those are, I think, you nailed it exactly, Josh, that there are three major issues, layers of the user experience and what that involves, and I think as a committee you are trying to work what methods or approaches can you use to address the problems or issues for each of those?

This rubric is really just a small speck in the whole sea of managing the user experience and the site and making sure it’s functional. So while, yes, we can use this rubric and add to it, it sounds like you had some discussions already with some very valuable criteria, and we can merge those. So we can definitely do that.

I am just saying that this overall is not going to lead to a long-term sort of fix to the issues of managing a site such as this versus to look at those three layers that you have identified and figure out approaches to address — because those are major pieces, the data access, the user experience. So in the short term, yes, we can blend this rubric with those other criteria that you mentioned, and then you can kind of see what you could expect from this method. I do think there are other approaches that will expand your ability to change the site. But I’m just describing some of the limitations of this tool in essence, and don’t expect us to be the fix.

DR. MAYS: No, I don’t think we were. I think we were actually thinking that given that you have given us some volunteer time to do this, that we would actually push this out to see what it is we get from this, what the limitations are, and where we should go.

Suzie, that was how I was thinking about this, and I just want to make sure that that would be useful in terms of, you know, for Jim’s needs and for us to kind of push it out and see kind of how it happens.

MS. BEBEE: Neely, correct me if I am wrong, but what I’m thinking is that the list that we have of things that we want to target specific to data could be either an add-on, or it can be incorporated into these nine categories possibly. So for instance, if you go out to SAMHSA, if you go out to SAMHSA and look for their data, just if you Google SAMHSA data, it will bring you to their data site, and what you see is a usability piece that talks about fast finder, data archive, the type of data, evaluation services, quality metrics. So that’s the way SAMHSA set it up, and data is embedded within that.

So all of the things that you have talked about here are one high level evaluation of a website, but maybe the easier sweet spot to make this worthwhile, a win-win, is to then target that particular data page. That’s my thinking.

MS. CURRENT: The data section of the website.

MS. BEBEE: It is the public use files is where this particular page I’m looking at for SAMHSA.

MS. CURRENT: So it sounds like the next steps forward would be if you send me the other categories that you discussed, both the data-specific ones and then I think one of you mentioned the past work you have done and the categories or the criteria, if you sent those. You mentioned content and I’m not sure who was speaking that listed them out before.

DR. MAYS: Here is what I think we should do, which is come up with our other categories. I am going to ask Josh to put in writing the three layers, and I am going to pass that around to the workgroup first to just kind of get a sense of for the SAMHSA website, what we may want to do, and combining it with — we will do this over email. We don’t need a phone call for this. We can do email.

And then send that to you, and I just want to make sure I’m clear. So your graduate students have or have not used the SAMHSA website before, in terms of the data?

MS. CURRENT: They gave just begun looking at the data piece.

DR. MAYS: Okay, I think that that’s a good group, because they are not yet at that stage of being data entrepreneurs. They are not at the stage of necessarily being trained researchers who just go to it automatically, but they have a skill set that should allow a person who is just ferreting around for data, which is what graduate students often do, to see how they fare on this. So I think that that’s actually very good.

So they have just enough and they have enough experience that just to get through the technology is not going to be their issue. So we really will get to see how it is that they kind of try to approach the data itself, and that is what we actually want to know.

So I think this is a win-win. So let us see how quickly we can push this stuff out to you, because one of the things I am cognizant of is students will leave in about three weeks. They should be gone. So we want to try to do this within the week to get this out to you so that you can get it to them and get it back to us.

Any other suggestions about this particular project? Okay, thank you for doing this. This is very helpful to the committee. We really appreciate your putting the time in, because it will help us to move ahead on this. So we want to see kind of what happens. We are excited to see what is going to happen, and I think it will kind of launch us in moving forward. So thank you very much for your time. Thank your students who are being willing to do this.

MS. CURRENT: Thank you for having me today and look forward to continued discussions and what we can come out of the rubric here and adding to it.

So the end result for this stage is just a written report that describes the findings. So that is what you could expect, I would say, by the end of the year for sure. I can give you a better date once we have the tool finalized.

DR. MAYS: We are thumbs up on that. All right, thank you. Have a nice Thanksgiving next week.

Great. So I think we have made some progress. What Lily and I will do is work on getting this out to you. You have the rubric. We are going to point to some areas of the rubric where we are going to say should we drill down more on this so that it is just very specific to the data, because what she is really thinking about when she started is being very specific to the website. We want to pull her back and get her back to being a little more specific in terms of the data.

So when you look at it, we will take a first look and say should we adjust this, and then we are going to ask you to have the same eye of looking at adjusting it to be a data focus as opposed to just a website focus. Then the other thing we will do is include for those of you who haven’t seen it — you know what, Lily, we need to give people access to the — so we have lots of materials. So what we want to do is give you access to it. It will give you a sense of some of the past materials that we have. You also will be able to see kind of the beginning of the guidance that we were developing, which Kenyon did a — we did a presentation at NCHS about. Kenyon talked about that part. I talked about some other parts. So we have all that stuff there. So we want to share that.

I kind of forget that they are new. Oh, yeah, we have that. But I need to make sure because we are trying not to operate with paper all the time, that you just have access to these things. So let’s, so you and I will talk about that and get that out to you.

So I think we are doing good on that one. So let’s see what is going to happen.

Agenda Item: Feedback on Population Framework

Okay, we just have a few things left that I want to try to see if we can do a little bit on, and then we can get people out of here.

You were sent — and Bill Stead talked about it a little bit — you were sent this framework, and for those of you who haven’t been here, Bill explained it to you, but let me try and put it in context. Leslie has heard the framework stuff for a long time. But anyway, in our work at NCVHS, part of what we try to do is to put ourselves in the space of a learning system, and in thinking about that, what we wanted to do particularly for population health is to have a framework, and that framework is really designed to try to be an architect for bringing in — in our subcommittee there’s populations, there’s privacy, there’s standards, and this framework, if it works correctly, should really allow us to be able to build a lot of these things in in a way in which it is effective; it’s efficient.

So Bill talked about this when we came to the table, and I think you have received it. What I would like to do is to see if there are questions about the framework, and if not, I’m actually going to ask Greg if I can get him to take a lead on this, because you are the newest. So you haven’t heard that much about it. You didn’t sit in the meeting. You only heard what Bill said, and so I think for some of us, we have looked at so many iterations of it that I think having your sense of it and we sent along specifically his questions in addition to what he said today, would be very useful for you to give us the first feedback for us to then kind of respond to.

So would that be workable for you?

PARTICIPANT: Sure.

DR. MAYS: Okay, so let me first see if there are any questions in the room. Some of us have been with this framework bit for a while.

Okay, so we have sent it out. If there are no questions about it, then let’s start with it as an assignment and to get you to do some commentary as to from the side of technology. That is what Bill is really wanting is are there things that we can suggest and do that would make this work better, and Marietta, do you have the roadmap, the little figure with the roadmap? Do you have an extra copy of it?

This is the roadmap about the framework, and so we would like you to be able to see that. I was on the bus this morning saying, well, maybe we could put a cloud in it or something to talk about the technology part. So we were kind of losing it. So this is why it needs to be somebody totally different at this point, because I think we have kind of been using it for a long time.

Leslie, I forgot you weren’t here today. So you will at least be able to see it. It was the other day. It was Tuesday. So okay, let’s start with that, and feel free to email us if you have any questions, but we are going to make sure that you get some of the materials which would also be just the pictorial thing.

DR. FRANCIS: One of the ongoing questions about the roadmap and the framework, just as a way of putting this for people, is how far apart are the kinds of things that the roadmap highlights and the datasets that are actually available, because the roadmap, just to give you one example, it really wants population, not only population data but neighborhood data. None of these public use datasets get close. So that just might be a way for people to think about it.

DR. MAYS: Okay. Let’s make sure you also — we will send you the graphic as well, Leslie, because they have done some updates to it. So I want to make sure you have the same one that we are working with. So you have the file. That will be okay. But I also want to send you the graphic, which was what was used in the hearing.

Agenda Item: Suggestions marketing/dissemination plan for privacy toolkit

DR. MAYS: All right. Thank you very much. Chris, are you still on? One of the things that we wanted to talk with you about — and Debbie, we need you at the table — is our privacy subcommittee has a toolkit. Did you see the toolkit when you were here last time, because I remember they were asking you about whether or not they could present it at Datapalooza and things like that. So I wasn’t sure whether you also saw it. So we will actually have to get you a copy of the toolkit.

But one of the things is that you did a great job at distributing your report. We picked it up in lots of different ways. So I am going to tell you about the toolkit. The toolkit, which is kind of under the auspices of data stewardship, it really talks about how to use data and it gives a lot of guidance about — for data owners — the way in which to be good stewards about data and to respond to concerns about privacy, confidentiality, and security, particularly when you start using this data at the level of the community.

So right now it is online. So if you were to go to the NCVHS website, you will actually see it online. Jack, can you go to the NCVHS? Oh, okay, Lily can, I think.

Debbie, if you can tell us a little bit about the format you have it in and where it has been distributed, and I want to get some ideas from Chris about other ways that we may be able to blast on different places and distribute it.

DR. FRANCIS: This is Leslie. I sent Lily the single slide about where it has been disseminated.

MS. JACKSON: Just so you know, Leslie, your name was mentioned with praises as they discussed the toolkit, and the penetration at we think the basic traditional level of getting it out to the various audiences, we have used it. We have sent it out to the NCHS list with Vickie presenting Vickie and Kenyon presenting and Jim presenting at the National Statistical Conference. That was very successful, too.

Linda brought it out to AHIMA, and then I think there was a community comment was I think our mother lode that was an unexpected find that was really Chris Fulcher kind of grabbed it and grabbed the ball and went on that.

So what we have been doing generally is what the committee has been used to doing is first layer and level of penetration and dissemination, getting it out to friends and family and advocates and folks that we know about, but what Linda wants to do in her launch is to go to a next level of social media, which is kind of above and beyond where the committee has gone before, and that is where the review, where this working group will be the eyes and ears and hands really in letting us know just what and how to get this out as you were talking about various forms and blogs, and it was very exciting that people would not know about, would not generally have information about, at least at our committee level.

So we are looking forward to hearing, going to the next level, Chris, and with the working group and using this almost as an example of what can be for the next iteration. We have the toolkit dissemination posted now.

So essentially it is to help the committee go to this next level of social media penetration. When the kit was developed several years ago and really kind of got up on its feet this year, there have been vehicles and display places that really were not even in existence like three years ago in terms of where the committee was.

So the committee is kind of, I must say, catching up to what this group and other groups have been doing all along, and we would like to just keep, get on that wave, and we have the toolkit now posted. It’s on the web, and this is on your screen now.

DR. MAYS: Chris, one of the things that would be helpful for us, and you can also email us once you have — because for some reason, I guess I thought when we talked about the toolkit you actually had seen it, if you could look at the toolkit and give us some advice about two things: how to just get the toolkit itself out, and if there is anything like you would say, like, if you did a one-pager or if you did a tweet or if you did something else that would help us to penetrate into different networks other than the networks that were there.

Can you put the networks back up, the slide that Leslie sent us?

We will actually send you this slide so that you can see where it is. We have two different decks we are looking at. So we want to send this to you so that you have an idea.

Then if you could just forward us some suggestions, that would be great. Now what I really liked about your report is you had infographics. You had all these other things. We didn’t do it when we first started. So I don’t know if there is a going back and doing an infographic, even though we kind of wondered about whether you can do an infographic with themes or something like that, kind of that have been discussed after a meeting has been done. But any suggestions that you could forward to us about penetrating in different networks would be very helpful.

MR. BOONE: I think you have hit on a lot of major points already. First of all, regards to the infographic. You have the opportunity to — I mean, the hardest part about infographics is developing the concept. So if you can find you a good developer of infographics and how you display that information, it totally works. The next thing I will say is the social media for us has made HDC appear much larger than what it actually is. So if you are not on that way, then it would behoove you to capitalize on that social media movement using LinkedIn, Twitter, and other things to really socialize this particular toolkit.

But the most important question I would have, and this is always something that we have to ask ourselves, but who is really the target audience and what do you really want them to do? Is it just to be made aware, or is it something to actually utilize? In either case, that would be the determining factor for how you approach it and how you approach your own dissemination plan for it. You know, you can develop it and just leave it out there.

What we really wanted to do with the report that we released earlier this year is have it be a thought leadership piece that would drive discussion and then we thought subsequently that we would have some working groups that would take a component of each of that report and really expound on it. Obviously time and resources and all those things played, were limiting factors, but I think you can take that same philosophy and approach to the dissemination of this report as well.

DR. MAYS: I think we are trying to get to a group that we are not in all the time, and that is to get this at that community organization level where they are the ones that are either requesting to use data and need to understand kind of the data stewardship issues or that they are mashing data up, because they want to utilize it in their town, city, et cetera, and so we want to get this in their hands.

So we had individuals from those communities who attended the meeting, and we have made sure to get it to them, but we wanted to kind of exponentially kind of do snowball to get it out beyond them. So that is one of our target audiences.

MS. JACKSON: This is one of the first products from the committee that is kind of asking for back, give and take, in terms of communication information. The committee’s previous products have all been the reports, as you asked, Chris; what is the target audience and what do you want them to do. We wanted to get the reports out to people’s hands, but this is the first time, and you can have some comment, Leslie, one of the first times that the committee is trying to develop something where they can get feedback as well, when the way it is developed it’s a static document, it’s a pdf, we don’t have access and ability to do that.

But I wanted to put in your back pocket in understanding that that was kind of the design, a visionary design, of what they really would like at the 21st century level where, like you said, Vickie, where people at the community level would take it, use it, put feedback in as well. We are just trying to figure out just where and how to do that.

DR. MAYS: See, I would say that we should have something on the web that every time they request it, it comes back and says tell me something about how you have used it.

DR. FRANCIS: Also, there may be as data availability and data analytic techniques change, and all of that is changing rapidly, there may be need for additional tools, for modified tools. You know, some of the kinds of things that the toolkit says like identify the responsible person, well, that’s not going to change, but some of the other things in the toolkit might easily need updating at some point.

DR. MAYS: I think in the marketing and dissemination, one of the useful things would be in — and I can do this and Chris may have some ideas — that is what to do on your data analytics in terms of what — and I don’t know in terms of the federal government, but what you can collect from the downloads. You have very different privacy rules. But you might be able to collect some things that would actually tell you who is getting it, but it’s not going to tell you much about what they are doing.

But there is a way. There are websites in which when a person requests something, you actually can ask them to tell you how they plan to use it before you give it to them, but I don’t know if we are allowed to do that.

DR. RIPPEN: Actually, usually we do studies for that, but the effectiveness of dissemination and effectiveness of tools, but with regards to the dissemination and if we are looking at communities, the three that I didn’t see when I quickly glanced on the slide was CDC; CDC has a major resource for communities that provides all kinds of toolkits. It’s big.

The other is big — Robert Wood Johnson Foundation, I mean, if you think about who they fund. They fund communities to implement all of these great things.

DR. MAYS: You can go and put an announcement to their LinkedIn group as a matter of fact.

DR. RIPPEN: Yeah, and I think that you could probably talk to them about how comfortable they might be, if they would be interested in using it as part of their package to communities, and you could ask to see if they could get feedback on whether it met their needs or not, because I think they also want to make sure that everyone is a good steward of data. So I think that is important. If you really want to get out in the state and local, you could go through the governors’ association with regards to, hey, here’s tools to help your mayors and whatever do X, Y, and Z. Could backfire, but if you want to hit the communities, that’s states to local mayors.

Then the other one, which goes to more kind of workforce development, if you think about who are the cadre of professionals that are going to be working in the communities to help activate them, is it the schools of public health? Is it the social works? Put it in their curriculum.

DR. MAYS: So I am going to suggest that in terms of RWJF that what you see is whether or not they would let them do a webinar for their website. Not the website, but a webinar for their group of — there is a name they have for them. It’s like the up and coming or the new professionals or something like that. Which would really penetrate, because many of them are working on community based participatory research kinds of things. So if we can do that, those — and it stays on the website. So a lot of people access that website.

That’s great. And then, Chris, if you have any suggestions for us as well, we will receive them with bells on our toes.

All right, folks, we are right on time to end this meeting. I know that it’s dark and rainy and people have travel needs. So I want to get us out of here, but before I do I want to say thank you. I feel like — I don’t know about you, Kenyon; you and I and Josh have been here a long time. I really feel like maybe I got out of the quicksand a little bit today. So we just have to keep this. We have Lily. So we are going to keep moving, but I really do appreciate the participation.

Greg, we are going to see if we are going to get to where you said if you were sitting here you wanted us to be. So we are going to work on that, and we are going to invite you to work with us. So we are going to try to be the doers and not the talkers.

So all right, everybody, travel safely and thank you very much. Bye, everybody online, and thank you for tolerating.

(The meeting adjourned at 5:00 p.m.)