[This Transcript is Unedited]

Department of Health and Human Services

National Committee on Vital and Health Statistics

Hearing on Minimum Data Standards for the

Measurement of Socioeconomic Status in Federal Health Surveys

March 9, 2012

National Center for Health Statistics
3311 Toledo Road
Hyattsville, MD 20782

Proceedings by:
CASET Associates, Ltd.
Fairfax, Virginia 22030
(703) 266-8402

TABLE OF CONTENTS

P R O C E E D I N G S (8:46 a.m.)

Agenda Item: Welcome

DR. GREEN: Good morning. Welcome back to the National Subcommittee meeting
on Population Health concerning the minimum data standards for the measurement
of social economic status in federal health surveys. This is day two of a very
interesting meeting. We’ll commence by introducing ourselves around the table
first, then we’ll go to people on the phone, and then audience. I’m Larry
Green, a member of the full committee, co-chair of the Population Subcommittee.
I have no conflicts.

DR. MAYS: I’m Vickie Mays, University of California Los Angeles. I’m a
member of the full committee, and I’m chairing the hearing. I have no
conflicts.

DR. QUEEN: Susan Queen from the Assistant Secretary for Planning and
Evaluation.

DR. BREEN: Nancy Breen, economist at the National Cancer Institute.

MS. JACKSON: Debbie Jackson, National Center for Health Statistics,
committee staff.

MS. GREENBERG: Marjorie Greenberg from the National Center for Health
Statistics, CDC, and Executive Secretary to the committee.

DR. SUAREZ: Walter Suarez with Kaiser Permanente, a member of the committee
and the co-chair of the Standards Committee and a member of the Population
Subcommittee.

DR. COOPER: Dr. Leslie, NIH. I have no conflicts.

DR. CORNELIUS: Dr. Llewellyn Cornelius, chair of the NCHS Board of
Scientific Counselors and faculty member, University of Maryland. I have no
conflicts.

MS. O’HARA: Amy O’Hara from the Census Bureau’s Center for Administrative
Records, Research, and Applications. No conflicts.

MS. PARKER: Jennifer Parker from NCHS’ Office of Analysis and Epidemiology.
No conflict.

MS. GREENBERG: Let me spare you all, because only the members can have
conflicts. The rest of us have conflicts also, but not for this purpose.

DR. BARON: Sherry Baron from the National Institute for Occupational Safety
and Health, CDC.

MS. JONES: Katherine Jones, CDC National Center for Health Statistics and
staff to the committee.

MS. COOPER: Nicole Cooper, staff to the committee.

MS. HOLMES: Julia Holmes, Division of Biostatistics, National Center for
Health Statistics.

DR. MADANS: Jennifer Madans, National Center for Health Statistics.

Agenda Item: Summary of Previous Day

DR. GREEN: The first item up this morning is for us to remind ourselves
about the purpose of our meeting. Dr. Mays is going to help us review the
results of yesterday and where we’re arriving this morning. Vickie and I
thought that we should start off restating our purposes. Mr. Jim Scanlon did a
nice job summarizing this for us at the beginning of the hearing yesterday
morning. There are really three foci for our work here.

The first question we’re really focused on is what is the state of the art
and the standards for collecting data to measure SES in federal surveys today.
Jim also reminded us to make a distinction between possibilities in standards
with standards being a higher bar. The second issue is what are the variables
that are being collected? The third is what opportunities exist to standardize
these variables across federal surveys? With that, I’m going to turn this over
to Vickie.

DR. MAYS: What I’m going to do is spend a bit of time talking about
yesterday just to try to lead us to today. We were really very fortunate in the
kind of stellar presentations that we had yesterday. Where we find ourselves is
to talk about data linkages and methodology today.

As background to yesterday, we started off talking about trying to get a
good sense of the definition of social economic status. In thinking about
social economic status, what our presenters were helping us to really focus on
is the components.

As all of you know who were with us yesterday — and if you’re just joining
us, I want to kind of recap just a bit — social economic status is an issue
that has been looked at. There is a significant amount of work, and there are
many ways in which to look at it. The discussion that we had yesterday was
looking at the issues of income, education, and occupation.

One of the issues that was raised in terms of thinking about our current
focus on social economic status, is whether or not we wanted to include the
issue of class. The issue of social class did come up. We worked a bit back and
forth about the American perspective on social class versus looking at social
class in other countries. We decided to discard social class as an actual area
that we’re probably going to focus on.

But instead, we will make sure that we pay attention to things like social
status, prestige. Going forward, that will actually be kind of one of the
litmus tests as we think about the issues of education, income, and occupation,
because in the literature on social economic status as well as social economic
position, what distinguishes it is really the issue of where it is that you sit
in society, and sitting in that place in society, its implications for health.
I think that we had some great presentations yesterday to help us make some
distinctions about that.

Moving forward, what we began to do was to look at the various components.
The first panel that we had, which we had stellar presenters on, Dr. Bauman and
Dr. Wong — what we heard was the ways in which education can stand as a proxy
for other things. It was like struggling between whether the focus is
educational achievement, educational attainment.

We were faced with listening to some of the issues around the quality of
education and how, for example, in a federal assessment when you have data
collection that says one has a certain grade level, whether that grade level is
really equivalent. Are there methods for being able to determine equivalency? I
think what we heard in the education panel is not only to be careful about our
definitions, but to be careful about what we pick. Should we pick and make a
suggestion about a standard? We should be very careful about making sure that
it’s something that’s going to serve us well.

We also heard kind of steps to move forward in terms of thinking about ways
in which to look at education that may add quality to the measurement of it so
that it is within the context of social economic status. Again, each time what
I want to do is make sure I bring us back to we’re not just measuring education
for the sake of education. We’re talking about these items as a part of a
construct, which is social economic status.

The next panel that we had, which was also equally as exciting as well, as
equally as challenging, was the panel on income. In thinking about income
relative to social economic status, it’s very complex. I think it’s probably
for us going to be something that we’re really going to struggle with.

We heard some very clear messages about continuous variables, about making
sure that the measurement that is collected is one that the user can actually
modify in ways in which it fits their particular needs. This was a panel, as
well, that raised issues about there are many different ways that income is
collected in the surveys. There are many reasons why, because the surveys vary
in what they’re using the income variable for. We also heard that we should
consider the poverty measure, the new approach to the measurement of poverty,
and look at that for the value of that particular approach.

Finally, the thing that I was also heartened about is that there were some
specific directions given to us in terms of people’s learned opinions about how
it is we should measure this in terms of recommending it, if we do, as a
standard, what the minimum would be.

Finally, our last panel was the occupation panel. I think we were very
excited also by the occupation panel in terms of understanding lots of changes
that have taken place. We heard a report about O*NET. We heard a report about
auto coding.

I think that going forth in terms of occupation, some of us thought that
this was an area in which we would really be struggling with what to do. But I
think, as we heard in the presentations, there are a lot of very exciting
things going on in terms of the way data is collected, the way data is being
coded to make the usage a bit better going forth.

I will stop here and invite my colleagues to also give any input in terms of
any additional comments that they want to make. I was trying to do just top
highlights as opposed to our next stage, which would be the deliberations.

MS. QUEEN: Susan Queen from ASPE. I just want to express my thanks to the
committee for taking this on. Keep in mind that the challenge for our surveys
in trying to implement any new standards, what we’re facing now with the
standards that were just adopted, which are perhaps much more straightforward
than anything related to socioeconomic status variables. Even implementing such
changes of those standards is definitely going to be a challenge.

It’s those kinds of things that we have to keep under consideration as we’re
moving forward with this, the survey purposes, survey costs, survey time, et
cetera. Just want to keep that in mind. I appreciate your taking this under
consideration.

DR. BREEN: I want to thank the committee chairs and the chair of the hearing
for taking this on and doing such a great job and finding such great speakers.
I know we’re here to talk about standardizing SES, but I think some of the good
news that we found yesterday was that there’s already a lot of standardization

There’s already a lot of good thinking about relatively minor tweaks, like
in the poverty measure there are ways to improve that that have now been
explored for about two decades. We may want to consider improving the poverty
measure, for example, and to urge the department either through the Data
Council or through some other mechanism to have a more ongoing dialogue among
the different surveys. Maybe that would ASPE.

I’m not sure where that would come out of, but that we consider that
possibility, because I think there’s an interest on the part of all the surveys
to be more consistent in the way the questions are asked. As we found, it
doesn’t look like the task is really too difficult. I would urge us to consider
initiating a dialogue like that or conversation like that that we can do
regularly in order to make and keep the surveys fairly well harmonized.

Also, I think what we learned was that there’s an enormous amount of
innovation that’s going on in these surveys all the time. Of course, we want to
encourage that and keep that going and keep the dialogue going there so that as
good ideas are brought up and implemented in one survey, they can be used in
other surveys.

DR. GREEN: Walter, are you there? I know that you were on the line
throughout part of the day yesterday. Do you have any comments that you’d like
to add?

DR. SUAREZ: Absolutely. Then comments of the presenters were superb. I add
my thanks to everyone. The comments I want to make actually come from my
perspective on the Standards Subcommittee. Generally speaking, we know that we
have sort of three different major sources of information about healthcare.

One is in the surveys. Of course, I understand that the focus of our hearing
isn’t that. The other two major sources are the medical record of patients and
the administrative data that is captured and maintained, for consumers,
enrollees, members, and patients in the healthcare system.

I just wanted to bring the perspective that I think it’s going to be
important and useful to create a mapping of the way we would be looking at
recommending standards for capturing this type of information, socioeconomic
status information, in population-based surveys.

Then map that with the standards that have been and are being adopted for
how to capture this same information in electronic health records, for example,
and in administrative data. It will be very useful to try to ensure that there
is some level of mapping and harmonization, if possible, across and between
those three major sources.

We all know with Meaningful Use, the Meaningful Use standards are adopted
for data that needs to be captured or that electronic health records might be
capable of capturing. It includes some of the elements that we are talking
about with socioeconomic status.

Then on the administrative side in the administrative world there are
standards that are being used to capture in enrollment forms and in enrollee
data and in the claims data and in the reporting of encounters. There are
standards that have been and are being adopted to capture socioeconomic status.

This, of course, doesn’t have to be done during the hearing, but in the
background after we complete the hearings and begin to work on the information
that we receive in testimonies, we can look at mapping this to the data and the
standards that we use to capture this information in electronic health records
and in administrative datasets.

DR. GREEN: Thank you, Walter. Dr. Cooper, I know you were on the line quite
a bit yesterday also. Do you have any comments?

DR. COOPER: No. I just want to applaud the committee in setting up this
panel and selecting the speakers that shared information with us. This is
unbelievable information and very enlightening. I’m very interested in terms of
how do we actually look at the issue in terms of education, given the fact that
requirements for completing a high school degree vary within states, as well as
across states, so how do we adjust for that as some type of a proxy measure for
moving forward with SES?

I definitely would like to thank Walter for bringing out the point of the
medical records, because if we move closer and closer to having increased
utilization of electronic medical records, that’s an unbelievable data source,
and we need to think now in terms of how to capture SES.

MS. JACKSON: From the staff perspective, I want to follow up with what Nancy
mentioned. That is the value of this kind of interchange, because that came up
when Larry asked last night, where does this kind of communication occur.

Of course, the Data Council is there, but that’s at a different level than
what’s really kind of going here where the kind of details from various types
of groups that generally would not talk to each other otherwise, at this level
of machination of looking at the elements for data collection and really
getting to the heart and soul, the undercurrent, that came through in the
income, for me, where the code is one thing, code for a position, but what is
actually happening when you change location from an assistant in the White
House to assistant to somewhere in the neighborhood for a family? That kind of
communication and interchange really kind of came through. I’m looking forward
to continuing.

MS. QUEEN: Can I follow up on Nancy’s comment. When the recently adopted
standards were developed or agreed upon, there was a lot of collaboration.
Under the auspices of the Data Council, there was a work group formed. It
involved representatives across HHS, OPDIVS, OMB, and Census Bureau. I would
expect that kind of continuing collaborative process.

DR. KAPLAN: Just one quick comment on Walter’s comment about electronic
medical records. There is a government effort to harmonize the psychosocial
components of the electronic medical record, largely by Russ Glasgow at the
National Cancer Institute. I think it would be nice to try to coordinate these
two efforts.

MS. GREENBERG: In that regard, this summer I was at what they called, summer
camp, but it wasn’t exactly the way I remembered summer camp. It was from the
HIT Policy and Standards Committee, that were part of the HITECH Act out of
ONC. They were vocabulary and clinical quality measures work groups that I was
participating in.

We had weekly teleconferences to try to identify all the different standard
vocabularies or code sets for a variety of measures that are being required in
Meaningful Use. They, again, looked to what the Data Council had done in these
first areas of primary language and race, ethnicity, et cetera.

Then when SES came up, they specifically deferred to what was going to come
out of the next round of investigation by the Data Council, which now the
subcommittee is helping to collaborate with. I think there is very much of an
awareness of where this is taking place, where these discussions are taking
place. Once this process is worked through, then they’ll pick that up again.

DR. SUAREZ: One specific example of real opportunities to not just try to
create a consistent harmonization of these standards, but begin to even expect
that some of this information is added to the capturing and the maintenance in
systems, on the occupational information side, recording of occupations.

I wanted to just enter into the record the report that was published back in
September of last year not too long ago by the Institute of Medicine that
focuses specifically on – this is the title of the report — Incorporating
Occupational Information in the Electronic Health Records.

There was a letter report format issued by the Institute of Medicine, and
the conclusion basically was there are basically at this point three important
data elements that are mature enough to be incorporated into EHR capabilities
requirements. Those were occupation code, industry code, and work relatedness.

I think we would have an opportunity here out of this hearing and in light
of, for example, the current review of Meaningful Use requirements for states
to consider a recommendation to the Secretary and to CMS and ONC regarding the
incorporation of this type of information into the EHR capabilities. So we move
the bar a notch forward with respect to capabilities for capturing SES in
electronic health records.

I just wanted to enter that on the record. I’d be happy to share the URL, if
people would like to see it, where you can see a copy of this report.

DR. MAYS: Walter, there is a going to be a response to you, but I would
suggest that you say what the URL is so the people online who are listening are
able also to access it. Dr. Baron from NIOSH is going to comment.

DR. BARON: Thank you very much for that comment. Actually, NIOSH was the one
that commissioned that report for the IOM. We have a very active work group on
electronic health records and would be very happy to collaborate with you.
There’s an effort now to try and get things ready for phase three of the ONC.
We would be very happy to work with you on that and have quite an active team
that’s been involved in this activity.

DR. SUAREZ: That is terrific. I won’t be able to repeat the URL because it’s
so long that it would take me 15 minutes.

DR. GREEN: I think we are ready to proceed to our next panel. We have an
hour and a half. Yesterday some of the most productive and useful parts of the
meeting were when our presenters started asking each other questions and
talking to each other. It was in the interaction. We are anticipating that
we’ll continue that model with these panels. We have an hour and a half for
presentations and also then for interaction and reactions to them. Vickie’s
going to lead us through these next few panels.

DR. MAYS: Thank you for being here. What we’re going to do now is move to a
panel in which we’re going to talk about data linkages. I think today is a day
that’s very important to us. Starting the day will be Jennifer Madans, who is
the Associate Director for Science here at the National Center for Health
Statistics.

Agenda Item: Panel: Data Linkages

DR. MADANS: Thank the committee for inviting us. I’m going to make some
brief opening remarks, and then I’m going to turn it over to my colleague
Jennifer Parker who leads the linkage program at NCHS. I think I bring the
historical perspective, but I realize that I probably do on everything because
I think I’ve been here now longer than most other people. I also wanted to
start with some reporting on behalf almost of OMB and the chief statistician
because some of this activity is coming out of her office.

But before I do that, I just wanted to comment on some of the discussion
before this on electronic medical records because I think it does have
something to do with data linkage. I think many of you know at NCHS we have a
long history of using health records as a source for our provider surveys.

One thing we have learned from that, as well as from actually the Vital
Statistics System, is that when you don’t have control over the source of the
information, you sometimes are not happy with what you get. On our surveys, as
we discussed yesterday about education and income and occupation, there are a
lot of challenges in collecting some of that information. There is a lot
missing on income. If people don’t describe what they do in their job in
sufficient detail, you can’t code it. Education’s a little bit easier, even
just talking about attainment.

But we have control over that. We write the questionnaires. We train the
interviewers. We have all kinds of fancy things on the computers that we use to
collect the information to probe and to get the information that we need.

When we don’t have that control, when someone else is the primary data
collector, you have much less control over the quality of that information.
When you’re talking about items that are perhaps not yes, no, check something,
or something that people may not be all that interested in providing in that
context, you have to be very concerned about the quality of that information.

I’m not saying that one shouldn’t pursue and that this isn’t a worthwhile
endeavor, but I feel duty-bound to say that one has to also be very concerned
about quality of that information. If we get occupation on the death
certificate, we’re not exactly sure how good it is. If we get educational
attainment on a birth certificate of a mother, we really have no way of really
showing the quality of that.

Other kinds of evaluations suggest that that information is not of the same
quality that you would get on a survey. We do have control over the data
collection process. Actually, it’s through linkage that we try to fix that,
that we have ways of combining data so that we can address some data quality
issues. Jennifer will talk about that a little bit, I believe.

Let me go back to OMB. This also kind of relates to other things that have
come up about just how much coordination is there in the statistical system.
What do we talk about? I don’t quite agree with Debbie. I think there’s a lot
of this conversation that goes on all the time, particularly in the area of
data linkage.

I think there’s been an acceptance for a very long time that data linkage
has a lot of advantages for the statistical system both from a point of view of
data quality, but also in terms of cost. This is not to say that data linkage
is free. It is not free. It has extensive costs, but relative to primary data
collection, they are less. I think the statistical system in its individual
parts and across the system would like to make its best use of all sources of
data as possible.

Many of the individual agencies, NCHS in particular, have been doing data
linkage for a long time. I think we started in the ’80s, at least linking to
mortality data. It has not been an easy process. I think that one of the first
agreements we tried to get — we weren’t even linking it, we just had to get an
agreement with the other agency — took us four years to do that, to just come
to an agreement.

At the statistical system level I think this has been recognized in that by
working together, we might be able to address some of those joint issues
better. There are committees. There is the Federal Committee on Statistical
Methodology — I mentioned that yesterday — that is specifically dealing with
administrative records and how to best use them.

Also, there is the Interagency Council on Statistical Policy, which is all
of the statistical agency heads get together. It’s chaired by Kathy Wallman and
the chief statistician. There are major activities involving the heads of the
agencies to try to develop practices and guidelines so that it is easier to get
access to the administrative data to bring the statistical community in itself
to other parts of the government, bring us together so it’s easier for us to
share information.

But then once we can share it, how do we do the linkages? It’s not exactly
straightforward how to do that. What do we do when we don’t have linkages? How
do we analyze the data? How do we make the data accessible to other users? It’s
no longer confidential when you do all this data.

There are a lot of things going on now that I think will improve data
linkage across the federal system. There will be a lot more consistency in how
we do it. We are happy about moving forward on that. NCHS is happy to be one of
the key agencies that’s been involved.

Coming back to the specific topic of this hearing in terms of the standards
and what one might do, there’s a lot you can do with linkage, but you have to
have something to link to. We like things to link to that are national
databases, that are consistent across geography, and that contain the whole
population that we’re interested in.

Mortality is a good one for us to link to. We actually have all of the
deaths, and we can link to them. But in terms of linking to a file that has SES
data on it that we could put on our surveys rather than ask about it, there’s
not a whole lot. There aren’t nice little inventories of everyone’s occupation
or everyone’s education.

There is information on income. We do have tax records. We have earnings
records. Those are very hard for us to get access to. There are a lot more
hoops to jump through to get that kind of information. One has to weigh the
cost of doing that with what you’re going to get.

I guess it’s been our opinion so far that that’s not worth it. First of all,
it takes a lot of time. We have to ask it anyway. I think we’ve looked more to
those kinds of linkages, if we could do them, as expanding on the information
that we would have on SES rather than using it instead of asking on the survey.
That may change in time, but right now for our data collection systems, the
population data collection systems, we’re probably going to have to ask about
these basic SES variables.

We did do a project, but I can’t remember how long ago it was. I tend to
tell people everything happened last year, but it must have been maybe five or
six years ago. We did a statistical link to CPS because CPS does get a lot more
detail in terms of income than HIS does. The idea was if we could do a
statistical match, not a real match, not an in-person match, that we could pull
that information to augment the survey. It was a complicated process, took a
fair amount of time. I think we have not done it again because it was so
time-intensive. But it is another possibility should the linkage become easier.

To get SES information about the individual people in our surveys, I don’t
see that happening in the near future. There are other kinds of linkages that
we can do that would add to that. We can do a lot of other kinds of linkages
through other data systems. I’m going to turn it over to Jennifer Parker to
talk about that a little bit.

DR. PARKER: I hope I don’t repeat too much of what you’re saying, but I just
want to emphasize that I’ve just got into the data production world, but I’ve
been a longtime data user. Typically we do rely heavily on the information
collected in the survey — it’s pretty high quality — to augment what we get
with the linked files.

Not only do we have more control over it, as Jennifer said, but if it’s
missing, we have developed wonderful imputation models for the Health Interview
Survey, so we can use all sorts of information collected on the survey and by
the design frame to even make our data better. Again, we have control over
that.

When we link to mortality, for example, many people have used the SES
information to look at disparities in mortality outcomes. It’s not just the SES
information; it’s the race and ethnicity information, which isn’t the topic
here, but because all these things interplay with each other, being able to
simultaneously look at both of these things in terms of mortality outcomes with
other admin records like the CMS data, Medicare utilization, and experiences
with Medicare program. Most of our data users who come to the Research Data
Center to use these linked files are very interested in the socioeconomic
status information on the surveys.

When we do these linkages, as Jennifer said, we like to have the entire
world, or at least the whole country. We need the whole universe of people who
are eligible. Deaths is a good example. We also have program participation. We
have the Medicaid population.

Not everybody is in Medicaid, but when we link to these sorts of sources, it
allows us to examine things, like using our information about SES, how these
things factor into program participation. The fact that program participation
changes over time and we have a single measure of income provides some analytic
challenges, but I think that’s part of the strength of our surveys, to be able
to uncover some of these things.

The other thing that linking to programs has allowed us to do in pilot
studies — we have a pilot study with Texas where we’re linking up to food
stamp information. We’re able to actually use that linkage to make some of our
questions to determine program participation better. I think that some of these
linkages aren’t going to be directly related to asking SES on the surveys, but
are able to give us a better picture of the socioeconomic status of the program
participants.

I’ll just say we do have another pilot project linking to a single state,
the Florida Cancer Registry data. We took one year of our Health Interview
Survey and linked to multiple years of the Florida Cancer Registry, not related
to SES. But I’d like just to point out that about a third of the people linked
in the Florida Cancer Registry were not in Florida during the survey. They were
in another state when we conducted the survey.

When you link to subsets of the country, for example, if you could get all
high school records from California, you’re not necessarily going to get the
people who were in the survey in California. You might get people who moved
there from Montana. That provides some analytic challenges, but it means that
what you’re getting isn’t necessarily what you think you’re getting. We’re
working on some of these analytic issues.

Finally, we haven’t really talked about the contextual information. We are
doing a little bit more of that in house, but I know that when of the biggest
uses of the Research Data Center is to attach contextual information to our
surveys.

We are in the process of coordinating some of the Geocodes in house. It’s
fairly straightforward. If we can use the census data at the census tract, at
the census block, at the county level, we can just merge to outside information
on the median income, that provides a way of augmenting the socioeconomic
status data in our surveys. There’s a huge literature on this, as most of you
are probably aware of.

From a statistical purpose from our agency, there are a lot of challenges in
deciding what the right unit of analysis would be, and still we have our
individual level of data. We know that there’s a lot of variation even within
these units. We would always defer back to our individual-level data, even if
we are augmenting with external population or area-level information. We’re
doing a lot of research in this research in this area too.

In short, I think our surveys have good information on the socioeconomic
status that we use for the record linkages and for the geographic linkages. I
don’t see us getting better data from the administrative records.

DR. MAYS: Tell us where you’re from and your position there.

DR. O’HARA: I’m from the US Census Bureau. I’m in charge of the
administrative records research section and the Center for Administrative
Records, Research, and Applications and the Research and Methodology
Directorate. The center I work in is abbreviated CARRA, which is handy because
it rhymes with my last name.

Within the center and during my time at Census, like Jennifer Parker, I’ve
been a data user. I’m going to give you some information about the record
linkage capabilities we have at Census and also echo the information that both
Jennifers have presented, that record linkages works well when you have data to
link to.

But that said, in my time at Census I’ve conducted record linkage with the
American Housing Survey, the American Community Survey, the CPS ASEC and the
CIP. Looking at those surveys, each one of those has offered unique challenges
whenever you need to validate the records and get them ready for record
linkage.

I know that my colleagues at NCHS have experienced this. You don’t always
have consent to link all records in a survey, and you don’t have the ability to
link all records in the survey. So there are data quality issues with the
endpoint data. For the topic of your hearing this is very challenging because
the best information that you’re likely to get on income, education, and
occupation will be attached to those survey data. You have to understand how
you’re going to address incomplete linkages that often result from missing
data.

One of the large projects that we’ve been conducting in the center at Census
is a match of administrative data to the 2010 Decennial Census results. To give
you an idea of the scale of missing data, 10 million of the census records
lacked name or data of birth information, which are our key identifiers that we
need to conduct record linkage. If you’re looking to conduct a person-level
match, those records are off the table right away.

But similar to what Jennifer has said, if you don’t need to do a person
match, if you can do an address match, whether it’s to the actual apartment
unit or to the building itself or to a broader area unit such as block or tract
or county, there are many opportunities to take data that can be assembled from
administrative sources that could be very useful for your purposes.

Back to the broader array of what we have at Census, we’ve conducted record
linkage projects to evaluate data quality. We do collect the income information
on all of those surveys, but we want to benchmark it against another source of
income data. We’ve matched to both the W-2s and the IRS 1040s. I believe Fritz
is going to be talking about this in a few minutes, so I won’t labor on that
topic.

We have also matched the surveys to the Medicaid data, to assisted housing
data, to FHA loan data, to understand whether the information that we’ve
collected in the survey seems to be capturing the information as indicated in
the administrative records data. The sort of match that I’m referring to is a
direct match. We’ve looked to find the same person in both files.

Other programs at the Census Bureau have used administrative records in
indirect applications. A good example of that is our smaller area estimates
branch. They conduct the SAPI program and the SAHIE program. These programs
take the survey data that we’ve assembled and augment it with administrative
records as predictors for the estimate of interest.

The direct versus indirect is key there to understand where the sources are
available. I believe for the SES you would be looking for more indirect and
relying on the survey data points that my other colleagues at Census, I’m sure,
discussed yesterday.

What Census can do, because I believe I was invited here to discuss what
Census could do in terms of data linkages, is we have some data at Census —
and as I mentioned, Fritz will discuss these linkages to the IRS data. Title
26, the IRS tax law, states that they are to provide extractive data to Census
for Title 13 benefits.

That means that Census can get tax data and use it in analyses, provided
they have a census benefit. That’s going to get pretty squirrelly when you try
to match it to an NCHS survey. It’s not going to meet that standard. Title 13
is the Census Act. It’s the Census Bureau laws that describe what we will do
and how we will do it.

But Census does have the capability to conduct linkages for incoming survey
data with various forms of identifiers. Ideally, we like to see Social Security
numbers and complete name and complete date of birth and complete address, but
in the past two years we’ve started acquiring commercial data, which offer
record linkage challenges because the quality of data is not the same as the
information collected in federal administrative records or the federal surveys.
Again, I can’t state enough the challenges that come from record linkage,
particularly involving data quality.

But should someone come to the bureau with two lists, one of them being an
HHS survey and another being some magical list of occupational data with person
identifiers, we could conduct that linkage. Through the Census Bureau’s
Research Data Center network, it is feasible that we could host access for
individuals to come and use those data. We have the capabilities, but I just
wanted to restate the lack of obvious dataset candidates to link to the various
surveys. There’s a lot of promise in these datasets. You just have to have the
right inputs.

DR. MAYS: Thank you. Fritz Scheuren is Vice President of Statistics and
Methodology at NORC.

DR. MADANS: I know you all know this, but sometimes we forget. The US is
unlike other statistical systems. In Statistics Canada they’re all together,
and they have a statistics act that basically says everyone has to give them
information. That’s in the law. The US is not like that. Every agency has their
own legislation, and there’s very little of this you must give them these
things because it’s important for the statistical system.

The Census Bureau’s legislation Title 13 and its relationship to the IRS
data, does not include NCHS. We do not have that relationship with IRS. That’s
why I said we cannot get that data. If we get certain kinds of approvals and we
work things out, it would take a long time. But our authorizing legislations
are very different in this case in what they allow us to do and also what it
makes other people do.

One of the things that the ICSP is looking at is ways to work within the
system to maximize. We were talking yesterday about we ask a lot of health
questions and a lot of behavior questions on our surveys, and people answer
them, but ask them income and forget it. IRS data is the most sensitive data.
To get access to that, that is tightly controlled. I think that’s because the
population feels that way.

There is a lot of work being done now on trying to understand better what
the population thinks we’re doing, what they want us to do, through some trust
surveys. But I think that you need to kind of keep that in mind about the
highly sensitive nature of tax data and the fact that our legislation does not
kind of make it easy for us to get them.

DR. O’HARA: The Title 13, our statute does allow us to ask because it is
written in our statute that we are to improve data quality and reduce
respondent burden through the use of administrative records data. We have the
ability to seek them from federal, from state, from county, and from private
entities, including individuals. We can ask. That doesn’t mean people have to
agree and share their data with us.

DR. SCHEUREN: I used to be head of statistics at the IRS before I went to
work for Amy, which is one of the people I work for, by the way. I used to be
at the center. They had a scientific community thing. It was really nice. It
was in this room. There’s a lot about linkage that I know, and I’ll tell all of
it to you, but you’re not giving me too much time. I do know a little more than
17 minutes.

I have some ideas that I’ve organized. You have the handout. I’m going to
use it line by line. This is a huge topic. You’ve already hear from three
people, and you must have a sense that you’ve looked at only one part of the
Moon, the only part that you can see. The rest of the Moon you haven’t seen. If
I believe that’s right, it’s a little over a third of the Moon that you can see
from the Earth. Until 1968, we hadn’t seen the other side.

I’m going to guess that the main link to you is something I say in the
second or third slide about no free lunch, this hugely difficult thing. Can you
earmark a specific focus activity and get something achieved? Absolutely. There
are some pets I will bring out.

One I might particularly mention now that I will come back and echo is I
work on Native American issues, Aborigines, as they are called in Australia. I
was just in Australia two weeks ago. If you have a very small population — and
the Aboriginal population in the United States varies from 2-4 million — and
you want to look at health issues, if we have numerator and denominator
problems and we have misclassification problems both in numerator and
denominator and misclassifications are not correlated, which they’re not, then
you have a serious problem.

This actually extends to other minority groups. You just heard about the
geographic problem earlier here about how when you go to match in one place and
you find people are not in that place — that’s really an issue to come back
to. I’ve been working on Native American issues for 11 years in the US. I have
a book coming out on it. I know nothing compared to what I need to know. You
heard the business about the Moon. The percent I’ve seen is a lot smaller than
that, even though I’m part Seneca, by the way, which is one of the Iroquois
Nations and worked with George Washington. First they were allies, and then
they were enemies, but they were always friends.

I’m going to talk a little bit about context weaving and a little about
paradigm. Context is the context we’re doing the linkage in and paradigm is how
you do linkage. It’s very cartoony, very sketchy, but it’s a way to open us up.
Then I’m going to give an illustration.

Amy has already mentioned that I’m going to be looking at matches to the CPS
that Social Security requested. They browbeat the IRS. You have to do that. I
used to be in Social Security too. Your earnings data is shared jointly between
the IRS and Social Security, and so Social Security purposes also bear here.

These are Social-Security-driven purposes because they’re part of HHS.
That’s an access window for you, but be careful. Amy has already warned you. On
the other hand, the HHS people at Social Security are part of your team. You
should avail yourselves of them. They really are good. They were even good when
I was there.

I’ll talk about the purpose of linkages and administrative data. That has
already been done. There’s a nuancing that you need to do with administrative
data, particularly in the linkage world. If the administrative data is being
used in linkage, like the W-2s are matched to the 1040s to check whether the
earnings data on the 1040s, not on the CPS, is okay, then they pay a lot of
attention to get that linkage right. But if you’re interested in a linkage just
to CPS or NCHS, you’re going to have a different problem.

You really need to move from a focus on data to a focus on information.
There is a big difference because you can get the information from weak linkage
and get it pretty well in the sense you get a decent point estimate and you get
a variance that you can measure.

You’d like to have an even better point estimate in a very narrow variance.
That’s not going to happen. You can’t afford it. But you can do the other if
you’re careful. Some work that I’ve done with Bill Winker(?) and others, and
some work that you’ll hear about in a few minutes illustrates that.

There’s this content versus coverage. Amy’s goal is coverage. She wants to
be able to find out who is in the census that’s in the administrative records
and vice versa and who’s in now commercial records and vice versa. That’s
because the goal of the decennial and the goal of the census is to count
everybody. A lot of other goals — and probably not your goal — are more on
the content side, not the coverage side. Both are needed.

For an example I’m going to use a project that Amy had. Amy worked out an
arrangement with the Treasury Department to look at EITC, earned income tax
credit, and to see to what extent it was being employed by everybody. That was
a joint project, but for a specific treasury purpose very different from the
decennial purpose. She knows about that. She can talk about that if you want to
ask her.

I used to work on EITC when I was at Treasury, and we had a wonderful
response rate. We had interviewers that were packing, if you know what that
means. They had something right here. We had a very high response rate. I have
never had as good a response rate in any other survey. I don’t recommend that
to people, but I do want to tell you that that was what was happening.

Let’s look at some externalities. One of the things that you do when you do
linkage is to look at the editing and data quality problems. There’s a cost of
editing and a whole lot of issues. Susan, did I go off the reservation on that
one?

DR. MAYS: No. We were just discussing that we just wanted to make sure
you’re aware that you’re on a public record, that you’re being recorded.

DR. SCHEUREN: I don’t have any problem with that. It’s a true story. It was
a marvelous process because the people at the IRS don’t really want to do EITC.
They see their job as collecting taxes, not benefits. I need to send some more
of them up to Canada because in Canada they collect benefits and taxes at the
same time. The Canadians are a wonderful neighbor for us.

We really need to learn a lot more, as I recommend you do, too, from the
Canadians. They have some really wonderful record linkage ideas. We’re going to
be talking about Fellegi-Sunter, that’s a Canadian paper, in a few minutes. Go
to Canada. They’ll be amazed that you’re there. Listen carefully. They know
their stuff. They’re very good, arguably the best or the second-best
statistical agency in the world. I was in Australia, and I think Australia may
have edged ahead of Satscan(?). I’m on the record here. Don’t kill me, Ivan.

The cost of editing is an issue. Confidentiality is a double-way issue. One
of the problems with confidentiality is that we now live in a data-dense world.
We used to live in a data-sparse world. The surveys we did and still do were
designed in a data-sparse world. They’re now being augmented with data from the
administrative records as we move towards a data-dense world. We’re not there
with a data-dense world yet.

One of the reasons within the statistical system in the government is
because of difficulties across agencies, but that is being addressed. You’ve
heard it being addressed. It’s being addressed very cautiously. I guess they
knew they were on the public record and I didn’t, but what they’re saying is
right.

What they don’t give you a sense of is a vector. There is movement. It is
moving, but it is hard, but it is moving. The thing is that it is moving, the
Heisenberg principle, you know exactly where you are or exactly at what speed.
If you want to know both at the same time, you have to be a little bit
flexible. Jennifer did a very nice job of focusing on where we are. They are
doing a good job. Census is doing a good job. It’s hard to do, especially when
you’re dealing with Title 26.

There’s a complexity issue. We think when you match them together that you
can treat them as if you collected them at the same time. You know that’s not
true. You have all kinds of context issues that change. If you ask a question
in a survey and the next question comes from the administrator source, it is
going to be answered differently than if you’d asked it in the survey for
obvious reasons. That’s a problem, but not a bad problem. Explainability and
comparability are issues too.

This is really hard to explain. You’ve listened to some of the experts
already. They did a good job, but it’s hard to explain. Here’s the most
important bullet I have. This is very hard. Don’t try to get it done fast. If
you are like this — and there are people like this who believe that the way to
manage something is to give people a date when it’s due — don’t do that. Give
them things to do and let them tell you when they can get them done, and then
hold them accountable to what they tell you they can do. Don’t start the other
way. It’s just much too hard.

Let’s talk about the paradigm itself, the linkage paradigm. You’re going to
match two or more records together from different sources and you want to do it
uniquely, as distinct from doing it statistically. There’s a whole literature
about statistical matching, a very fine book by three Italians on statistical
matching called Statistical Matching. I did a book review in JASA a few years
ago. It’s typically called data fusion in Europe. I like the data fusion title
better.

Anyway, you try to match two records uniquely, so it really is the same
person or the same entity or the same unit. At some levels you’ve already heard
— the geographic levels, high enough geographic levels — the matching is
almost unique, at lower levels less so, and at the individual level you have
all kinds of issues around data quality of the match itself because usually
unless it’s an administrative source being matched to another administrative
source that was designed to be matched, then you’re going to have these
problems of it didn’t quite match. Amy’s story about 10 million people without
names — wow, that’s a pretty big story.

What about the themes here? Obviously Fellegi-Sunter,, a very famous paper
in 1967, JASA, it’s a mantra. Everybody says it. None of the things that are in
Fellegi-Sunter are we doing anymore, but the idea of having two limits, an
upper and a lower bound, still exists. I’ll be coming to that in a moment. It’s
a mantra. We should keep that focus, but the tools we’re using now are very
different.

Computer technology is very different. The role of paradata, which is a
recently coined word, which is how we got the data that we have, the process
itself, who the interviewer was, things like that, those turn out to be very
important. When we were doing the earlier work in CPS before the current
method, the earlier work was based on asking the Social Security number in the
CPS, but we aren’t doing that now, it mattered very much who the interviewer
was, because some interviewers would not ask that question. They just wouldn’t
ask it. Of course, if you look at the data, you could tell that.

We’re going to do a little bit about imputation because that’s my goal and
the work I’m doing and how to validate results and how to complete the
inference. I will not be able to complete my discussion of the inference today.

There are essentially three bounds. There are the true links, the upper
bound, which is really the thing you have to focus on the most; the non-links,
which is lower bound; and below the non-links you should start spending money.
That’s really the way I look at it.

Then there’s the middle ground, where there are links in there, but you
can’t tell. At the data level you can’t separate the true link from the true
non-link. The way you deal with that is by using methods that Bill Winker and I
have used and others have used. The kinds that were used in the old days.

Let’s look at this picture. These are the log of frequencies, because the
non-links are huge relative to the links, and we have to push that down. There
are two bounds in here. There’s the lower bound right here and the upper bound.

The ones above the upper bound are ones that we really believe are links.
They’re probably not all links, but there are so few non-links up there that
you have to model them, which we’ve done, to estimate, because you can’t afford
to study them electronically or any other way.

The middle group are the ones which used to be sent for clerical review.
I’ve done some studies with clerical review not just in this context, and
clerks aren’t always right, because they’re human beings, but they’re pretty
good. What they’re really doing is looking for other variables that weren’t
involved in the match itself that was used in the link. That’s what Bill and I
were doing and that’s what can be done now. The word we’re in is a data-dense
world, and we have lots of other variables that are computerized that we can
use, but that’s a new challenge and still open.

Let’s talk a little bit more about bounds. The bounds depend on the match
variables themselves. You really want to look at improving the match variables
if you want to do this well in whatever context you’re in. However, even if you
improve them, you’re still going to have a problem with the linkage
probabilities. They are model-based. What you can do is look at the robustness
through the models. But if you want to really nail that sucker down, get
another job.

All the variables on the files can be used for the inference, not just what
are typically called the match variables. That’s a paradigm shift for the
linkers. Fundamentally we think of, in a Deming world, you have a different
bunch of processes, someone finishes one process, they throw it over the wall
to the next person who does the next thing, and so forth all the way to the
end.

Get rid of that idea. Look at it as a total system. You have to bring the
linkage into the analysis itself. You have to bring the analysis goals
backwards into the linkage. That the linkages are done ahead of in time of the
analysis should not be a barrier to you.

I’m going to say just a very little bit about this. I’ve said it already.
The one thing that I didn’t talk about is that we’re now linking not just two
systems or two sources; we’re linking multiple systems. That can mean that the
lower bounds are going to be different. The upper bound is set by the user. I
want this much quality.

In the context that Amy’s staff is in the upper bound is what the focus is.
There are many records being matched, and it’s the upper bound that has to be
done well enough. It’s so well done that you can’t estimate in a normal way
what the errors are above that bound. You have to simulate it and you have to
see to what extent the simulation results change the final results.

But the low bounds are different because the quality of each record system
depends on the source. The sources are not designed for your purpose, except if
the sources are within an administrative system.

Let me talk about paradata. When I first started to learn about paradata —
I did a paper a few years ago on this — I was thinking that the linkage
process already had a lot of paradata, and we should use the linkage process as
a model to get better at what we were doing for surveys. But then I got into it
a little bit more and I realized the survey people have already moved beyond
the linkage people, and we need to turn it around and look at the linkage
process and do better at the paradata.

Some of the work that I’m doing with the CPS is focused on using the
paradata that’s on the CPS to aid the linkage. That’s very important. If I’m
doing the next linkage project, I’m going to focus a lot more on the paradata,
the context in which the information is being obtained, from all the sources
being used.

One thing you could do for that middle period between the true links and the
true non-links is to model that uncertainty. We’re recommending that you use
multiple imputation. I’ll say that in a minute. You have to move away from
counts and towards estimates. You cannot expect the linkage to be good enough
unless you’ve done it on purpose and paid for it. There is a big difference in
the context of using statistical methods for matching to administrative
sources.

Let me say a little bit about imputation. I’ve already mentioned multiple
imputations to you. This is an idea of Jon Rubin’s to handle the uncertainty in
the middle. You not only do a better job that way, but you get a measure of the
quality of the job you’ve done. That’s the great advantage of multiple
imputation. You get a distribution, not a single-point estimate or vector of
estimates.

I recommend this highly, and I recommend you begin doing it and we begin
doing it. We are in CPS doing it. We learned from that what we needed to do to
do a better job of collecting the paradata, because if you get into surveys,
you’ll realize that not everybody is collecting the paradata the same way
within any survey — CPS, NHIS. It’s not being done the same way. There are
regional differences, for example. There are differences with interviewers and
supervisors. All of those need to be looked at if we’re going to use them at
the inference stage.

Let’s talk about validating results. I think you need to use small samples
to ground-check the data. Even though you have computerized everything, that’s
not good enough. You need to use this idea of Kaisen. This is a Japanese word
which is usually translated to continuous improvement. But if you focus on what
the Japanese are really doing, you keep this word because it’s a cultural
issue. It’s not a technology issue. We need to have that built into our
culture.

If you turn out to have worked as many years and in as many different
agencies as I have in the United States, you know it’s there. It’s just not
high enough up on the list of priorities, because we are competing with the
need to be comparable as well. That’s a challenge. Both are needed. I would
recommend you go to Canada and benchmark what’s being done internationally. I
don’t recommend you go to Australia because it’s so far away. I just came back.
I’m glad I’m back.

I mentioned Rubin’s multiple imputation. There’s a wonderful book by Bishop,
Fienberg, and Holland called Discrete Multivariate Analysis, 1975, that has a
great chapter on linkage. The chapter on linkage misses two points.

There’s nothing about the fact that you’re not going to get a perfect match.
That’s kind of big. There’s nothing about calculating the variances. That’s
kind of big too. But it’s a wonderful idea. The hardest part is to explain what
you’re doing in a way that allows the other persons involved to actually find
out whether you’ve been doing it, for their purposes, well enough.

We’re going to look at matching CPS to the DER. That’s the Ernst(?) system
at Social Security. It’s coming under Title 26, but because Social Security
wanted to do this, HHS asked about it, it’s being done. Our original goals were
to do this to look at robustness of the CPS poverty estimates, the changes in
imputation rates.

The imputation rates have changed enormously from March 1962. I’m sure you
all know this. That was the first year that CPS did imputation for its income.
That was the year that Mollie Orshansky used to begin the poverty estimates.
Mollie was a friend of mine. She’s gone now.

Mollie’s first report was Children of the Poor, which came out in 1964. We
are going to — and I’m working with a bunch of people who worked with Susan at
Census — do a 50th anniversary series around that. A lot of work
has already been done. You don’t start on the year; you start way before, and a
lot has been done.

The relationship between CPS and DER is very important because imputation
matters. Imputation matters a lot more than it did when Mollie started in 1962.
By the way, interestingly enough, although I can’t get historically the
context, the matching to the CPS also started in October 1962, again asking the
Social Security number question on what used to be called the control card in
CPS.

They don’t ask the Social Security question anymore in CPS, although they
still do in HIS. In HIS they have a much better way of convincing people that
it’s needed. I think that’s the main reason it continues, because it really is
a very important variable to have. But you can do pretty well without it, and
if you are willing to accept a somewhat wider confidence interval, what’s being
done at Census works fine.

I did mention proxy there. We’ll talk about that, but I’m going to show some
pictures in a minute. Another little footnote here is we’re looking at two
measures of earnings, one from DER and one from CPS, that have been matched. In
this context they were matched using the system at the Census Bureau because we
don’t have Social Security number. They are intervals of $10,000. We’re looking
at the agreement between when they’re in the same class.

Here’s a nice little picture. This is a histogram. This is the histogram
which looks at the people that agree and people that disagree, one below,
smaller class, one above, smaller class. The rest of these are all like this.
I’m just going to use this one.

You’ll notice, which you don’t expect, that the CPS data is bigger than in
the Social Security. That’s probably definitional. That’s very interesting
because we’ve been saying forever that there’s underreporting in the surveys.
I’m sure there is, but there are a lot of other things going on when you’re
matching surveys and administrative data, definitional.

One of the issues here has to do with some jobs are not under Social
Security. I used to be a newspaper boy. My sister used to be a babysitter. I
never was covered under Social Security at my newspaper job and she never was
covered under her babysitter job. Some jobs just aren’t covered, even though
maybe they should be covered. The rest of these are very similar.

This is the no imputes. This is what you want, very high agreement rate, but
look at what happens to the imputes — a lot more variability. That’s a
problem. Interestingly enough, when you’re looking at poverty — and Joan
Jurich(?) is the driver of this — there’s enough cancellation.

The poverty estimates that we’ve been producing for nearly 50 years are
robust to lots of data problems that exist in the survey. That’s a very
important policy finding. I would not have believed that. In fact, when I
started out, I didn’t believe it. I thought the opposite, and I kept pushing to
find out if, in fact, I was right. Eventually, even I gave up. There are
problems that have to do with misclassification, but even those problems are
small.

This is just how we did it. You have all of that. Let’s ask questions or not
ask questions. Thank you very much.

DR. MAYS: We’re going to take questions of everyone. One of the things I
just want to start with to say is that I’m very appreciative of this
presentation. It’s just incredible. I really learned quite a bit. It is a lot
harder than I thought, so I have a great appreciation. Let’s start with
questions.

DR. KAPLAN: I am curious about your thinking about this issue of tolerable
range of error. Sometimes in the census assigning congressional seats precision
is very important, but in epidemiology when we’re just trying to get a sense of
relationships and we’re in a hurry, how do you address those sorts of issues?
What’s the thinking about how much error is tolerable for social science
research around public health?

DR. MADANS: I guess we often talk about fit-for-use. The question really
relates to what it is you’re going to do with the data, as you implied. Some of
the things that we collect for some of the analyses we probably could tolerate
a lot more error, and for others we can’t. The problem is we’re using the same
data collection system for both.

If you look at it from an NCHS point of view — and I think this would apply
to Census — we are kind of monitoring the nation’s health, so a lot of focus
of what we do is on this monitoring function. Have things changed? What are
monitoring now? We’re monitoring a lot of things about the healthcare system.

Small changes are important, especially in a short period of time.
Everybody’s timeframe has really shrunk. In the old days we used to put data
out years after it was collected. Now where is last month’s data? There has
been a big push for us to get more stable estimates and more valid estimates so
that the policymakers really do know what’s happening across time and things
like insurance rates and characteristics of the uninsured population. For those
we can’t tolerate as much error, whereas if you’re doing more of kind of a
multivariate, looking at various relationships, maybe you can.

There are some things that you have to do, just kind of the infrastructure
costs, to make sure that the things you need a high level of quality you’re
getting. Then for some of the others you can kind of let up a little. That’s
why we do have different, perhaps, quality requirements for different items on
the survey. But in general, since we don’t know what the information is going
to be used for, really trying to maximize the quality all over.

I think as a statistical system, we feel some responsibility for putting out
information that those who are not very familiar with the data collection will
accept as valid. A lot of the information we’re putting out is not to an
academic community where they can kind of look and say I know there’s a lot of
error in this income stuff and I’ll take that into account, but a very
different kind of audience.

There’s the accountability issue, the transparency issue, and just the
credibility issue. I think that tends to push us towards having less tolerance
for error than we might have if we were just providing data for the research
community. We do that as well, but it’s almost a byproduct of the other things
that we’re doing.

DR. O’HARA: Jennifer pointed out that NCHS wants to monitor what’s going on.
I think Census is similar, but we really want to measure what’s going on. To
echo what she said, fit-for-use is essential. You must know what you’re trying
to measure or what you’re trying to monitor.

In an application where we’re conducting record linkage, to try to
understand the characteristics of persons who appear to be eligible for a
program, but are not participating — we would like to know that we have the
same person in list one and list two.

If you’re looking at the characteristics of neighborhoods or housing
structures in areas experiencing foreclosure, again, it’s the unit of analysis.
If I’m matching at the address rather than the person, it has to do with what
questions I’m attempting to address.

Even in the algorithms that Fritz was describing, the data that you’re
putting into that match and trying to say is it the same person or is it nearly
the same person — our tolerance for that has to do with the eventual question
that we’re trying to address.

DR. SCHEUREN: There is a tendency — and I think rightly so, and you just
heard it said — to over-engineer things. If you want to build a bridge, it
better be good. It better be able to last a long time, and it better be able to
be such that anyone can get across it. If you’re in a jungle and you have a
single line, only an expert’s going to cross over it. But if you’re talking
about getting the semis across as well as ordinary people across —

On the other hand, my focus is on the conference interval. You’re the user.
You’re the client. You’re the one who has ultimately the resources to do this.
I’m not an employee of the Census Bureau. I work for the University of Chicago,
NORC. They don’t have enough resources, either one of them, to do the job that
you want at the conference interval that you want, which is really tight.

You won’t get them that by getting them to be smarter and faster. They’re
already doing a really good job, by the way. I’ve done this international
comparison. I’ve been in Canada a lot. I just came back from Australia. They’re
already doing nearly best-in-class, but they don’t have enough resources to do
it. This is for the record, too, I realize. They don’t have resources to do it.

Some of the problems we have, like with HIPAA — I was at a conference on
the Hill this week, and there are a lot of issues. We need to think about as a
total system, each user has to be cognizant of the other users so that the
resources that are needed get aimed relative. There were some things that are
fit for some uses, but not all uses. Those things are the things that should be
at the data centers for the academics. What should be published should be such
that anyone can use it.

DR. QUEEN: Fritz, you mentioned using paradata to approve record linkage.
How do you do that and what paradata do you use?

DR. SCHEUREN: What we are doing now is looking at the information the Census
Bureau asks already, not the interviewer name, but which records were done by
which interviewers. That turns out to matter a lot. There are regional
differences. Some of those are confounded with the fact that there are real
differences in the society in different regions, but some of them are
organizational issues.

The thing that I’m driving most for is this proxy respondents. I don’t have
those charts right now, but you saw those beautiful histograms. That comes from
something you’ll see at the end of the handout that I recommend you look at. It
really gives you a way of seeing what’s going on in terms of relationships.

But when we look at proxy respondents, we’re going to find that the proxy
respondents fall between the high level of self-reporting and the lower level
of the imputed data. There’s a lot of variability in them. They’re a form of
imputation, in fact. They’re imputation by the people in the household.

Sometimes my friends at the Census Bureau will tell you that you can’t say
it that way. They’ll say you have to say this is the self-reported person and
this is everybody else, because the other persons could be in the household at
the same time during the interview. You don’t know that.

DR. KAPLAN: Do you do much in the way of what you might call planned
linkages? For example, CHIS is thinking about linking up with other activities
where they say can you provide consent to your electronic medical record. I’m
just wondering with income information, could you ask people with consent can
you do a credit check?

DR. PARKER: We do have an informed consent process. The survey respondents
are asked if we can link up their data for statistical and research purposes.
We don’t say we’re going to link to CMS specifically, but we do ask about
medical records. We don’t ask about the IRS and income data. We’re not allowed
to do that, income and credit histories and things like that. There has to be a
case that directly relates to health, and I think you can make a case that some
of these things do, but it has to be pretty close to that.

DR. MADANS: There is one of these committees at SCSM that is looking at the
whole informed consent issues. That’s why Jennifer and Amy said that there are
a bunch of our records we cannot link because we were not given approval to
link. There’s an actual question that says can I link. Then how do you write
that informed consent so that it’s informative but not constrictive? Because we
don’t know what records are going to be available later. We work on that
wording. We would like to be able to link to records as they kind of come up,
as new databases come up.

My understanding with IRS is their requirement is not just that we say they
said it’s okay for you to get this. You have to have written consent. The
consent has to be with a certain amount of time, so after three months it’s not
good anymore. I think there are even some issues about how long you can keep
the data.

This is where the two agencies, the IRS and us, are not in sync on this
linkage process, which is very different than the relationship they might have
with the Census Bureau, because of our different authorizing legislation. Yes,
we do exactly that, but we cannot link to any record. Some of the record
providers have their own requirements about that linkage.

DR. KAPLAN: This is maybe pushing it to the extreme, but is there anything
in the statute or in the human subjects regulations that would prohibit you
from paying people to consent?

DR. MADANS: We prefer not to refer to it as paying. We do offer incentives
in some of our surveys. We do a survey of immunization where we have to go to
the immunization provider, so there has to be a consent, but it’s a different
kind of process. We do have incentives.

As part of the OMB process and the IRB process you have to have that kind of
things approved. The IRB is concerned about coercion. If you pay too much
money, then a certain part of the population is going to agree to do it even
though they don’t want to do it, even though we don’t quite pay that much.

Then from the OMB process, this is part of your civic duty. They are very
wary of allowing incentives, although there was an entire conference on
incentives and how they should work. Usually they will require that there is
some experiment that shows — and they’re mostly interested in response rates.

We’ve never actually tried incentives just to get approval for linkage. I
think there is a feeling about the linkage that kind of gets close to a Big
Brother kind of thing, that you have to be very sensitive to invasion of
privacy. Unlike the ACS you heard about yesterday, which is a mandatory survey,
all of our surveys are voluntary.

If we present this data collection in a way that really turns off the
respondent, then we’ve kind of made it worse in terms of the data quality.
Again, you have to weigh how important is that linked data versus how much of a
negative effect it is going to have. That is kind of constant evaluation that
you’re going through all the time, which is also affected by what’s happening
in the environment.

Anytime there’s a breach on any kind of federal system like what happened
with VA, that has an effect on our relationship with our respondents. It’s a
fluid situation. We may do something one year and the IRB best practices change
the next year and you have to change. I assume it’ll be different in five.

DR. KAPLAN: In the commercial world there are linkages going on all over the
place. Every time you swipe your card at Safeway you’re actually getting linked
to all sorts of databases. They know who you are.

DR. MAYS: I’m going to turn to questions here at the table. I know I have
some and you have some.

DR. GREEN: I have three totally unrelated questions. One of them is about
this issue that we went by very early on in the morning about whether or not at
a federal level there is the location and place, what I’ve heard you call the
statistical community, which I suspect is understood by you guys better than it
is by me. I think you know who each other are. What is the adequacy of the
current situation in terms of bringing the statistical community together to
tackle hard problems? Is it ready to go? Is it highly functional? Is it working
well? Is it sort of working all right? Come clean here. What’s the deal?

DR. MADANS: I will talk from the federal system. I’m sure Fritz will have an
outsider’s perspective. There’s an OMB statement that identifies the agencies
that are part of the federal statistical system. There are 12 that are the main
agencies. We have a lovely little chart that shows all of them on a star with
NCHS at the top. These are kind of the big 12. Then there are a bunch of other
agencies that have statistical functions, but they’re not considered a
statistical agency.

NCHS is a statistical agency, Census, BLS, CS. There are these 12 of them.
Their primary function is data collection. Even though they’re embedded in the
departments, they’re governed by principles and practices of federal
statistical systems. That’s something that comes out of SIMSTAT(?). It’s a
book. We can give you copies if you like. There are also international
principles that we also have to kind of go by. There are directives that come
out of OMB that really control what we can do as a statistical agency versus
some other programmatic or policy agency in the government.

The heads of those 12 agencies meet monthly as part of this Interagency
Council on Statistical Policy. The focus of that is in OIRA, the Office of
Information and Regulatory Affairs. It’s in OMB on the management side. It’s
headed by the chief statistician of the United States. There is this ongoing
group that meets and deals with issues of the federal statistical system, and
they meet every month. They do a wide range of things. They’re looking for
things that crosscut.

There’s this other group that I mentioned yesterday and today called the
Federal Committee on Statistical Methodology. It’s a group that is chaired by
and organized by this group in OMB. It has representatives from agencies, but
you don’t go as an agency representative. The members are selected because of
their contributions to the statistical system, something like that.

They meet quarterly, but they have work groups. If you look at their site,
they do a lot of white papers, provide standards, guidance. This is the group
that put out the standards for statistical surveys. They historically have this
long set of white papers and guidelines. There’s one now on cognitive
interviewing that just started. There’s one on the administrative data. Even
though that group only meets quarterly, there’s lots going on.

Yes, there’s a lot of work. There’s a lot of coordination. There are a lot
of things going on. On the other side of the coin, we are not like Statistics
Canada. We do not have one authorizing legislation; we have 12. They are not
often in sync.

The reason that the US statistical system is the way that it is is that
there was more interest in having the agencies embedded in the departments so
they would be closer to providing the information needed by those departments
to do policy and program development and evaluation. If we were all in
Statistics USA, we would maybe have a lot of things that are much easier to do
and we’d have a lot more coordination and a lot more consistency, but we would
be separated from the needs of HHS or for Labor.

We meet with the Assistant Secretary for Planning and Evaluation and other
parts of the department all the time. There’s the Data Council, but there’s
also a lot of back and forth. What do you need, Madam Assistant Secretary, to
monitor changes in the healthcare system? There is a big connection there.

That means we have our own authorizing legislation and we also have our own
requirements to our departments. Sometimes those requirements to our
departments are in conflict with our desire to be a more cohesive federal
statistical system, so it’s a constant balancing act. Sometimes the departments
win and sometimes the statistical system wins.

I think we hope that in the end, we can meet everybody’s needs, but it does
mean that some of the coordination activities perhaps don’t go as quickly or as
directly as they might. But I think the counterargument is that’s because are
primary goal is to meet the needs of the departments in which we are embedded.

I actually think lately it’s been working quite well. There’s a lot of
interaction on these cross-cutting things. We’re really trying to move towards
more consistency where we think it’s appropriate, best practice, especially in
linkage. Probably linkage, I would say, is the one example where the federal
statistical system is working very well together, even though we’re not going
to get IRS to change Title 26.

DR. SCHEUREN: Congress changes Title 26.

DR. MADANS: You are right. I’m sorry. We’re not going to get IRS to tell
Congress to change it. Our little statistical world is very little compared to
changing IRS legislation.

DR. GREEN: You actually covered most of my second question that I’d like to
ask you on balance. What is the statistical community’s position in terms of
current law? Do you have the laws that enable you to do the right thing that
needs to be done, or do you have laws that impede you? Where is the balance
disparity?

DR. MADANS: The answer there is really the same. There are some laws that,
if they were changed, would be better for us, but there is always the other
consequence. I prefer not to answer more than that.

DR. O’HARA: As I stated earlier, Title 13 gives us the authority to request
data from the various entities, but those entities have their own governing
regulation statute that often prevents them from sharing data with us.

DR. SCHEUREN: I used to be on this committee of 12. SOI Assisted Income
Division, which is where I used to be at the IRS is one of these 12 agencies.
IRS sits at the table, but it sits in a different way because of its different
focus, but nonetheless it’s central that we use the tax data for many purposes.

We have a good system — I think you just heard it well represented — but
it’s not like it was in the ’40s when we were really a single country and we
came together to solve a problem as a single country. Agencies matter more than
they should. Legislation is what it is. Its interpretation could be improved,
and some of the things we can’t do we could do if we would simply sit down
together and talk about how to interpret, and then get clarity.

One of the great things that the national center here has is an IRB. That is
enormously important because it’s the link between the agency and the American
people and the policy and thinking of the people. That’s a crucial step. You
should ask how many other agencies have record linkage activities that are
covered by IRBs.

DR. GREEN: To bring it back to our topic, so much of what we’ve talked about
has much broader implications than just assessing socioeconomic status. Coming
back to SES, a question for you, Dr. Scheuren. I didn’t understand the point
you were making about measuring poverty near the end of your presentation.
Would you mind running back over that again?

DR. SCHEUREN: The system of record that is used to measure poverty in the
United States is the Current Population Survey. Now it’s changed its name, but
still the same system, which is basically the March income supplement to the
CPS. We’ve been looking at that. It was the data system that Mollie started
with in March 1962 and her first paper, and there was a subsequent paper in
1965. We’re based on that.

The interest in linkage, I think, grew out of Mollie’s interest in measuring
the poor. That was a very full area. But in terms of the CPS itself, in early
going there was hardly any misreporting, hardly any understanding of
misreporting, and hardly any non-reporting. Remember, we’re still in the halo
of WWII. We were just very cooperate people then, so we got very high response
rates. That has really changed. About a third of the income that is measured in
the CPS is imputed, and yet we’re still doing pretty well.

DR. MADANS: The CPS is very concerned about making an estimate of the
poverty rate. That is where it comes from. It’s reported very often. We collect
information, as well, on our surveys, and we use the formulas. For each person
we identify where they are on the poverty threshold.

We are not interested in making estimates of the poverty rate. That doesn’t
come from our surveys. But we are interested in looking at health
characteristics by poverty status. Our concern is much more with
misclassification and the amount of misclassification when I say you’re 100
percent of poverty or 200 percent of poverty.

When we do our linkage back to CPS, because CPS can then link to IRS, we
know if we get the exact amount right, then, of course, we don’t have any
problem. But if we don’t, where are errors coming? Where is our
misclassification coming? Are we making people who are poor not poor or not
poor poor, and where in that continuum? That’s where our fit-for-use comes. If
we’re off on the total and it doesn’t affect the relative ranking, it doesn’t
matter to us.

There is a lot of conversation now in terms of we don’t have official
statistics, really, in the US. There are things that come out of the Census
Bureau that are kind of official, but after that there really is nothing that
this is the official statistics. We collect disability and they collect
disability and we get income. Which is the number? There is no number.

We have to figure out as a statistical community how to present this so
everybody’s not confused. Part of the key to that is I’m collecting it not
because I’m making the estimate, but because I need to use it for other things.
Of course, CPS uses it for other things as well. They’re looking at other
things collected on the CPS by poverty status, but their primary thing is to
collect poverty status.

DR. SCHEUREN: Joan Jurich led an effort to look at these cross-comparison
among the surveys, and John Sheiker(?) has the results. That’s a very important
activity. John may even be in the room now. That was a good answer, but that’s
where to go for a deeper answer.

DR. MAYS: Let me ask some questions about linkages, specifically in the
areas of SES. I want to start with education. One of the things that we heard
is that maybe an approach in terms of thinking about getting data that gets us
better quality data about education is to begin to link to information about
schools. It might be the nature of the school. It might be the context of how
well they perform. There are those type of statistics that exist in the
Department of Education. I’ve heard you all talk about linkages that are
predominantly between Census and NCHS. Can you talk about any plans or things
that you think you can do in terms of linking with education data?

DR. MADANS: Most of our linkages are not with Census. We don’t link at all
with the actual census. We link with contextual data from the census, but most
of our linkages are actually with CMS. It’s the healthcare data. It’s the
Medicare, Medicaid, and things like that.

If there is a national database that has consistent data at the Department
of Education, we could link to it. It would mean we would have to get the
linkage information into the survey. What I was hearing yesterday is we have to
get the name of the school. We would have to look into what kind of burden is
that and how hard is it to do and what’s the quality of the data. Then we’d
have to talk to Education about how would that linkage go, what do they have on
their files. We’d have to go back to the IRB and say do you consider this
within the informed consent. They may say no. I don’t know.

Given that there’s a cost of it, what would be the payoff? I think we would
have to justify internally that getting that other piece of information puts us
further enough down the road that it’s worth getting. I guess I’m not sure,
from what I heard yesterday, that the cost would be worth the benefit to just
get where the school is or what kind of a school is it.

California has the achievement data. That was a little bit more interesting.
But that piece of information is on the causal path to things. How one uses it
in terms of the kind of information HIS collects, which is cross-sectional —
how would we use it? I think we would have to think hard about that. I’m not
saying we wouldn’t do it. I’m not saying it’s not an interesting thing to do.
But because we have limited resources, we’d first want to make sure there was
something linked to. We would have to do all those things together.

DR. O’HARA: From the Census perspective, I’ve had some limited experience
negotiating with the Department of Education when we attempted to acquire the
FAFSA data, which is the free application for federal student aid. We got
pretty far into the drafting of an agreement to share the data before their
lawyers questioned what Census Bureau’s intent was with the data.

Their statute states that the data could only be used for the administration
of that program. Because our interest in the linkage exceeded that boundary
because we, of course, wanted to understand the quality of information that had
been collected by the Census Bureau, we then shelved that agreement.

You will get back to an authorization legislative change argument
potentially. But as Jennifer said, if there was a national database with the
school-level information, that would be great. They could attempt to approach
that. To my knowledge, right now they’re managed at the state levels. The
Census Bureau has been somewhat foolish enough in the past to try to negotiate
state by state.

That’s why we have a program that’s called LEHD, the Longitudinal Employer
Household Dynamics program, that in order to get income data, agreements were
written with each state to get the unemployment insurance wage information.
Then the data were collected at Census and harmonized in order to build this
data product.

If a national-level resource doesn’t exist, some agency may have to go and
investigate whether it’s worthwhile to negotiate and try to assemble the
national database, understanding the risks and the timeline and the cost
involved in gathering all the pieces of information and attempting to make them
into a national-level resource.

DR. SCHEUREN: The Census Bureau has a School Staffing Survey, which they do
for education. You could look at that. It’s done relatively frequently. It’s
quite good. It’s done for public schools and private schools.

DR. MAYS: The name of it is?

DR. SCHEUREN: School Staffing Survey. It’s quite a good resource. I’ve used
it before. There are public records for public schools. There’s what we call a
frame in the sampling business. That’s available to you, and it’s pretty good,
by the way. We’ve actually analyzed that frame. Private schools are different,
but there’s some information about private schools that’s available publicly,
too, but it’s not complete.

DR. MAYS: Let me ask about a statistical approach here. One of the things in
terms of income is that we know that income may vary by area, so often you have
some type of geographic approximation that allows you then to be able to make
comparisons. In the area of education is that possible?

For example, if you linked this data, my next thing that I start worrying
about is ninth grade in Alabama is different than ninth grade in California.
You’ve managed in terms of income to be able to come up with a way to do a good
comparison. Do you think that there are statistical approaches where you would
be able to do the same for education?

If I were to buy a house, for example, in Alabama, the cost of the house
might be $60,000. Say, if I were to buy a house in California, the cost of that
house would be maybe $160,000. What individuals are able to do is if you’re
trying to give me a package that’s a housing package, you could tell me what
percentage of my salary makes a difference in terms of being able to buy the
house in one place versus the other.

Sometimes in terms of money what we often talk about is the cost of living
in A is different than the cost of living in B, so therefore your wages in A
would be different than your wages in B, but you could say that you’re really
still getting paid in a very similar fashion.

DR. MADANS: It sounds like you’re saying can we make an adjustment to the
quality of education. No, I don’t know any way of doing that. Maybe if the
education people said based on the standardized tests, we can kind of do a
discounting of what — we know what percentage of the population that census
tract has a given level of education, and they can do a discounting. If they
did that and we could attach it to the geographic, then it’s easy for us to get
it, but we’ve never done that.

DR. MAYS: That was the only thing that was suggested yesterday, was the
potential of the standardized test scores.

DR. SCHEUREN: Have you looked at the work they’ve done internationally
comparing fourth grade math courses and eighth grade math courses? You don’t
have to speak German. You don’t have to speak Japanese. Just see the same
lesson taught in three different cultures, and then understand what our problem
is. You can see that regionally, too, but that work hasn’t been done, to my
knowledge, not much of it anyway.

DR. PARKER: As a health statistics agency, we do focus on those things, but
all sorts of data users come to the Research Data Center with their own
contextual data. Certainly, people with expertise or education or income bring
area-level data, and we can merge it, as I said, quite easily using any of
those census geographies. Or if it’s more pertinent, there are lat and longs
for most of our survey respondents now, and different types of measures can be
done. We rely on the people with those expertise, with those data, to bring
them to us.

Those data are owned by the people who bring them into the Research Data
Center in that we do not keep them and distribute them to others, although I
have got personally some research arrangements with people to do that. I think
those sorts of things can be done, we just don’t, as a rule, give out other
people’s data when they bring them for research purposes.

DR. MADANS: The other thing Jennifer should have mentioned is what we do do
are some of these kind of higher-level SES in terms of where do you live. Some
of those measures, let’s say, are air quality. We do link to EPA data on air
quality and transportation.

DR. PARKER: As a specific example, with the air quality, I worked with
people at the EPA to do that. Part of the arrangement was that those data would
be public, and also the transportation data. Both of those required expertise
from other people. Part of the in-kind I had a personal relationship to make
those available to the public.

Again, people just coming into the Research Data Center with their own data,
for example, education quality, for lack of a better word, at the census tract
or census block or county level, those would be owned by those people unless
they were collaborating with someone in NCHS to make them more public. Then we
would stick them on our webpage and say they’re available, and that would be
great.

As part of that, part of the linkage would be an evaluation of it. We’re not
just going to take data from somebody else and say we’re going to release this
as an NCHS product. Both the air pollution and traffic data, our part of it was
an evaluation of fit-for-use — I like that term — but what does it mean, what
are the caveats, before we released it to the public, because we aren’t experts
in air quality or transportation.

DR. MAYS: Let me just ask about in terms of occupation, if there’s synergy
between NIOSH and NCHS.

DR. BARON: In 2010 we did an occupational health supplement as part of NHIS.
I think that’s probably the strongest place where we’ve collaborated.

DR. MADANS: Also, hasn’t there been some on the death certificates?

DR. BARON: We have something called NOMS, the National Occupational
Mortality System. We collect the occupation codes on the death certificates and
have a huge data system where people can look at that.

DR. KAPLAN: I am still sort of hung up on this roundtable of this dozen
agencies getting together and talking about things. I was just thinking about
harmonization is something that we’re interested in. On the other hand, there’s
a cost to harmonization also. As we learned yesterday, when people do things a
little bit differently, we discover discrepancies, and that leads to
improvement of the methodologies. I was wondering about this discussion among
the agencies. Is there discussion about sort of common methodologies that you
might experiment with in the future?

DR. MADANS: Yes. That is actually where a lot of the cross-agency
methodological work is done. It’s more on the methods side than it is on the
substantive side because where we’re all separate is in our substantive
interest, our subject matter interest. Where we all come together is in the
methods. That’s where most of the conversation is.

I think where people try to come together is, first, does it really make a
difference if we’re all the same, or does it just look odd, but it doesn’t
really matter? Or can we kind of come together where we have a common core? I
think this is where the idea of the minimum standards comes from.

You start with this core, and then you’re going to go off and do a whole lot
more stuff in income that we’re not going to do, but we’re going to do a whole
lot more stuff in health that you’re not going to do, but you’re going to have
those disabilities ones that we agreed on on your survey, so we can kind of do
a little bit more cross-walking. I think the key question is when are
differences important enough to get rid of and when can we harmonize them and
when do we really have to kind of come together?

The Census Bureau actually has another interest in this. That is they are
the data collector for us on a lot of our surveys. They would really like that
we all ask education the same way because their interviewers will do a better
job if they have to — oh, god, HIS does it this way. There’s been a push for a
long time from the Census Bureau to say let’s standardize the demographics. I
think the other push is, let’s standardize the core, and then kind of have
everybody go off on their own direction.

DR. MAYS: Nancy has to ask a quick question. You can tell this linkage stuff
— we’re really happy to have you here.

DR. BREEN: This is potentially linkage and also kinds of harks back to
Larry’s question about whether the federal government statistical agencies were
linked. Now that we know that there are some really good mechanisms for
collaboration among the federal agencies, statistical gathering agencies, I’m
wondering are there also any links with state and local?

That’s another thing this committee has been interested in, is community
data is very powerful for changes in public health and probably changes in
other things as well. But it doesn’t have an infrastructure, as far as we can
gather. It’s pretty catch as catch can. Is there any formal or informal
networks or ways in which the federal statistical agencies or individuals in
those agencies to communicate with people who are doing the same kind of work
or trying to do the same kind of work in local and state contexts?

DR. MADANS: I would say probably every agency is different on that. Probably
the Census Bureau has more contacts than anybody else, perhaps Labor, because
of the way they’re organized and get some of the information from state
agencies. Our biggest connection with the state and local is through the Vital
Registration System. We don’t have a lot of contacts with state-level data
collection. A lot of that comes out of Atlanta for the surveys that they do.
Part of the problem is it’s not always in the same place. We don’t have that
infrastructure. I think in the past there were more connections. I don’t think
they exist as much now.

DR. BREEN: Could you elaborate on that?

DR. MADANS: I think there have been some mechanisms that kind of linked
NCHS. Don’t quote me because I don’t know a lot about it. I kind of have this
vague memory, but it’s a long time ago.

MS. GREENBERG: Obviously, I think the core was always the vital statistics,
but then we were quite instrumental in helping to develop state centers for
health statistics. We have some intention and even an objective of gathering —
at one point we had responsibility for manpower data, and we try to collect
that now at HRSA. There are other areas, too, hospital district data, et
cetera, but we really just didn’t have the funding for it.

AHRQ actually works with all the states for their HCUP, the Health Cost and
Utilization Project. They work closely with the state entities, as Jennifer
said, some of which are state centers for health statistics, some are state
hospital associations, some of them are universities that collect. That’s one
of the strongest beyond the vitals.

As was observed and recommended in the report that the committee recently
put out, there isn’t a good infrastructure across the broader areas. There are
some states that have much stronger statistical capabilities and centers. Some
of these state centers for health statistics never really did that much more,
not that it isn’t important, than the vitals, but others are very robust. I
think that could be explored further by the committee.

DR. SCHEUREN: I was going to make a comment about the BFRSS. That has an
advantage in that it doesn’t have an OMB clearance. I’m saying that publicly. I
said it privately too. They have a distinction between fitness for use and
performance requirements. They’re using performance requirements definitions of
quality where fitness for use depends on the user. That should be what is done.
Census Bureau finds a way to balance that. The BFRSS does that.

DR. MADANS: Except it also causes a lot of non-comparable data.

DR. QUEEN: It has one income question, and then they have a single
categorical education variable.

DR. MADANS: On the health outcomes, some are very similar to the national,
some are not, because they collect what they need at the state level.

DR. O’HARA: Two of the agencies that sit around the ICSP roundtable are the
Census Bureau and the Economic Research Service of USDA. There is no national
database for SNAP participants, which the program that’s formally known as food
stamps that is now the Supplemental Nutrition Assistance Program.

We are engaging state by state to try to acquire and assess and harmonize
SNAP participation data from the Census perspective so that we can understand
whether we are collecting it appropriately in our surveys and also for
downstream usage in the supplemental poverty measures Fritz was describing. But
it is state by state.

In several of the states we are fortunate to be partnering with research
institutions like Chapin Hall Center for Children at the University of Chicago,
the Ray Marshall Center at UT Austin, and Jacob Franz(?) in Maryland, if you
can find research partners that have that in with the states and they really
understand their area’s data

In New York we didn’t have a research partner in that capacity, so we
actually wrote an agreement with the New York State Office of Temporary and
Disability Assistance in order to enter a data sharing agreement. There is a
different level of understanding of the data that we’re getting because we
don’t have a research partner cut in, but it’s time-consuming and it’s
coalition-building in order to gain information on the sub-national programs.

DR. MAYS: Thank you very much. This has been very useful. As you can see, we
could spend a day with you, but we’re going to do a couple other things today.
Thank you very much for being here. We really appreciate it. We’re going to
adjourn until 11:10.

(Break)

Agenda Item: Panel: Methodology

DR. MAYS: We’re about to move to our panel on methodology. One of the things
in terms of thinking about the recommendations that we want to make, we also
want to understand lots of things about that and make sure that they are
reasonable. I think part of what we have had the opportunity to do is to hear a
little bit about things that have gone on in terms of the other 4302 standards
to give us some insights to help temper our thinking. Let’s get started with
John Czajka from Mathematica who is going to be talking about measuring income
simply.

DR. CZAJKA: I want to make some acknowledgements of sources of support for
some of the things I’ve learned and will be presenting here. I’ll talk a little
bit about some work that I’m conducted currently for ASPE and for the Census
Bureau. Another part of this talk comes from a report that I prepared for ASPE
with a colleague, Gabrielle Denmead, a few years ago.

I wanted to make a few general observations before getting into some
details. I suspect the observations may be more helpful to you than the details
that are coming. I have a couple slides devoted to a few conclusions.

If we’re talking about income data, there’s really no survey that gets it
right in all respects, despite the great effort the different surveys spend to
collect income data. There are limitations to all of these.

Another is that income is most difficult to measure in probably the bottom
third of the income distribution. Way up at the top there are great challenges,
but those people don’t show up a lot in our surveys, and we don’t report most
of what they say anyway. But the lower third, which is where the policy
interest is, is where we have the greatest challenges. This comes from the fact
that both income and family composition are less stable down there, and the
sources of income that people draw on are more varied.

Another point is that it is, for some purposes, important to distinguish
between current income — monthly income may be a way to think about this —
versus annual incomes. This is important because many of the means-tested
programs determine eligibility based on current monthly income. This is
something that most of our surveys don’t collect. Most surveys collect annual
data. For a lot of the population you can divide annual income by 12 and that’s
really what people are getting every month. That’s not the case in the lower
part of the income distribution.

Another point that I think occurred multiple times already is that
non-response to survey income questions is among the highest rates of
non-response that we seen in surveys. Although I’m not going to talk about
asset data, it’s worse for asset data. It’s a combination of people feeling an
invasion of their privacy in responding to these questions, and many of the
things that they’re being asked to answer they cannot answer readily or not to
the degree of accuracy that they think is being expected of them.

Something that we see in survey data and I’ll be looking at later is that
there can be a lot of rounding in how people report their income. This has much
bigger implications the lower you get down into the income level. This is part
of the difficulty of measuring low-income people, measuring their income.

Another point that’s really not often distinguished is most of our surveys
are household things. They go to an address, they collect data on the people
who live in the household. But within the household you have family, and it’s
not always the case that there’s one family in the household. Then there are
individual people. It can be easy to lose sight of this. We talked about
household income, but when we measure poverty, it’s family income that we’re
using. Something I’ll be showing you is that the poverty rate is sensitive to
how we define the family, who is included in the family.

Another point is that if we’re looking at ways to simplify or even improve
income measurement, it’s important to allot attention to the sources of income
to the extent that they represent total income. Some talk about with earnings
it’s 85 percent of income or something close to that. Presumably you want to
spend more effort collecting that than something that accounts for a tiny
fraction of income.

The last point — and this will also be the last part of the presentation —
is that the collection of retirement income presents a growing challenge. Given
levels of poverty that we can see among the elderly, this is a significant
problem going down the road.

There’s really no gold standard for survey estimates of income. We have the
Current Population Survey that is considered the official source of estimates
of income and poverty in the United States. In work I’m presenting here I’m
looking across six different federal surveys.

Given that there is a general tendency to underreport income in surveys, as
a general rule, if you get more income from your survey, you’re probably doing
better. Based on that standard, if we look across several surveys at some
aggregate amounts of income, we find that across the Current Population Survey,
the American Community Survey, the Medical Expenditure Panel Survey, and the
National Health Interview Survey, there’s just a 5 percent difference in total
income despite huge differences in measurement.

The estimate here for the National Health Interview Survey is based on one
question. The estimate for the Current Population Survey is based on 20
sources. The estimate for the Survey of Income and Program Participation
collects 68 sources monthly. Yet, as you see here, SIPP’s estimate of total
income was 11 percent below the Current Population Survey and 6 percent or so
below the National Health Interview Survey.

Another survey, the Panel Study of Income Dynamics, had a weighted
population that was considerably below the total for the other surveys, and yet
it had the highest aggregate income of all of them. These are differences
across surveys that devote varying amounts of attention, and you don’t
necessarily see that the more effort that goes into income produces the biggest
total.

We’re more interested in looking at how income breaks out across segments of
the population than these aggregates. The aggregates can be driven by what’s
happening at the very top, and that’s less so when we look here. Setting aside
the PSID over on the right, one of the things that’s very striking is how
similar the American Community Survey is to the Current Population Survey.

The American Community Survey — and I guess you heard about this yesterday
— is basically the long form for the Census pulled out and turned into an
annual survey. They basically have eight income questions, but they do ask them
of every person in the household.

Two-thirds of the responses to these data come in through a mail-back.
There’s not an interviewer. People are sent the form, they fill it out, they
send it back, and yet this looks very much like the Current Population Survey
in terms of how income is distributed across quintiles. What I mean by
quintiles here is we take family income and we determine the highest FIPP, the
next FIP, and so on, based on persons.

We find that the Medical Expenditure Panel Survey, which includes a waiting
adjustment to make it line up at the poverty distribution in the Current
Population Survey, nevertheless has these differences. It’s higher in kind of
the middle part of the distribution, a good deal lower at the bottom.

SIPP, I mentioned, which devotes all its effort to collecting income, does
capture more at the bottom than the other surveys. It’s about 5 percent more
than the Current Population Survey, but it trails off pretty steeply as we go
up, and it’s only about 82 percent in the top quintile.

I just put these up to give you a sense that the population totals aren’t
the same across these surveys, even though they’re referring to basically the
same point in time. There are differences of a few million, and that can have a
difference in total income when you weight it up. If we divide things through
to come up with a per capita count, you can see what the differences are like
and actually how much per capita income varies across these quintiles.

You still see that SIPP is higher than CPS and ACS at the bottom. The PSID,
this is a long-running longitudinal survey. There may be issues about how
representative it has become after the 30-40 years that it’s been running, but
it’s quite a bit higher. You see the Health Interview Survey.

What you’ll begin to see is one question they get close to the total, but as
you start to look further at the data, you’ll see that there really is a price
you pay for that simplicity, and a big chunk of the price is down at the
bottom. This will come through more clearly later.

Earnings accounts for, overall, a little over 80 percent of the income. If
you look here at how these surveys differ, I think the most important thing
here is in the bottom quintile, that the ACS and SIPP are getting quite a bit
more earnings than the Current Population Survey.

In the case of SIPP when they’re going out and doing three interviews a
year, one could say that this is a case where it’s really helpful that people
with fluctuating incomes, you’ve got to get out there a lot to pick up all
these differences. But then how do you explain the ACS, which sends out this
mail questionnaire and gets even more income in terms of earnings from the
bottom?

If we look at unearned income, we see a big difference. The ACS is quite a
bit lower than the CPS at the bottom. Some Census Bureau people have said that
part of this may be a classification issue. People are not necessarily counting
earnings in this survey the way they might in the CPS. They’re giving you
similar totals, but they’re breaking it out differently.

The National Health Interview Survey here has an extremely low figure. Where
this comes from is that while they have just one question for total family
income, there is an earnings question that’s asked of everybody. That’s what we
used in the previous slide. As you saw, even that was higher than the CPS.
Every survey was higher than the CPS in earnings in the bottom quintile.

What we did was to get another income estimate, we took this total and
subtracted the earnings. There were actually a lot of negatives. People
reported more income when you asked them to provide earnings for individual
members than they gave as their total family income. In some cases you look at
the data and it was pretty clear they weren’t counting this kid when they gave
a family total. It does show the hazards of oversimplifying the data
collection.

There’s a lot of interest in the poverty rate. It’s an important measure.
This is what things look like across the surveys. One thing that stands out
here again with the Health Interview Survey and this simple approach is that
the poverty rate measured by the survey — and these are back in 2002 — was
2.5 percentage points higher than the official rate that comes out of the
Current Population Survey. That’s part of the issue of really not getting a lot
of income out of the lower part and having it show up this way.

The PSID was a good deal lower. SIPP had the lowest poverty, slightly, but
it had the most people in this next group from 100-200 percent of poverty.
Other than HIS, it was slightly higher than these others.

There are pretty substantial differences — and I think Linda’s going to
talk a lot more about this — with respect to how surveys capture participation
in various federal and state programs. If we look at the combination of
welfare, which is what used to be called food stamps, SIPP’s approach clearly
is showing a payoff. It’s capturing these at a much higher rate than the other
programs.

The Health Interview Survey with their question on food stamps is getting
only 5 percent of the population participating versus 11 in SIPP. The actual
should probably have been around 13 percent at this time. We see a similar
thing with SSI. The bottom deals with health insurance coverage. That’s a
different issue we’ll pass over today.

The CPS has a family definition that’s been in place going back to before
the poverty rate was established. It basically includes all persons who are
related by blood, marriage, or adoption. Two or three critical pieces that are
not in here is that unmarried partners are not put together into the same
family in the CPS. There are actually instances where an unmarried couple both
parented a child, and yet one of the parents will not be in that family for the
purposes of measuring poverty.

Both the Health Interview Survey and MEPS apply a broader definition of the
family where they include unmarried partners. They also include foster children
who are not included in the CPS family. We were able to compare for these two
surveys the impact of using one definition versus the other.

This is something that’s been pretty consistent across other surveys, that
you will find that it affects the poverty rate by about a full percentage point
if you lower the poverty rate by a percentage point by including unmarried
partners and foster children as well. When surveys set out to define the
family, they may not be thinking about this, but this is a little implication
for income measurement that can come through.

As you see at the bottom, there were big differences among different subsets
of the population. Not surprisingly, single parents makes a huge difference
because a fair number of these parents have an unmarried partner.

You heard about non-response. Basically, to try to measure this across
surveys, it differed dramatically in terms of how many questions they asked.
Rather than looking at the frequency with which people failed to respond, we
looked at how much of the total income was imputed. We find pretty substantial
differences across surveys, starting with the Current Population Survey. In
this particular year about 34 percent of the total income that’s measured in
that survey was imputed. It was assigned to people who did not respond to the
questions.

The American Community Survey had a rate of only about half of that. It’s
not clear why it’s so dramatically different. The ACS is a mandatory survey.
Perhaps people carried this over to their willingness to respond to income
questions. But it’s a clear outlier. Everywhere else you’re seeing things that
are pretty comparable to the CPS.

We made a distinction here between whether the imputation was done using
information that you could say is related to the missing information. Maybe the
best example is SIPP, where people are interviewed every four months. You might
have a response the previous four months for an income source that you’re not
getting this interview. That’s information that could be used. Arguably it’s
better. We don’t know that for sure.

MEPS has a lot of non-response to their annual earnings question, but in
this survey they get somewhat better response to wage rates and hours worked,
so they do an imputation based on blowing up hourly wages to an annual figure.
In any event, they end up with between 40-45 percent of their total income that
had to be imputed in some way. HIS is very comparable to CPS as well. One real
consistency across the surveys is how unwilling many people are or at least
feel themselves unable to respond to the income questions.

We didn’t have any mechanism in our analysis for assessing how accurate the
reported incomes were. There’s been a lot of linkage work. I don’t know if you
heard anything about this yesterday or this morning. One of the things we can
look at is how much the responses are rounded. In particular, if you look at a
figure, what percent of the people are giving you an answer that’s divisible
exactly by $5,000 or even $10,000?

The problem with the rounding if you’re thinking about a poverty measure is
that rounding to the nearest $5,000 can make an enormous difference in whether
you’re considered poor or not. The other thing is that in trying to look at
changes in income over time, if people are rounding a huge amount, you’re not
going to capture those changes. Then you may see a huge shift.

We cap this at incomes below $52,500, and we look separately at personal
earnings, which is generally one question, versus total family income, which
could be the result of many questions, and ask how often was the figure exactly
divisible by $5,000. In the Current Population Survey it was 28 percent of
earnings was exactly divisible by $5,000. Even 11 percent of total income was
divisible that way.

In ACS it was slightly worse. SIPP is extremely low because it’s monthly
income that they’re collecting, so somebody would really have to be pretty
devious to figure out how to report monthly income that round out at the end.
MEPS was a good deal lower than CPS. But look at the result of the
single-question approach in the HIS. 36 percent of total family income is
divisible by $5,000 evenly. The figures run for personal earnings at about 40
percent.

That’s something we can quantify about the quality of the data. It’s a much
bigger concern lower in the distribution than it is. I think in the ACS when we
looked at it, the fraction of people who reported exactly $20,000 as their
income was really quite surprising.

I want to say a little bit about where income comes from. In the first
column here we were looking at the percent of families who report each of these
sources or a set of sources here, then continuing with less important sources
on the next slide. For the percent of families reporting, these don’t add up to
anything in particular. People can have many sources.

On the right we look at what fraction of the total income fell into
individual types. We see that wage and salary earnings for the whole population
was 77 percent of the total. Self-employment added about another 5 percent. You
look down to Social Security, you combine that with retirement income, you’re
talking about 10 percent.

But then there are a lot of components that are really pretty small with
respect to how much of the total they account for. When we move to some of
these other sources, they’re well below half a percent of the total. One does
have to ask what is the value of going extensively after all of these sources?
There are really two reasons why you want to go after a lot of sources.

One is you think you get a better total if you ask for a lot of the
components because people may forget that they had this account over there that
they get a couple thousand dollars from. But you may also want to know these
particular sources.

Way back when the Survey of Income and Program Participation was being
designed, there was a real focus on being able to measure eligibility for
various means-tested programs. A lot of these programs treat different sources
differently. They may not count income from a certain source or they may
discount it. So there was a need to bring in all of that detail.

You do see that you get the most bang for your buck going after earnings
because that’s such a big chunk. It does differ by income level. Here I’m
comparing people below poverty and people above 400 percent of poverty. The
share that’s due to earnings falls off quite a bit when you get down to the
poor. It’s about half, and you see that Social Security is pressing pretty
close to a quarter of the total.

But many of these other sources are small for both. We still collect data on
alimony, even though hardly anybody pays it. Even between the low and the high
end of the distribution, it’s really tiny, whereas child support is really
where most of this money is going these days. SSI becomes very important for
the poor, about 10 percent of total income. It’s almost nonexistent above that.

But the fact that among the poor these aren’t frequently really tiny is part
of this problem. This means for particular families many of these sources may
be the bulk of what they’re relying on for income. That’s why a simple approach
is not going to serve you as well with the low-income population as it does
higher up.

We had some comparison of earnings between SIPP and ACS and the CPS. This is
an area where ACS, SIPP, and CPS are pretty similar in the fraction of families
and unrelated individuals who report that they have earnings. ACS and CPS are
also very similar down at the bottom in how they compare in total dollars,
whereas SIPP falls off really substantially despite this approach that SIPP has
of going to people monthly.

One of the theories about that is that when you ask about earnings
frequently and ask people to report it by month, there may be a tendency for
people to give you take-home pay, even though you’re asking them for total,
because at that level your take-home pay is the salient number. You know that
that is more than the gross.

But when you get to the end of the year, it kind of flips. People can tell
you their gross salary at the end of the year, but they have more trouble with
the take-home pay. That could be a lot of what’s going on with SIPP in losing
such a big chunk. You get the benefit for people with erratic income, but for
people with extremely regular income, this approach doesn’t necessarily work.

There’s another huge difference in self-employment here where SIPP gets
dramatically more people showing that they have it and dramatically more total
dollars. This was part of a consciously different approach in SIPP to define
what self-employment income is.

SIPP asks people to report not only the profit that they get from a
business, but what they draw in salary, because there are businesses out there
that are failing, doing terribly, but the owners are still paying themselves a
salary. That’s a part of their income. That’s something that has not been
addressed in these other surveys.

The last area I want to talk about is retirement income. The traditional
defined benefit, as it’s called pension, where your employer promises to pay
you a certain amount of monthly income for life after you retire is really
becoming history in the private sector. It’s partly a function of change in the
laws governing how adequately pensions had to provide for their future
payments.

But increasingly, employers in the private sector have turned to defined
contribution plans where they’re making a contribution to an account that they
employee can frequently manage in terms of how it’s invested. The employee can
put money in that. The IRS came along with complementary changes in tax law
that set up individual retirement arrangements where people can contribute to a
retirement account that accumulates funds with tax deferred that has no
relation with their employment. It’s something they put in.

If you look at where the funds are, in 2009 private defined benefit plans
held about $2 trillion in total assets and annuities, which is a way of
converting accounts into monthly payments, accounted for about $1.4 trillion.
But the DC plans were almost double what the defined benefit plans were and the
IRAs were comparable. We’re not yet seeing this stuff come through in income,
but it’s going to keep growing dramatically, and down the road it’s going to
become more and more important as a source of retirement income.

The basic problem is that the surveys haven’t really adjusted to this
change. If you have a 401(k) and an IRA, you don’t get a payment; you make a
withdrawal when you want the money. Withdrawals from savings have not been
considered income over the years.

But what’s basically happened is that we’ve changed the whole retirement
system from something where people got a steady income stream that was very
easy to measure in surveys because it was so consistent to one where people
have these savings accounts. So a lot of this retirement income has kind of
been moved offline with respect to how the surveys capture income.

Over time, part of the difficulty in dealing with this is that there’s this
conceptual question of what is income, really? As I said, we haven’t tended to
count withdrawals from savings as income, but that’s how these things work.
Unless we come to grips with this, more and more of the income that retirees
have access to is going to move out of the surveys and not be counted as part
of their support.

We’ll be looking at Social Security as basically most of the income that
elderly people receive, maybe classify more and more of them as poor when they
may be sitting on these multimillion-dollar — not everybody, but certainly
some — accounts and being able to withdraw money at will more than they’ll
ever need, and yet it’s not being measured.

We see a difference here between surveys. SIPP does a better job at this.
The top line here we’re looking at Social Security. Here we’ve got SIPP on the
bottom. CPS is hitting in terms of the number of families reporting at about 87
percent of what SIPP was.

But look down at these other sources. Income from a pension, which would be
mostly the defined benefit, was about 76 percent, other retirement benefits
about 37 percent, and then these withdrawals from IRAs only about 11 percent
being reported in CPS compared to SIPP.

The dollars look a little better for CPS. Social Security is pretty
comparable. But there’s still a 19 percent difference in pension money, about a
37 percent difference in other types of retirement, and then about almost an 80
percent difference in what’s being pulled from IRAs. This is back to the point
where CPS is our official source, and yet for something like this it’s doing
dramatically less well than another survey. That was it for prepared remarks.

DR. MAYS: Thank you very much. Our next presenter is Linda Giannarelli, who
is a senior fellow at the Urban Institute and works in the Income and Benefits
Policy Center. Thank you and welcome.

MS. GIANNARELLI: It’s a pleasure to be here today. I’m the Project Director
for the TRIM3 Microsimulation Project at the Urban Institute. Not surprisingly,
my comments are going to be pretty micro-level. We heard from John about some
of the big-picture comparisons of income collection across a bunch of surveys.
I applaud that report. It’s a fabulous report. It’s huge. It has some amazing
tables that really compare across surveys how they went about collecting
income, what their sampling frame is, all kinds of comparisons. If you’re ever
interested in those things, that’s the place to go.

The context for my remarks is as a user of the federal surveys that collect
data on income and program participation: CPS, ASC, SIPP, and to some extent
also NHIS and MEPS. I’m also going to focus primarily on the low-income
population and subgroups of that population.

I know that you all are interested broadly in socioeconomic status, but a
lot of the work that we do is focused more on that lower end of the income
distribution. Some of the things that in a broader context, may not seem to be
as much of an issue when we’re narrowing our focus to the low-income population
or even subgroups of the low-income population, they can be more of an issue.

Some particular issues are analysis of program participation, who is
receiving benefits from various programs, level of dependency, which is
something that ASPE is very interested in. There’s an annual publication on
indicators of welfare dependence not just looking at who’s receiving benefits
from programs, but also the extent to which people are dependent on benefits
from those programs. Also, the ability to assess the impact of hypothetical
changes in policies. That is some of the background to what I’ll be talking
about for a few minutes.

I want to focus primarily on two big challenges that users of these surveys
face. One of them is misreporting and primarily underreporting, but not always
underreporting. We can look at that in a couple of different ways. We can look
at aggregate counts in the survey data compared to actual totals. I guess I
should put “actual” in quotes because there are questions about
administrative datasets as well, but as close as we can get to actual. Also,
findings from exact match projects, which we also heard about a little bit in
the prior panel both from the agency staff and from Fritz a little bit.

The second issue that I want to touch on is missing data and how missing
data are addressed through allocation. Finally, I’ll close with a little bit of
a wish list.

Here’s a big picture. I just pulled this from the work done by some
colleagues at the Urban Institute, Austin Nichols and Karen Smith. They, along
with some other staff, worked on a project that was funded by the Census
Bureau. Urban Institute partnered with Westat in doing this project. The focus
of the project was on the income data in the ASEC. You probably heard about
that or may have heard about that from Chuck Nelson yesterday.

But just to kind of set the big-picture stage here, CPS-ASEC underreporting
looking at calendar year 2009 income data that were collected versus the
national income and product accounts. For each one of these line items we could
debate the suitability of the NIPA or alternate sources, but this is just one
overview way that’s comparable across all these income items.

You can see up at the top that according to a comparison with the NIPA, that
this year’s ASEC data captured 93 percent of wage and salary data. That’s
consistent with what we heard earlier, that the wage and salary reporting is
pretty good. But then we see some rather depressingly small numbers, 27 percent
of interest income, 14 percent of dividend income.

If we’re going to focus for a moment mostly on the lower-income population,
dividend income is probably not the most important thing, although I will point
out that if someone’s primary income is from dividend income and they do not
report it, then they may appear to be low-income when, in fact, they are not.

But if we look down at the bottom two rows at two programs that are very
important for the lower-income population, in that year’s ASEC data it looks
like about 75 percent of SSI income was captured in the data and about 56
percent of SNAP income was captured in the data. I’m trying to be a little
careful with my terminology here and say captured in the data rather than
reported in the data because there is a fairly wide gulf between those things.
We’ll look at that in a minute.

The previous slide was looking at dollar amounts, but often if we’re doing
some sort of research project, we’re not necessarily interested in exactly how
much someone received from TANF. We’re interested in, perhaps, the population
of families who received TANF versus non-TANF.

This slide is trying to look at to what extent is enrollment in these
programs captured in surveys. Again, I’m picking on the ASEC data here, but the
picture would not be terribly different with another survey. SIPP does do a
somewhat better job at capturing enrollment in these programs, but there is
still a substantial amount of underreporting of enrollment in these programs,
even in the SIPP data. I’m sorry I didn’t bring a slide on that.

In this particular slide if we look at the TANF enrollment reported in the
ASEC data for calendar year 2009, 58 percent of that enrollment is captured in
the data, including not only the truly reported enrollment, but the allocated
as well. 58 percent or so happened in this particular year to be the same for
SNAP enrollment.

Medicaid and SCHIP, if we combine those two and look at them together,
because there is often a lot of confusion between those for the reporting of
those programs, we do get much higher, 87 percent for SSI. Our internal
calculations for that are 76 percent.

Even if we include the allocated responses, we’re not getting a full picture
of the people who are enrolled in these programs. To the extent that someone is
doing research that is a shorthand for a portion of the low SES population that
is looking at enrollment in these programs, they’re not getting the whole
population.

The last couple of slides looked at the ASEC data. To touch real briefly on
the ACS, as John mentioned, one thing that’s quit different about the ASC is
that many respondents are filling out the form independently, and also very few
income sources are collected individually in the ACS data. I’ve listed them
here.

The only ones that have their own little box for you to write a dollar
amount in are wages and salaries, self-employment, Social Security, and SSI.
Everything else is a box that would be combining a bunch of different things.
All public assistance and welfare income is written down in the same box. All
asset income, interest dividends, and rent is written down in the same box. All
retirement income and all other income is written down in the same box.
Veteran’s payments, child support, workers’ comp, unemployment comp, and
anything else is all written down in the same box.

To look some possible implications of that, this slide says likely TANF
income versus program data. The reason it says likely TANF income is that we
don’t know exactly who is reporting TANF. We know that people are reporting
public assistance or welfare. I’m defining likely TANF income here as income
that is reported as public assistance or welfare payments that is reported by
low-income families with children.

If I call that likely TANF, we did some work at the Urban Institute recently
where we were focusing on poverty in three states using the ASC data. That’s
where those three states were coming from, from that project. If we compare the
likely TANF income to actual TANF benefits paid out in the state according to
program administrative data, in Georgia the ACS captures, including
allocations, 84 percent, Illinois 96 percent, Massachusetts 23 percent.

One’s initial reaction to this might be way to go, Illinois. Look at what a
great job they’re doing reporting their TANF income. Sadly, we don’t think it’s
actually such a great thing. Because we were focusing on these individual
states and we were also working with some partners in those states, we did look
quite closely at these data because that 96 percent just seemed too good to be
true.

We looked not just at the public assistance or welfare payments that are
reported by low-income families with children in Illinois, but we added up all
those dollar amounts across everyone in Illinois in that year. Then we talked
with our partner in Illinois to ask what other public assistance and welfare
programs are there in Illinois that people could be reporting there. We tried
to add up all those dollar amounts that logically it seemed could be reported
there.

We could not come up to the total that was reported in the ASC. Honestly, we
really don’t know what was going on there, just that it appears that some of it
was not actually public assistance or welfare or at least not what the survey
designers had in mind by that question.

Here’s the same kind of comparison for likely unemployment compensation.
We’re defining likely unemployment compensation as income that’s reported from
any other source that is reported by individuals with apparent unemployment.
Someone has written down something on that other line. It could be child
support. It could be veteran’s payments. We tried to make a pretty good guess.
If they were a veteran, we figured it was probably veteran’s payments. If they
were a single mom, we figured it was probably child support.

If we compare likely unemployment compensation income versus program data in
those three states, none of them cracked 30 percent. I think this is just
another illustration of John’s point that, in general, the more questions you
ask, the more you are going to get people to remember certain sources of
income.

Finally, SSI income, which does have its own box in the ACS in each of those
states — it appeared that over 80 percent of the SSI income was getting
captured, although the data do show a higher incidence of very high SSI
amounts. By very high, I mean we know what the maximum SSI payment is for
someone in a particular state, at least the maximum payment. It’s the maximum
monthly federal payment plus whatever the state supplement is in that state.

If we take that and we multiply it by 12, ignoring for a moment people who
are getting a retroactive SSI payment, we should know the maximum that anybody
should get in a year. There’s a higher incidence of very high SSI amounts in
the ACS data versus the ASEC data, which may be related to greater confusion
with Social Security, since you are responding to this survey on your own in
many cases.

I want to just mention real briefly what the exact match studies tell us
about how we’re doing with our current methods of capturing some of this income
for low-SES individuals. We heard in the earlier panel about one use of these
exact match studies, which is to see to what extent are people who appear to be
eligible for programs receiving or not receiving those benefits. But these
studies can also be used to look at how well are people reporting.

We heard from Fritz about the exact match work that he’s done with ASPE and
with partners at Census Bureau matching CPS with DER data. We also heard from
Census Bureau about some of their exact match studies. In general, focusing on
those projects that are looking at program benefits, many actual recipients
fail to report enrollment, and some reporters are not actually enrolled,
according to the administrative data. I just want to put up two examples.

This is published work by Julie Parker at the Census Bureau. At the end of
my handout I’ve got the citation to this. This was an exact match of 2005 ASEC
data with SNAP program data for three states. These are weighted numbers of
households, but keep in mind these are just those three states. She found
833,000 correct reporters, so they said they got SNAP and they were in the
administrative data. She found 922,000 false negatives. That is not a typo.

This is not in any way remotely to criticize the amazing work that’s done at
the Census Bureau, but I do think that sometimes when we say there is
undercounting and there is missing data, we forget, or maybe it’s just too
depressing to think about the extent to which that is true in some of these
cases. She also found 92,000 false positives.

Here’s another exact match project that I’m pretty sure most of you are
familiar with, the SNACC Project, or the Medicaid Undercount Project. I hope I
caught all the agencies there — CMS, ASPE, NCHS, AHRQ, Census Bureau, SHADAC,
RWJ. I just pulled a couple of numbers from the phase five research results,
which are also public. For calendar year 2005 they found 24.8 million people
correctly reported that they were enrolled in Medicaid.

There were 18.7 million false negatives, in other words, someone who was in
the administrative data as enrolled in Medicaid who did not report that they
were enrolled in Medicaid. There were 7.7 million false positives, people who
said they were enrolled in Medicaid who were not. Some of that was confusion
with Medicare. They came up with an adjusted undercount estimate of 32 percent.

Why don’t people report their benefits correctly? Here are a few possible
reasons. In the case of the ASEC data, one possible reason is the long
reference period. If the last point you got benefits was last January and
you’re being surveyed in March, you may forget. Confusion between programs —
we think there’s confusion between SSI and Social Security, between Medicaid
and Medicare, possibly between TANF and general assistance or state and local
assistance programs. Stigma is also a possibility.

Interview fatigue — you’re just tire of answering questions. You’ve caught
on to the fact that if you say yes to one more question, there’s going to be
six more, so you say no. There is a lot of ongoing research on this topic. I do
not want to pretend for a moment to be the person who knows the most about this
stuff.

Census Bureau has a follow-on to that project. The Westat and the Urban
Institute has done work. There has been a lot of cognitive testing work on many
aspects of this. These exact match analyses are becoming more frequent, and
we’re continuing to learn more about who is and who isn’t reporting.

Just a couple of brief comments about allocated data. Just to prep this for
a second, one of the speakers on the earlier panel mentioned that there are
many different kinds of users. There are very knowledgeable users, people in
government agencies, people in large research organizations, who may be well
aware of the extent to which allocated data are a part of what they’re
analyzing.

I think there’s also a lot of users who just look at a survey like the SIPP
or the NHIS and say look, noticing data. I don’t think it occurs to all users
that what they’re analyzing was not actually reporting by the individual in
many of these cases.

This slide is just looking at the impact of allocated data on what we’re
looking at in the surveys for three particular low-income programs. The
percentages are the CPS-ASEC annual dollar amount as a percentage of the
administrative target. For TANF if we only look at the dollar amounts that were
truly reported by the respondents as TANF income, that’s 40 percent. If we add
in the allocated amounts, we get up to 57 percent. In the case of food stamps,
the figures are 43 and 55. SSI gets the highest numbers, 60 percent without
allocation and 82 percent with allocation.

I think that the Census Bureau does an amazing job of doing the allocations,
and the other data-producing organization. They have hundreds of area goals
that they need to fill in. They cannot spend a year on each one of them.
However, as a data user, we do see some issues with the allocated data,
particularly for low-income families. I put down three of them here.

One issue is that we’ve seen that people who are allocated to be enrolled in
a program, in other words, they didn’t answer the question, but the
data-producing agency has zapped them with a yes through hot-decking. People
who are allocated to be enrolled in a program are less likely to appear
eligible than actual reporters. If we run a very complicated, very detailed
microsimulation model, TRIM, to assess eligibility, we get a much higher
eligibility rate among the people who truly reported the program than among the
people who were allocated to be enrolled in the program.

In some cases it looks like the allocated data for the benefit programs is
inconsistent with known policies. It’s very easy to see how this might happen,
because the hot-decking cannot be sensitive to all the very detailed things
that are going on. I’ll just give you one example of that.

States have highly different earnings disregards for determining eligibility
for TANF. In one state it might be possible to have quite high earnings for a
portion of the year because a large percent of those earnings are disregarded
as an inducement to TANF families beginning to work. Other states do not have
nearly that generosity in their earnings disregards.

If someone who failed to answer the TANF question in a low disregard state
is hot-decked to someone in a high disregard state, you could wind up imputing
TANF receipt to someone whose earnings would simply not make it possible for
them to receive TANF in that low disregard state.

It is generally recognized that allocated income amounts can also make a
person who actually reported a benefit appear to be ineligible for that
program. When we look at things like who looks like they’re eligible for the
Medicaid program and compare that to did they actually say that they were
enrolled in Medicaid, what we do see not infrequently is someone who actually
reported Medicaid.

They don’t look like they’re eligible for Medicaid. Their income just looks
too high to be eligible for Medicaid, but then when we look more closely, we
see that their earnings were actually allocated. So they truly reported
Medicaid, they didn’t mention their earnings, and they were allocated to have
an amount of earnings that is inconsistent with the fact that they — I’ll stop
because one person is nodding and saying that that is making sense.

Implications. In the aggregate the impacts of misreporting and allocation
may not be significant. It may not make more than a couple of tenths of a
percent of difference in the overall poverty rate or in the income
distribution. But the impacts for specific studies for subgroups where many of
us spend their time — not even in the weeds, but one little corner of the
weeds — can be substantial. Just to state the obvious, even though we use
allocated data as though it’s the same as the truly reported data, it’s really
not.

My wish list. If I were queen, this would be some of my wish list, because I
read the instructions and it said that we could provide a wish list, so I’m
taking that opportunity. ACS, I would ask more individual income items. As
someone filling out, I’m not sure that I would fill that it was more burden to
see more boxes, given that I can skip over them if they don’t apply to me.

If I were queen, I would reinstate the question on work-related disability
as being perhaps more tied to some of these programs than the ADL limitations.
Also, asking if a household lives in public or subsidized housing, which is
very key piece of information for a family’s economic wellbeing.

CPS-ASEC — I haven’t spoken earlier about any education issues, but
thinking about just, in general, issues around SES — and I know that your
interest is not only in the short-term, but in broader, longer-term issues —
being able to identify in the ASEC data individuals who are combining school
and employment at any age, not just 16-24, rather than only being able to see
school attendance or evidence of school or training if that’s a reason why
someone is not working.

My general wish list is continued cognitive testing. Every dime that the
Census Bureau and other agencies put into cognitive testing is a dime well
spent. Why don’t people report all of their income and benefits? It’s not the
same to have people not report it and to then make it up afterwards.

The more that we can understand who we can change the question ordering, the
question wording, so that we get more of that information from the get-go, the
much better situation we’ll be in. That work is ongoing, and I hope that
whatever is learned from that can then be used in multiple surveys, not just
the ASEC.

Continued refinements to the CATI/CAPI systems, possibly more prompts or
checks when someone says something that is inconsistent with something else
that they have already said. Finally, refinements to allocation methods. We are
never going to get to the point where everybody answers every question, much as
we would like to, but refinements to those questions, recognizing that the
Census Bureau can’t spend an enormous amount of time on any one question, but
possibly trying to think of ways that those allocation methods can be refined
to avoid some of the apparent logical inconsistencies that currently appear.

DR. MAYS: Our next presenter is Susan Queen, who is actually the lead staff
for this hearing. Dr. Queen is with ASPE.

DR. QUEEN: I am just sort going to sort of wrap things together in terms of
what we’re looking at for SES, relating it back to the standards that were
recently adopted for Section 4302 of the Affordable Care Act. Section 4302 has
special provisions related to disparities and specifically listed the variables
of sex, disability, race, ethnicity, and language as requirements to be adopted
within two years by the secretary.

The Data Council took charge and had an implementation work group that
involved representatives from OMB, from Census, and then across HHS to consider
how to implement such standards using the ACS primarily as the model for the
data collection, since between the CPS and the ACS, these were the measures of
the official statistics. Standards were adopted in October of 2011, announced
at the ABHA meetings.

These are the variables. Race and ethnicity, the requirement for that
variable was that it still had to be able to conform to the OMB standards that
had the five minimum categories. The point of the standards was to come up with
a minimum standard that could be more easily complied with by the surveys. It
wasn’t meant in any way to limit data collection.

The middle section of Asian, Indian, Chinese, Filipino, et cetera, all of
those can roll up into the one Asian category that OMB specified. The Native
Hawaiian, the four at the bottom, they roll up into the category of Native
Hawaiian or Pacific Islander.

This is the same with ethnicity, which was required under the OMB standards
for ethnicity. Hispanic, yes or no. Here they’ve just expanded the categories.
Again, they roll back up into the minimum.

Sex was biologic sex or sex at birth. Then how well do you speak English, H5
and above. Then the disability questions came from the ACS.

The point is they were construed to be a minimum and not in any way to be
limiting the data collection or the granularity that will be collected from the
federal surveys. They can collect more data as long as it can be rolled back up
into these categories.

The standards that could be applied to HHS national population health
surveys, specifically those surveys where the information is being collected
from the respondent, they’re self-reporting, or you have a knowledgeable proxy
reporting for the family or the individual.

Fritz mentioned not having a date for implementation. In this case we don’t
have a date. Implementation is to be carried out by the agencies for new
surveys if they’re developing new surveys, go ahead and include these within
the new survey. If you have an existing survey, we don’t want to interrupt or
interfere with the routine data collection, so when there are revisions being
made and perhaps at the time when surveys are going back to OMB for their
clearance, that’s the time to implement the new standards.

A monitoring tool has been developed to keep track as agencies are
implementing the standards to the surveys. It was disseminated through the Data
Council. It just asks each OPDIVS to report for each of their survey
instruments about how are you collecting these questions now? What’s a
timeline? What’s your schedule for implementing the new standards?

Common language, et cetera. I do want to mention something that Nancy said
this morning for HHS surveys. For the majority of the major surveys like NHIS
and NHANES, NSFG, and MEPS they’re either complying with the standards already
or getting greater information than what’s contained in the standards,
particularly for race and ethnicity.

The one area where I personally think there may be a challenge relates to
the questions on disability because the questions have to be asked using the
wording and response categories that are in the ACS. So a survey that gets
considerably more information or different information, like say you have a
disability that inhibits your ability to work, they’re going to have to ask
these questions. They can tree off from those questions, but it’s going to be a
change for some of the surveys, a big change for that particular area. But for
the most part, they already are getting the information as the ACS is.

Jennifer talked about the Federal Committee on Statistical Methodology and
the Interagency Council on Statistical Policy. Some of the work groups already
do a lot of the coordination and examination of how we ask what we ask. There
are also a lot of informal collaborations going on within the department and
across the federal community to look at research methodology, research
questions, how we’re asking what we’re asking.

A lot of the presentations over the past few days made references to the
work that’s been ongoing between Census, HHS, Labor. I think as we move forward
and we’re looking at the variables with SES, that kind of work is going to
continue. I think we want to encourage it because that’s one of the best ways
for us to benefit from data sharing and exchange from information and who’s
doing best practices with regard to asking the survey questions themselves.

If people are interested in looking at the data standards that were recently
adopted, these two links — the first one’s from the HHS Data Council, the
other one from the Office of Minority Health. Both of these provide information
about the implementation guidance. I think the OMH website has more information
on the background of the process that was used, the public comment period. I
think it may even include some of the comments that have been received.

DR. MAYS: Thank you. Let’s open this up for questions. I was actually
concerned about the data in your presentation. I want to look at the slide that
you have that says CPS-ASEC underreporting reported enrollment versus program
data. You said there’s truly reported and allocated enrollment. I didn’t
understand what that is. Can you tell me what that means?

MS. GIANNARELLI: What this particular slide is looking at is the information
that a user of the public use ASEC data would see, a user who’s using the
public use ASEC data, not paying attention to the allocation flag, but using
the public use ASEC data. I’ll just take the first row as an example.

There’s a variable that says whether someone has TANF income. Then there’s
actually another variable that says how many months they received TANF income.
That variable is also in the data. You can use that information to say
according to the public use data, what’s the average monthly number of families
receiving TANF. We think we know from administrative data the true number of
average monthly families receiving TANF if you just do that comparison that the
survey’s getting 58 percent of the average monthly caseload.

DR. MAYS: I guess what I was trying to get at — because the issue of
imputation has been something that we’ve been hearing. For some of this data
that’s what I’m trying to understanding, because it has implications in terms
of the income. You used the words truly reported and allocated enrollment. I
was trying to figure out whether or not this is truly reported and imputed
data.

MS. GIANNARELLI: Both of those are included in the denominator for this
particular slide. In this particular slide the numerator is both the truly
reported and the allocated. Even after you add in the allocated, the ASEC is
only capturing 58 percent of the average monthly TANF caseload, 58 percent of
the SNAP caseload. These are unpublished figures, but it’s not inconsistent
with other findings from other years and other surveys. SIPP figures would be a
little bit higher, but we’re not talking 80 percent for TANF and SNAP.

DR. MAYS: Can I ask about the one that’s called SNAP Exact Match Study in
which you say there’s 833,000 correct reporters, 922,000 false negatives, and
92,000 false positives? Can you talk to me about the quality of the data? Also,
this says weighted number of households. If it was actual number of households,
do you have a sense of what the numbers would be?

MS. GIANNARELLI: I should say that this is not my project. This was done by
Julie Parker at the Census Bureau. I have on the last page the citation to her
paper, which is public. These are numbers directly from her paper. I don’t
recall if she actually had the unweighted numbers or not. I think this is
useful more for the relative magnitudes of the correct reports versus the false
negatives versus the false positives.

I should say, since we’re focusing on this slide, that one thing that’s
going on in this particular study, since it was only a three-state study —
again, I’m sure getting data for three states was challenging enough, let alone
50 plus DC.

But one thing that’s going on here is that, for instance, some of the false
positives — someone could have received their SNAP in another state and they
wouldn’t be there. As I’m sure everyone in this room is aware, there are all
kinds of detailed methodological issues that are going to affect the exact
numbers that are here, but given these overall magnitudes —

DR. MAYS: So it’s the magnitude we should focus on.

MS. GIANNARELLI: Right. I think it’s the magnitude of there’s a whole heck
of a lot of false negatives and there are even some false positives. Regardless
of people moving from state to state and all those other issues, I think the
main point I was trying to convey with these two examples from SNAP and from
the SNACC Project is simply that for reasons I don’t think we completely
understand, a lot of people who are enrolled in these programs are not
reporting it in the survey data.

We would all have the same list of what are the 10 or 12 possible reasons
for that, but I think the work, to me, that’s very exciting is the cognitive
testing work that can be done by the statistical agencies. Forget the six items
or twelve items on my list of what it could be. What’s really important is
let’s do more cognitive testing to figure out what is it really so that then we
can change it.

DR. MAYS: My last question is on your last slide, issues with allocated data
for low-income families. There were two things. Could you help me understand
TRIM a little bit better? I was asking Dr. Queen over here about it. Then in
the slide who are these people? It says issues with allocated data for
low-income families. Can you give us any other parameters of these families? Is
it certain areas? Is it certain racial groups? Is it certain age groups? Is
there anything else we can learn about who these are?

MS. GIANNARELLI: To make sure I understand the question, you mean the kinds
of cases where we’ve seen these apparent logical inconsistencies?

DR. MAYS: I am looking at your last slide. I’m trying to understand people
allocated to be enrolled in a program are less likely to appear eligible than
actual reports. I’m trying to understand who might that be. And allocated data
for benefit programs may be inconsistent with known policy. Allocated income
amounts can make a person who actually reported a benefit program appear
ineligible. I’m trying to understand who those people are that this is not
happening for.

MS. GIANNARELLI: It’s really talking about two different kinds of cases that
we observe in the data. One kind of case that we observe in the data is someone
who truly reported their income. They skipped the questions on, let’s say,
whether they were enrolled in Medicaid. They just didn’t answer it. Whoever
they were hot-decked to was enrolled in Medicaid. So now in the public use data
they’re a Medicaid enrollee. We see a fair number of those cases where if we
run this very detailed simulation model, we can’t see how they’re eligible.

Full disclosure, any microsimulation model also finds true reporters who
appear to be ineligible, because there are limitations of simulation models.
But it is much more prevalent that people appear to be ineligible among those
who are allocated to be enrolled as opposed to those who truly reported
enrolled. I think the shorthand for that case would be truly reported income,
allocated to be enrolled in a program, but they look too rich or they have too
much asset income or something like that.

The other kind of case we see is sort of the reverse of that. They truly
reported program enrollment — Medicaid, SNAP, SSIS — they skipped some of the
income questions, and whoever they were hot-decked to had a certain amount of
income, and that income looks like it would make them ineligible for the
program. It’s kind of the reverse.

Obviously there are lots of complications in people who have missing data in
both areas, but I think those are the two kinds of inconsistencies we see in
terms of what they truly reported versus what was allocated and appears
inconsistent with what they truly reported. John knows more about all of this
than I do.

DR. LUCAS: I have a question about whether or not in any of the work you’ve
done you’ve ever actually compared allocation or imputation methods across
different surveys to see whether a single imputation versus a multiple
imputation produces different levels of disagreement. The problem that you’re
describing about people being sort of false positive or false negative —
another way for me to ask that question is has there been any look at the
effects of the type of imputation on the likelihood of that occurring?

MS. GIANNARELLI: That is not something that we would be doing at the Urban
Institute. John may know whether that work is going on.

DR. QUEEN: Joan Turek is working with Chuck Nelson looking at item versus
whole imputation, but not across —

DR. CZAJKA: I have looked across surveys at differences. There’s a striking
difference between CPS and SIPP in what appears to be happening in the
imputation of welfare and food stamps and benefits like that. You see much more
of the kind of inconsistency that Linda was talking about in SIPP where
high-income people have food stamps.

I don’t have anything with me, but I had some tables that as you went up at
the higher income levels and looked at what the portion of the food stamp
values that were imputed versus reported, and they’re mostly all imputed at the
high-income level in SIPP, which indicates there’s a problem. This is something
that was actually identified many years ago. They’re using the same general
methods.

DR. LUCAS: That was my question, because yesterday there was some discussion
about single imputation versus multiple imputation. I’m just wondering whether
the imputation methodology itself is having some effect on what you’re seeing
in the data.

DR. CZAJKA: I think it’s more important what variables are going in.

MS. GIANNARELLI: This has probably gotten more challenging over time, as far
as allocation for benefit programs, as there have been increasing variations
across the states in the administration of these benefit programs. There was a
time not too long ago when across the whole country the asset limit for AFDC
was $1,000.

Now there are states where you could legitimately have interest income that
suggest you have a few thousand dollars in the bank and still get TANF in many
states, although not still in some states. It may be a case that that may be
part of a problem getting more challenging over time.

DR. CZAJKA: Let me just add to that. The American Community Survey, because
it’s so large, actually does all of its allocations within the same state.
That’s a way of controlling for those differences.

DR. BREEN: Thank you very much for the very informative talks. They were
great. The work you’ve done is difficult and much appreciated.

MS. GIANNARELLI: Most of what I presented was not mine.

DR. BREEN: In summary, it sounds like the earnings data, we’re doing a
pretty good job of collecting that part of income, and that’s about 80 percent
of what we need to collect with income. I guess we don’t need to worry about
that, but if we do, you can say something about that.

In terms of the non-earnings income except for dividends and interest, in
other words, what you might call transfer payments or social welfare benefits,
SIPP seems to be the best source. They collect the best data. That was the
take-home message, especially from John’s talk, but Linda’s wasn’t inconsistent
with that. That’s where we stand.

Vickie was asking about the non-earned income. I wanted to follow up on that
a little bit. I think when you said imputation and allocation, this is the same
thing, isn’t it?

MS. GIANNARELLI: Using the Census Bureau terminology, there’s
“allocation,” and they use the word “editing” for certain
logical edits that they do. I guess we could combine both of them as some sort
of imputation. There’s the sort of hot-decking and then there’s editing.

DR. BREEN: You emphasized that you thought maybe one of the most important
things to do would be more cognitive testing on how to get these questions
right. But kind of implicit in what you said was that there is a lot of
information out there that maybe could be incorporated into the CAPI or the
CATI about actual limits that, as John said, sort of happen automatically with
the ACS because they’re only using data from the one state, so they’re not
going to end up making these mistakes, but to actually add in information about
these programs into the CAPIs or the CATIs so that these errors are likely to
come up at the interview point rather than later on.

MS. GIANNARELLI: I think if someone says something, then I think the
CATI/CAPI can do some sort of check. If you say you’re getting public
assistance benefits and you say that you received $20,000 in public assistance
benefits, then a check would be warranted unless you have eight children.

DR. BREEN: I don’t know how feasible that is, but I was just thinking there
may be checks that could be incorporated into the question process, because it
seems like what we want to do is improve the reports.

The third that struck me is training interviewers because we’re all trained
to believe that you can’t ask about income and get a very good response rate. I
think the interviewers go in with that point of view too. Maybe it would be
useful to spend some time and effort thinking about how we collect those data
and kind of turning it around with the interviewers so that they don’t feel
that it’s a sensitive question, because it doesn’t necessarily have to be.

The last thing that I’d like to hear a little bit more about is retirees.
They’re a growing proportion of the population, which I guess is why they’re a
growing problem in terms of collecting this information. But you also, John in
particular, mentioned that the kinds of benefits available to retirees are a
lot more complicated than they used to be. Is there something that we could use
as guidance for how we should be collecting that information going forward?
Does the Health and Retirement Survey to a particular good job? Is there an
example out there or some guidelines or recommendations you could suggest?

DR. CZAJKA: I haven’t done much work with the Health and Retirement Study.
It’s challenging data to use, and we did some work a few years ago that
produced some very high estimates of income for the elderly. I’m not sure we
got their data right. That was part of the challenge of working with it.

One of the things that we suggested with respect to retirement income and
especially the retirement accounts is that there is certain language that’s
used to characterize money you take out of an account. It’s frequently a
distribution, withdrawal, but calling it a payment seems to communicate
something very different to the respondent. I think that that could help.

DR. QUEEN: Do you know of a survey that’s asking it a better way currently?

DR. CZAJKA: I’d certainly look at the Health and Retirement Study, but I’m
not positive that they do. Actually, the Survey of Consumer Finances is one I
hold up. The Health and Retirement Study people have a different view of that
survey, but that’s one that’s connected by the Federal Reserve Board. Its focus
is assets. The interviewers are trained to collect asset data. The respondents
know this is going to be an excruciating interview about your assets and your
income.

DR. BREEN: But they balk at the health questions on that survey.

DR. CZAJKA: They do very extensive imputation, very careful in that study.
One of the things that they do on the retirement side is that their approach is
to go by account as opposed to a type of income, laying out the different
accounts that you have, and then collecting asset holdings for each of those
accounts rather than having people try to make a distinction between different
types. There are many different names for these things, and they all kind of
act the same. But there is a lot of that potential.

There’s real ambiguity. We had a conversation earlier about the standard
interview prompt. What do you mean by that? Whatever it means to you. That’s
not what you want in collecting these data. You want them to tell you what you
mean. That’s a very different approach.

DR. BREEN: That may be an interviewer training issue too.

DR. CZAJKA: Then I mentioned that there is this conceptual issue about what
is income and what is savings and what is taking something out of savings. As I
said, the difficulty that’s been created is we moved our whole way of financing
retirement into something that’s now savings. There’s a reluctance on the part
of the surveys to collect income that way.

The ACS very explicitly says do not include withdrawals from savings. Then
how do I report something I took out? They don’t mention the word 401(k)
anywhere in the ACS, little things like that that indicate a failure to come to
grips with what’s really going on in retirement income.

DR. MAYS: First, let me thank you because this has been very enlightening.
Some of it has been wow in terms of things that we learned today, and it’s very
helpful to the work that we need to do around income. I want to thank you for
your time and putting together really good presentations that were very
enlightening to us. We’re going to break at this point for lunch.

(Whereupon, a luncheon recess was taken.)

A F T E R N O O N S E S S I O N (1:40 p.m.)

Agenda Item: Committee Discussion

DR. MAYS: What we will do is start to plan what our next steps are. We’ve
heard a lot over the past couple of days. I think we have a lot that has been
presented to us. But I think where we want to start is really to talk about
what is the product that we want to produce. I’m going to start us off with a
suggestion, and that is that since we’re talking about the June meeting, that
we focus our efforts on getting a letter out.

We had talked before about the issue of a report in a letter. It may be wise
for us, given the complexity of what we heard, that we get together very simply
with what we need in terms of thinking first about a letter. If that’s just
such a slam dunk, we can always back into a report. I’m going to put that on
the table for discussion.

DR. KAPLAN: Maybe it would be valuable to talk a little bit about health. It
would be valuable to talk about what we learned and where we think we need to
go. First of all, I thought this was really good. The stuff that we heard was
very well done. It seems to me that there are a couple issues that surfaced.

One was to what extent are the agencies getting together to harmonize? We
actually heard a lot about the 12 agencies talking to one another. I think that
we need to think a little bit about whether or not that’s accomplishing the
goal of harmonizing if it’s agencies talking to one another as opposed to
providers of information and users of information getting together to make sure
that they’re producing the right information. The second thing that I was very
intrigued by was the methodologic research component. To what extent are we
building the right research agenda to move this forward for the future?

DR. MAYS: Let me just comment on why I was suggesting that we start with the
discussion of a letter, and we can change a bit. My concern is that we have a
very specific task that really is about the minimal standards. I want to make
sure that we can meet that first, and then all these other things.

If you remember, we have a long-term and short-term goal. We’ve talked about
whether or not there would be pieces that we could continue with. What I want
to try to make sure of is that we can get us through because I think it’s a
very difficult process of deciding whether or not we have something to say
about the recommendation of a very specific minimum standard.

I want to make sure I acknowledge this. I think the other issues are
important to that, but at the end of the day, if we’re going to make it by the
June meeting, we really need to first get some kind of consensus among the
committee as to do we think that there is a minimal standard that we want to
recommend for each of the three variables.

I think the other challenge that was put before us is — and I think we can
deal with it quite easy — why or why not social class. Then I think everything
after that is pretty much gravy. I just want to hear what others said, because
June is around the corner.

MR. BURKE: Things I recall hearing when we were together in Washington a
couple weeks ago from Jim, who described what he thought of as the design
outputs of the hearing would be, obviously as you said, what are the minimum
standards? Should they remain or should they be changed? But beyond that, it
was about what a higher vision concept should be. It wasn’t just the as-is, but
it sounds like the request was what the to-be should look like.

DR. GREEN: We opened the hearing with this. We opened the hearing this
morning with it again. Our questions for the hearing were three. What is the
state of the art and the standards for collecting data to measure SES in
federal surveys today? Second was what are the variables that are collected?
Three is what opportunities exist to standardize these variables across federal
surveys?

We learned a lot about one and three and the background work that Susan and
others have been doing pertinent to this. But to follow Vickie’s lead here, we
also heard that the timeline for this is to bring something forward to the
queue for the NCVHS meeting.

I’ve been struggling in the last hour just trying to sort a lot of what
we’ve heard here. As I was saying to Susan, forgive me again for being cursed
with the brain of the doctor, but I feel like we had an acute problem that we
were assigned to take on. As we evaluated the acute problem, we discovered that
the patient has about seven chronic diseases.

When Vickie divides this up into a paper or a letter or whatever, that may
mean something different to you than it does to me, but that makes sense to me
in that if we head toward a letter and we meet our timeline, it’s really been
more narrowly focused on these particular questions.

At the same time, it makes no sense to ignore what we’ve learned. There are
a lot of issues that are as pertinent to other aspects of federal surveys as
they are to SES, that have rolled out in this. A lot of them are process
issues. It seems to me that we don’t have to restrict ourselves to one product.
If I heard you right, you’re saying maybe we need two. Maybe there’s some sort
of report or paper that comes out of the work, and then maybe there’s also this
specific letter that responds to the questions we were asked to address. Am I
tracking with you?

DR. MAYS: I think what I am talking about is the time to do the job that we
need to do. I think that the complexity of just deciding if there’s a minimum
standard is actually one that’s difficult to do, even in the time that we have.

I’m just concerned that we’ve heard a lot of things, and if we discuss all
these other things, which I don’t think we’re going to lose, and I think it
would be great when we have the other committee to bring them up to speed on
that. But I think in terms of deciding about a minimum standard, particularly
given what we’ve heard, is a bit of a onerous task because we’ve got to come
here among ourselves to some agreement, and then we’ve got to write a letter
that’s compelling enough to support that agreement.

DR. BREEN: This is an area that is near and dear to my heart and something
I’ve been thinking about my whole career. I found this whole discussion very
illuminating, but I was also really trying to stay on task because you can go a
lot of places with this information and really kind of go wild thinking about
it.

I think we learned that for education there’s a very clear recommendation of
what to do. I’d have to get back on the notes, but there were two surveys that
were doing it fine and two surveys that weren’t doing it fine. I think we have
the recommendation for that. I think NHIS was doing it fine. I’m not sure
everybody agrees with that, but I can bring out my notes and we can talk about
that.

For occupations, which I thought was going to be a nightmare, it seems like
that also is being consistently done. The problem there is with analyses. There
are many ways to do that, but in terms of data collection, that’s also pretty
consistent. I went over and asked the speakers afterwards, I said you’re
collecting different numbers of occupations. Is that a problem? No, they all
back to the SOCM. These are fine. They’re consistent.

I don’t think there’s a problem there either. I think our recommendation for
that is also clear, because if people are collecting less, they’re collecting
fewer occupations. Obviously you can’t get more out of less, but they’re
consistent in the way they’re being collected.

I think income is more of an issue. There are eight questions that the ACS
uses which seem to be the basis for most of the surveys. I think what’s unclear
is whether those eight questions are included in the MEPS, because the way that
was presented, it sounded like that might be different, or maybe it just might
be more information being gotten. I think that’s some research that we need to
do to see whether that’s consistent or not.

The two areas within income that I think need attention are the poverty
indicator. Right now we’re all using the simple poverty indicator, which is
income based on age of head of household and number of people in the household
who are living in the household. That’s the Census way of doing it and that’s
what NHIS does as well. Connie Citro has developed another way of measuring
poverty that includes transfer payments and costs of childcare, which we may or
may not want to recommend, this more complicated measure, but if so, that’s an
option.

I think the other thing is we heard that there needs to be some work on how
to measure non-earnings income. I think we’re particularly interested for
policy purposes of transfer payments of benefits that people are getting from
programs, but then interests in dividends, that doesn’t seem to be too accurate
either, then finally information about retirement income, that that area is
probably the one that needs the most work and is particularly important going
forward because that’s a group within the population that is growing.

MS. GREENBERG: I found that summary very helpful, since I wasn’t here
yesterday. In some ways it begins to answer some of my questions because I
wasn’t here yesterday. I’ve looked at some of the slides, but also I’m sure the
discussion was as rich or richer than the presentations.

I will just say it looks like those of you who worked on this put together
really a terrific day and a half. I thank you, and I was particularly pleased,
even with some of the limitations of meeting at NCHS, that you did meet at
NCHS, because I think there was quite a bit of interest among NCHS staff,
particularly yesterday. I guess what I heard today, too, were all the problems.
That’s why it was helpful to hear the things learned as all over the place, as
I might have come away from feeling just listening today.

I have a few questions. Other than checking out a few of these things like
you mentioned with MEPS or whatever, to what extent are the gaps in your —
there will be gaps in our knowledge on this subject until the end of the day,
but gaps to try to get some closure on whether you can make a recommendation to
the specific issue.

Another question I have is was there anybody who was saying just collect
education or just collect income or just collect occupation? If not — and I’m
assuming not — then would a minimum standard have to be a minimum standard for
each one of those, although when they say all federal surveys, I suppose there
are some that aren’t collecting any of those. We know the major ones you had
represented are collecting some, obviously, of this, and probably the major
ones all of it.

Is there any hierarchy? If you aren’t really collecting any of this now —
and I don’t know if there are any surveys that aren’t, but of the three, is
there some kind of hierarchy, or do you really need to collect some minimum
standard for each one of those three? I’m assuming you were talking probably
about coming out with some kind of recommendation on a minimum standard for
each of the three.

DR. QUEEN: I have to, unfortunately, leave in about five minutes for a
meeting. I would be hesitant about making an actual recommendation for using a
specific survey.

MS. GREENBERG: For what?

DR. QUEEN: Specific survey items as a minimum standard at this point. We
certainly heard that some surveys are doing it better than others. NHIS and
NHANES and MEPS are fairly closely matching on education, the ACS and CPS, for
example.

I didn’t think that the committee was actually going to be looking
specifically to make a recommendation in this letter of a specific
questionnaire to use as a standard, because we haven’t looked at the way the
questions are worded yet. Yes, they’re all collecting the occupation variable
and industry occupation codes, but the way they even ask about, first, are you
currently working. The capturing of whether or not somebody is working,
unemployed, working multiple jobs, part time.

MS. GREENBERG: Or unusual occupations. At the end of the day, there’s got to
be a specific set of questions, just like for the disability.

DR. QUEEN: But I do not believe that the committee was necessarily charged
with coming up with the final determination of what the standards themselves
would be.

PARTICIPANT: Then what was the committee charged with?

DR. QUEEN: Beginning the work that’s necessary to move forward in looking at
the surveys and determining which ones seem to be capturing information in the
best way, which ones are most applicable for the health surveys, for example,
which ones we can look to for best practices, why they’ve changed, to begin the
background work that would lead to the development of some more harmonization
coordination, but not to say this is the survey you’re going to use.

MS. GREENBERG: I thought the goal – but I’m not sure that you can get
to that goal by June by any means, but the goal was to assist the Department in
identifying a minimum standard for collecting SES in federal surveys.

DR. QUEEN: I don’t think the committee could actually say by June, without
having some sort of interagency committee or work group such as we did for the
HHS data standards involving Census and OMB together working, I do not believe
that that could be done.

MS. GREENBERG: I don’t know whether it can or can’t be. I guess that’s for
people who were here the whole time and your thinking on it. I think this was
supposed to be a little bit more than just gathering background information.

DR. QUEEN: Yes, but I did not think by June, I think there was a recognition
– Jim was talking just yesterday about coming up with a letter to move things
forward and to make recommendations, but I don’t believe that we would have a
set standard that was finalized.

MS. GREENBERG: Anything that this committee recommended then would have to
be vetted through the Data Council through whatever the process is. This
committee only makes recommendations. I don’t think the committee should
recommend anything that they don’t feel they are in a position to do. I think
that is the question. What will come out of this or any follow-on in the next
month or so?

I think you mentioned to me, Susan — and maybe I shouldn’t say this — that
June wasn’t in stone. The question is can you say anything by June, and then
what would you say?

DR. MAYS: Let me comment to bring us, since Susan has to go, to what I think
are some of the critical issues. First of all, I think if we go around the
table, we’re going to see that we’re going to disagree as to whether we think
everything is — I already know that.

Secondly, there is this additional work. For example, we do need to look at
some of the materials they’ve been preparing, such as are the tables that show
you all of the questions — what I’m thinking is there’s a lot of work to be
done. I think maybe in another month and a half we are clearer about where
we’re going with it, but I think at this point in time I don’t know how much
agreement we have. We have several people who aren’t here, so you already know
how that process is.

My concern is that I understood that we were making a recommendation and
that that recommendation, just like if NCHS or somebody else made the
recommendation, has to go through all of these different agencies to look at,
but that we would not make a recommendation if we did not think that we could
make a reasonable recommendation. The steps that I think it will take to make a
recommendation are the following. Susan, you have to help me before you go. One
is we need to look at the surveys that this would apply to. I think there are
16.

DR. QUEEN: There are 16 surveys that we were looking at, but they are beyond
HHS.

DR. CAIN: I think that’s a point that needs to be made. This is not a
recommendation for all federal surveys. This is just for HHS.

MS. GREENBERG: The other recommendations, the other 4302, were those just
for HHS?

DR. MAYS: Yes, they’re just for HHS health surveys. We have the list of
which surveys we’re talking about. We have the three variables that we’ve been
talking about and whether they’re collected. We have a URL that takes us to
that survey. I think the next thing that Susan had indicated she needed some
help with was actually to pull the actual variable itself so that you can see
it, because apparently that’s not an easy task to do.

DR. LUCAS: I just want to mention that even within the scope of just HHS
surveys, this report was released in maybe 2004 or 2005. Actually, we only
scratched the surface of HHS surveys. This covers the full spectrum of HHS
surveys. We haven’t even tapped into the 19 that they’re including. There are a
whole bunch of CDC surveys that we haven’t even tapped in that will be affected
by this standard because they are conducted by HHS.

MS. GREENBERG: I don’t think that we have to do all that.

DR. LUCAS: I was just saying there are many more surveys that are affected
by this that we have had the time to really cover or discuss in great detail in
the course of these deliberations. I’m just saying that there’s a bigger
picture.

DR. MAYS: Can you give us a sense of what other surveys? Because I guess I
thought that our domain were those 16, 19, so if it is beyond that. Is it
everything CDC does as well? Can you give me a sense?

MS. GREENBERG: Not some of the surveillance.

DR. MAYS: That is what I don’t know that is why I want to make sure.

DR. LUCAS: In the background document we provided the citation to the actual
language in 4302. I can make the specific text available to you. It covers all
HHS surveys and programs and other data collection efforts of the –, so it’s
pretty broad in scope.

The reason why I mention this report is because they did look at (side
comments) across race, ethnicity, socioeconomic status, and one other thing.
They laid some of the foundation of what we’re doing in here. (Side comments)
disparities is the report of the National Research Council. It’s called
Eliminating Health Disparities: Measurement and Data Needs.

MS. GREENBERG: Isn’t it just population-based surveys? Let me ask Susan, are
you actually thinking that every surveillance system is going to ask all those
disability questions?

DR. QUEEN: I never said that.

MS. GREENBERG: You’re saying it covers much more, and when I said every
surveillance system, you said yes, but you’re not thinking every surveillance
system is going to ask those disability questions.

DR. QUEEN: We broadened it beyond HHS with the ACS and the CPS, and so we
were doing that just for exploration of who’s asking what and how in the area
of socioeconomic status.

DR. CAIN: I also think there’s a question of if you look at the language,
it’s all HHS-supported surveys. Does that cover all NIH grants then as well?

DR. QUEEN: It is only in the case where a grant is specifically directed to
conduct a survey. Think of it as part of the Paperwork Reduction Act.

DR. COOPER: What about the HRS, for example?

DR. MAYS: What about the National Child Longitudinal Survey?

DR. QUEEN: I don’t know.

DR. CAIN: It’s an odd situation that hasn’t been clarified, so in some ways
it could be expanded to include all NIH grants, in other ways probably not.

(Simultaneous comments)

DR. MAYS: Is it like a U or an RO1?

DR. CAIN: It is not specified in terms of the legislation, so Us and RO1s
don’t talk about that. It sounds like what you’re saying is if it’s a grant to
specifically do a survey, then it might be covered by that.

DR. QUEEN: If it is a survey that is getting governmental —

DR. CAIN: That’s what I’m saying. Those are Us. That’s an interpretation
that has been made.

DR. QUEEN: It’s in the implementation guidance that was put up on October
31st to try to help clarify for the agencies which surveys would
even have use the current standards.

DR. COOPER: Are you saying recommended the U method?

DR. MAYS: No. She is saying because they would be the ones that would go
through OMB clearance. I don’t think that most of the RO1s would go through an
OMB clearance.

DR. COOPER: I don’t think most of the Us do.

DR. MAYS: No, you’re right.

DR. BREEN: It sounds like we need to figure out the scope of our task.

DR. CAIN: HHS definitely, as opposed to all government surveys.

DR. MAYS: Isn’t it also HHS health-related surveys? Then let me pump it up
one more time, because I also thought it was HHS health-related surveys that
were population-based. If you look at what other surveys were impacted by 4302,
I didn’t see 4302, which went into effect in October, kind of migrate down. I
thought that the individuals who we’ve had come in and talk is the level of
people who’ve had to worry about it.

DR. LUCAS: In the background document it says the law requires that any
federally conducted or supported healthcare or public health program survey,
collect, and report basic demographic data, race, ethnicity, sex, primary
language, and disability status; data at the smallest possible geographic
level; sufficient data to generate statistically reliable estimates; standards
for the basic demographic; and any other demographic data regarding health
disparities as deemed necessary by the Secretary of Health and Human Services.
It is not limited to just population.

DR. MAYS: Is it at all possible for us to get Jim on the phone because I
just don’t believe when he spoke with us – he started our day off, he gave
us our charge. Jim would know that we couldn’t do something in that amount of
time.

MS. GREENBERG: A lot of things are like this. That’s what the language was,
that the Department has made the decision to start with federal surveys.

DR. CAIN: As part of the group that had been taking a look at developing the
standards, what we did was say that in some ways surveys were going to be the
easiest thing to do or that was a place to start. The standards were developed
for surveys recognizing that there are a lot of other programs or more clinical
activities that we may be able to move to next, but this is a place to start
with the surveys.

In the legislation it specified those five different areas. It also said
that the Secretary could look at other areas that might be relevant as well.
When we were developing those five different areas, we said that since it had
come up in several cases, that SES might be an area where standards could be
developed because people had been working on it for 30 years and we thought
there was enough work there that people could take a look at it. But the
committee that developed those particular items was not prepared to recommend
what the standards would be at that time. That’s where this activity came out
of.

MS. GREENBERG: I remember from a few years ago — I’m really not sure what
the current status is, but some of you probably know — that some journals and
others came out with policy that said you should not or could not report race
and ethnicity, without some type of socioeconomic variable because they thought
it was misleading. Maybe it was only one or two journals.

DR. LUCAS: You mean that results being attributed to race and ethnicity were
really capturing differences in SES, and so just attributing a particular
finding to differences in race was not sufficient?

DR. BREEN: AJE would have been the most likely to do that, but I’m not an
epidemiologist. I don’t recall that, but I know there was a point in the
community, including the cancer community, because we don’t collect SES on the
Cancer Registry data, where people said we’re picking up a lot of information,
we’re attributing it to race, and there’s a good chance it’s SES, so we really
should work hard in order to get some SES data. There has been a change in the
community. I don’t know if there’s a rule or a policy.

MS. GREENBERG: In any event, I think the decision was made to start with
surveys, and probably more population-based surveys. That’s the most you could
really address at this point.

DR. CAIN: I don’t think the intention was to necessarily limit it to, for
example, national surveys, which is what we’re talking about now. I think it
really is in the interpretation. It wasn’t specified in the language, but it
sounds like in the implementation and interpretation they’re limiting it to
data collected under contract that has to go through OMB. That’s okay, but that
also covers a lot of other —

MS. GREENBERG: Does that mean they’re not including BRFSS? That doesn’t go
through OMB.

DR. CAIN: Then I would say probably not.

MS. GREENBERG: I think Jim knows whom this applies to.

DR. CAIN: Because all we had was the legislation.

DR. MAYS: I think it would be great if we could get him.

MS. GREENBERG: I will check to see if I can reach him.

DR. MAYS: I think in the interim what we can do is maybe we can do what
Nancy did, because I’d like to see how much consensus we have. If it would be
possible to do that, that in and of itself would be helpful. Larry, what are
your thoughts about each of the variables and where we are? I think Nancy was
being very clear about what she thought about each of the variables.

DR. GREEN: The things I learned in the last day and a half that I think are
right on mission and right on target here — it was well said in the second or
third presentation yesterday around defining SES survey measures. It was just
such an elegant simple statement. SES is a latent variable predicted by
education, income, and occupation, modified by race and gender. I think we
should adopt that as our position. That was a great expert summary of decades
of work and a lot of literature.

DR. COOPER: Can you restate that?

DR. GREEN: SES is a latent variable predicted by education, income and
occupation, modified by race and gender.

DR. BREEN: Michael Hout said that yesterday. He’s got a little graphic.

DR. GREEN: I heard nothing from any other presenters to refute that. That is
a great organizing conclusion, in my opinion. Why are you talking about
education, income, and occupation? That’s because that is what it is. These are
the predictors of the variable that we’re interested in.

The second thing I learned is more synthetic. I feel a whole lot more humble
about measuring SES today than I did at the start of yesterday. I have an
abiding conclusion that’s just personal. I think we want to be very careful
with recommendations that get extremely explicit about what to do because what
I’ll just call ramifying implications, which I don’t think we understand and
have command of. That leads me to the third conclusion from the hearing.

I actually think that we can write a useful letter that’s responsive to the
charge we started with. We just need to have modest expectations of how
dramatic our recommendations can be and how far they can go. I think it was
either Nancy or Susan or Jack that said we teased out the different locations
in which this thing colloquially known as the statistics community about who
that is, where it exists, how they get together, how they operate, and this is
an elite group that we should be respectful of.

It could be that we find ourselves making sort of midlevel recommendations,
and then a process recommendation about where to locate the follow-up work. So
a fourth conclusion I’ve come to is I don’t think NCVHS, given all the other
requirements that are placed on NCVHS, Privacy and Confidentiality Standards
Committee — you know the routine here.

Even if this were all we had to do for the next couple of years, I think it
would eat us alive. I’m not sure we’re the best group of people to dissect it
all, but I do think that we’re in a good position to answer our charge. I think
we need to commend Vickie for assemblage of this group of people that she and
Susan and whomever else was involved did. They obviously hit a homerun in
getting the experts and people who know about this in the room to talk to us.

I think we’re obligated to distill out of that some findings and conclusions
about measuring SES. I am personally disinclined to decide that we don’t have
to do anything for longer so we can think about it longer. I think we ought to
shoot for the June deadline, try to adhere to the charge we got, see if we can
execute it.

I may be misfiring here, Vickie, from your perspective. I thought you
started us off saying we really need to think about the products we want to
produce. Are we going to produce a letter? Are we going to produce more or
something else? I believe we need to produce a letter. As time goes further,
there’s more work that is going to be in scope for us here. It’s got to be
balanced, our budget, our workforce, our staffing, and all that sort of stuff.
That’s my own distillation of the conclusions about this.

I really like Nancy’s specificity. My version to Nancy is there are people
around that know what the next best steps are to do about education. There are
people around that know what the next best steps are to do with occupation.
There is not consensus what the next best thing is to do about income. That’s
where I would come down.

DR. KAPLAN: A lot of thoughts in my head, so I’ll try to spit these out in a
coherent way, if that’s possible. Just reacting to Larry’s comment, yes, the
latent variable model is great, but when the rubber meets the road, you have to
know what the coefficients are to get SES. In other words, conceptually that’s
fine, but you still have to have measures of education, measures of income, and
so forth. Then you have to have a model that provides those specific
coefficients to come out with a score. We’re a long way from that.

What I was kind of thinking was what’s wrong in our country in relation to
Canada where they have Statistics Canada? We heard this several times. Given
that we’re not likely to go to their structure, how do we get to some system
that provides a coherent overview of these things, given that there are
multiple players?

My reaction is that I think that we do need some super-oversight. One
recommendation that you could put on the table is the meeting of the 12 people
on this committee. Is that the right way to go? Do we have to look at that in a
different way? If so, how would you pull it together and where would that be?
One possibility would be that this is something that could be given to the
Office of Science and Technology Policy, the OSTP, in the White House, because
it is across agencies and bigger than HHS.

The second reaction I had was it seems to me there are two different things
that you need. One is you need the measurement piece and some sort of
harmonization. Then the other piece is the methodologic piece. You have to make
sure that if you harmonize, you still continue to engage in activities that
will make things better in the future. I don’t know if that would be sort of a
separate task, but I do feel that’s important.

DR. COOPER: The information that has been shared over this last day and a
half has been unbelievable. It has let me know what all I don’t know, and that
when you look at those measures of SES, oftentimes we make assumptions about
what we read and what’s been reported. What’s been shared with us is that there
are so many other things going on that we really don’t know that have been
included in the inconsistencies in the reporting and what some of the variables
do mean as well as what they fail to include. At the same time, we know that
SES is definitely a strong proxy for health outcome.

Yesterday it was an unbelievable task, but I do think that it’s very
important for this committee to come up with some kind of recommendation. It
doesn’t have to be a recommendation that solves the whole problem, but it
should be some type of a recommendation that somewhat summarizes what happened
during this hearing and what the potential steps are in terms of laying them
out and the task that we have at hand, and that we do need others to join in in
this effort.

When you look at the implications, it is not limited to just surveys, but it
may include NIH and others as well. That’s significant. What we really do not
want to do is continue to collect incomplete data when we know that there’s a
better way to collect that data, because that data has significant implications
in terms of programs that are supported and people’s lives in terms of
treatments and et cetera. When we look at SES, it’s a significant set of
variables. I do think we have a responsibility to do something.

DR. MAYS: From my perspective, when we look at the opening panel where
Michael began to talk about what SES is, one of the things that was throughout
the day is that the reason it’s SES has a lot to do with social stratification
issues. I think that’s where a lot of the difficulty is.

We have to remember it’s not just about an education variable, an income
variable, an occupational variable. But if we’re being asked to talk about them
not as who they each are, but how they work within SES, I think it’s a little
different. From what I heard, I thought that we weren’t settled on education,
whether it’s educational attainment, whether it’s educational achievement,
whether or not in order for it to work in SES, we need it to work with a couple
of other things.

Now I’m giving my personal input here. In terms of income, I think that if
we thought about the issue of what is the requirement of some of these surveys,
they have such different requirements that coming up with a minimum way of
asking about income may be very difficult, particularly right now when there’s
a challenge to how we measure poverty.

We’re kind of crossing poverty and income. Those are actually two different
things. I thought Connie had some great suggestions about it, but poverty is
actually mandated to be determined by the Census Bureau on behalf of the
federal government. It’s almost like we’re not talking about the determination
of poverty. That’s not really, as I understand it, our charge, but the income
variable to SES is.

I almost reached the conclusion that there is not a minimum standard to come
up with for income other than to say to collect it in a certain way. I’m not
sure that I know exactly what that would be other than to say to make sure you
at least have it continuous so that people can use it as they need.

For occupation I was kind of surprised. I thought I was going to be in a
very different place with occupation. I think it was much more exciting to hear
what they said. This is not from having an opinion; it’s ignorance. I don’t
know that I’m ready without a little more work because, very honestly, I didn’t
do the same amount of work on occupation as I did on the other two. I think I
tried to give everybody access to the Widgeo(?) site where I put up about 15 or
so articles. I’ve read most of those. I didn’t read as much on occupation, so
I’m unclear about that.

If I had to say where we are and what I think will work for us, I think a
letter that talks about what we found, what’s important, and an opinion about
the standards might be the best, because the conversation that I had with
Virginia during lunch, to realize the infrastructure that you all had in place,
to come up with those minimal standards, I don’t think we can do that. But I
think what we can do is to make a recommendation about whether we think there
could or couldn’t be a minimum.

I think, Bob, you were saying this. There are some other groups, and I think
then it needs to go to those other groups. But in addition, do we need to opine
about the kind of place and support that those groups need in order to be able
to achieve what we see as important to achieve.

I think this gets into the point that Larry’s making about those groups and
working together. Maybe there needs to be a process, but I don’t know. I don’t
want us to start trying to create an infrastructure to know that SES is the
only thing that’s going to be done, unless the secretary has other things that
she’s also going to do. Then it may be that making this much broader
recommendation about reengineering these groups may not be cost-effective right
now. That’s where I’m kind of settling at this point.

DR. KAPLAN: Let me just put something on the table for discussion. It does
seem to me that the level of precision differs as a function of what you’re
using the information for, and that there may be some circumstances in which we
actually don’t need to go into much more detail. I’ll give you an example.

The tension that we get all the time is investigators who say they want to
measure things in the most precise way possible, but they have to do it in a
very short timeframe. It’s got to be short because there are costs. Follow-up
studies and longitudinal studies, if the questionnaire is too long, then you
get more dropouts and so forth. So for some of the things that we do I’m not
sure that greater precision is going to make that much difference.

We know that there is a systematic relationship between socioeconomic status
and health outcome, for example. If we sharpen up the measures and make them
more precise, we’ll go to a lot more effort, and I’m not sure that we’ll learn
something that we’re not learning already, where for other purposes there are
certainly things about resource allocation in communities that, in fact, we’re
going to need to measure at a greater level of precision. I guess my point is
that one size doesn’t fit all. There may be circumstances in which the minimum
standard is much more a minimum than in others.

DR. MAYS: Jack, are you on line?

MS. GREENBERG: I left him a message and I gave him the phone number here.
The only thing I could find on the Internet was the thing asking for comments.
Obviously we need some clarification, but I do think the question at hand is
given the narrowest definition of your charge, what do you need to do to get
there? Take the narrowest definition, that it’s for federal health or HHS
health surveys. It’s true that the legislation is broader, but what would you
even need to make a recommendation on that, let alone some of these broader
issues?

DR. BREEN: Your own remarks reminded me of something that we might want to
add to the types of recommendations that we’re talking about. The reason I
didn’t mention the concerns about education is because we’re capturing
educational attainment adequately.

We’re definitely not capturing educational achievement, but as we probed
that, it sounded like we didn’t have the means for doing that because you don’t
have a means for measuring quality. Some states, like California, have that.
There are things like graduation rates from high school and proportion of kids
that go on to college, but I’m not sure if those are national measures or if
they would suffice. It just seemed like for that, that’s not sort of in the
ready position.

But an option would be to talk about things that need to be thought about in
the future in the near term or in the longer term, issues that have come up
that are critical to understanding SES, but that we’re not going to resolve
this in six months or a year or by the time that these surveys have an
opportunity to revamp or new surveys start up, because that’s really what it
seems like we’re — from what Susan said, we weren’t asking for immediate
implementation; we were asking for implementation when it’s convenient.

DR. MAYS: The only thing is that I would like to hear from the Department of
Education because I think we didn’t have people from there to know for sure
what it is they have and don’t have or what they can give us or what they
can’t. Then if that’s the case, then I’m done. But are we measuring just
education, or are we measuring the concept that really is SES? I keep going
back to what SES really is supposed to be about.

DR. BREEN: There is a National Center for Education Statistics, and the
deadline was too short for them to get clearance, so they weren’t able to come.

MS. GREENBERG: I will say that this news release that went out in June said
the proposed standards — now they’ve obviously been promulgated and I couldn’t
find that, but it’s somewhere — for collection and reporting of those five
measures in population health surveys are intended to help federal agencies
refine their population health surveys in ways that will help researchers
better understand health disparities and zero in on effective strategies.
That’s obviously what we’re talking about at this point, but exactly how
they’re defining population health surveys and whether that has to do with OMB
clearance and all that I don’t know.

DR. GREEN: What I heard Bob talking about and Dr. Cooper and Dr. Breen
talking about I don’t hear being contradictory to Vickie’s summary. I think
Vickie laid this out close enough for us to head toward next steps. I’ll say it
back to you, and then you fix it.

Basically Vickie said we can do a letter by June about what we found and, in
so many words, said it could be a range of findings. Then I added in we could
have findings that relate everywhere from policy to particulars. It was a rich
hearing. We got a lot. We could render an opinion about the possibility of
minimum standards for income, education, and occupation. Then she suggested we
could then go further with some possible suggestions about some sort of how-to
process to get to those minimum standards for SES.

Then I added one thing to it. We should add cautions because learned a lot
about places where we could do harm, from NIH investigators to misuse and
having something that’s working in MEPS that gets trashed and we lose the
ability to look at 15 years of sequential data. Is that close to what you said?
I’m scared of going too far with a set of recommendations. I am a humble guy
that just drank from a fire hydrant for a day and a half, and I’m afraid of
saying more than I know.

DR. CAIN: I just want to go back to the point that Vickie was making earlier
about the education. The point that I heard that sort of alarmed me and made me
despair of getting what we want was a realization of the age range of all these
surveys. If you’re getting data about a school that is concurrent with somebody
being in that school or a recent graduate, that’s one thing.

But if you’re interviewing a 70-year-old person who the school has changed
names three times and then disappeared, or when they lived in that particular
inner city neighborhood it was upper class or high income or whatever and you
go back to that same school now and it’s a very different population and it has
a very different quality, even if you were able to make those links.

I think we have to think about the range of surveys that would be affected
by the minimal standards. Maybe the minimal standard, as you say, might be
educational attainment in some way, and that more work needs to be done on the
achievement. But I just can’t imagine who you’d ever get that for somebody who
wasn’t an older person, given that the school has changed so much.

DR. KAPLAN: What I heard from some of the people was the other side of
minimal standards. I heard that we would like to be somewhere else, and we’re
not there yet. With education, for example, there was a lot of discussion about
educational quality. The speakers said we just don’t have anything like that.

Or with income there was this interest in getting to network. For people
with equal income, some people have resources so that if they’re threatened by
a recession, they have a pad where others are stuck in these miserable jobs and
can’t do anything. The speaker said we really recognize that as important, we
just haven’t advanced our field to that yet. It would be nice if we were able
to both talk about minimal standards, but also talk about where we need to go
for the future.

DR. MAYS: Larry, thank you very much, because I was busy talking, so I wrote
down what you said I said and it felt right. I think if we can reach consensus
on some things, we’ll do really well. When I was talking about a minimum
standard, I was almost talking about not the actual wording of an item.

But what I was talking about was, for example, if we said that within SES we
collect education, and here are the things that we benefit from, that in the
context of social economic status we may want educational achievement, and then
to say why we think that that’s good and that having that as a minimum would be
great, but that we stop there and not specify an item, and that we talk about
that these things all need work, study, et cetera.

Marjorie, I’m just trying to make sure in terms of from Jim whether he
really wants us to write an actual minimum for him to be working with, or
whether it’s more that what we can give him is advisory about what we think it
should be.

DR. KAPLAN: If you want to get this done, the recommendation might go one
step further. It might express it just as you did, very well, but then say now
we want to hand it off to some structure and maybe even go as far as specifying
what that committee would look like or where they would live.

MS. GREENBERG: I would agree with Susan. I think that probably the Data
Council would establish some type of working group to review what you’ve said,
but they wouldn’t have to redo what the subcommittee has done.

DR. KAPLAN: But we should say who to the Data Council. We should name them
so that it’s just harder to disappear into the atmosphere.

MS. GREENBERG: I think the Data Council is the locus. But then they will
also put this out. Whatever they agree to, they will put out for public
comment, just as they did with the other ones.

DR. MAYS: What we begin to hear about is the overarching federal statistics
group, because I think the issue isn’t just HHS getting it right. For some of
these the very issue is we need them to be working with Census and BLS and the
Social Security Administration and Transportation because it’s a social status
issue and we need them at the table.

MS. GREENBERG: I think they are. Also, you have to realize that HHS surveys,
data systems, with the exception maybe of Vitals, is the numerator. You need
Census, obviously, for the denominator. Census is not an HHS activity, but you
have to be consistent with them or you’re not going to have your denominator
data.

DR. MAYS: We have the Department of Education. That’s the group, the Big 12,
I think they’re called.

MS. GREENBERG: We can get information from them, but at the end of the day,
I think what the Department will do is promulgate this for some subset of HHS
data collections because the secretary doesn’t have any authority over anyone
else.

DR. CAIN: When we developed the original standards, it was a Data Council
subgroup and meeting of the Data Council itself, and Census was at the table,
OMB was at the table, the relevant agencies were at the table for the
development of the original standards. I don’t know any reason why they
wouldn’t be in the future.

There is certainly precedent for OMB calling groups together to tackle
particular issues. For example, I’m involved in one right now that’s measuring
relationships in surveys. That has Census. That has Labor. It’s got everybody.
They do have a mechanism for bringing people together for that kind of
activity.

DR. GREEN: I sense that we’re about to adjourn, and I want to come back to
Bob’s last comment, which I heard as being will this letter include in it some
aspiration statements about where things could go next. We didn’t get reactions
to that, and I just want to see how other people feel about that.

DR. LUCAS: I have a question related to your comment, Dr. Kaplan, and also
to the products that we’re going to produce. If our letter said something along
the lines of what you’re suggesting, that these are recommendations, and then
we’d like to hand this off to some other entity to study further, I guess my
question is how would they know the information we gathered here? How would
they even be able to build on it without repeating if all we gave them was a
letter?

MS. GREENBERG: I know you have been working on a background document.
Obviously they would get more than the letter. They would get that background
document, I assume, or was that just for your own use?

DR. MAYS: Given what I heard today and given what I know of the committee,
it really humbled me also. I think that getting the letter done is primary. A
piece of that document, for example, is this chart that I passed around and
will take Susan an enormous amount of time to finish. I think it can come
later. I think we can make a brief intro to it, but I’m not as convinced that
we can write, because I’m just trying to get the charge done. You’re welcome to
make a recommendation about what you think about that.

MS. GREENBERG: I think a lot of effort has gone into this document. I don’t
think there’s anything confidential in it. It should be available certainly to
the department, but maybe more broadly. My second question is my typical
question. Do you want minutes or a summary from this meeting, or do you just
want to pull what you need from it for either that background document or your
letter?

You’ve said how are they going to know what we’ve covered. There will be the
transcript, of course, but only real aficionados sit down and read transcripts.
We could have 15-20 pages maximum, a summary of the information, and then that
would also give something that could be passed on. People can always go back to
the presentations and the transcript if they want more detail.

DR. MAYS: Maybe this is not the way to do it, but it is what I was thinking,
and that is step by step. We need to get the letter and get it through the
committee. Then we finish the document and put the document up on the web so
that the letter goes where it needs to go, but that we have a little bit more
time in terms of finishing the document itself.

MS. GREENBERG: But the committee may not feel that comfortable even voting
on a letter if it hasn’t seen background.

DR. KAPLAN: What happens is the letter would be finding, recommendation,
finding, recommendation. The findings can be relatively brief, a paragraph or
two for each area that leads up to the recommendation.

DR. GREEN: I agree with that.

DR. KAPLAN: What I am suggesting is that the letter would say we held these
hearings. Here’s finding one. That led to recommendation one. Here’s finding
two. That led to recommendation two.

MS. GREENBERG: That is sort of a standard committee approach.

DR. GREEN: That’s our standard approach, and we should do it again. In
talking with Susan yesterday, just a quick comment about this, and then I want
to ask my question again. The first question in our charge was about what’s the
state of the art of where we are with collecting data about SES. One way to do
that is to try to write a tome. I’m opposed to that. Rather, I really thought
that the quality of the presentations were such that I’m inclined to post
presentations and get them up there and distill stuff, get this paper, let this
paper mature and that sort of stuff, and it’s okay for them to overlap.

But I think in our letter that we’re preparing we can redirect people who
want to know more about that finding to the postings from the hearing. We don’t
want to write a very long letter that explains all of our findings in long
detail. I also agree with Vickie that we don’t want to wait until we’ve got a
more comprehensive document and explanation to get our basic findings from this
hearing and our responses to them out.

DR. MAYS: I think the question of what is the standards really probably
should be done the way we’re currently doing it, which is to illustrate what
the surveys are, what’s collected, whether it collects it or not, and something
about it, because the people making the decision — I think this will be
helpful as opposed to the presentations.

What we have to realize is that in the presentations they were designed to
do this hearing. That’s not the field, though. There were people who would
challenge, for example, things that were said here. So I don’t want to hold
that up as that’s the standard. I think when they’re tracking the standard, we
really should say here is what is collected, and that we do it by these charts
and tables.

DR. LUCAS: I think some of the presentations dealt with things that are
going to be very salient to surveys, like some of the methodological issues
associated with collecting the data. For example, we heard today about how some
of the data collection methods associated with income do well in certain
incomes, but they don’t do well for people in other income brackets.

I think all of those things also have bearing on what kind of
recommendations you make for standards questions because they have an impact
and an influence. I’m not disagreeing with everything that has been said, but I
don’t think those two tables alone do it. I don’t know that any single thing is
going to capture the whole depth and breadth of what covered here, because it’s
big and it’s complex.

DR. MAYS: It was the standard. The first thing that we’re asked to comment
on is what is currently in the field, what’s the standard, not critique of it.
I think the stuff that was talked about helps to explain the why-ness of it,
but something as simple as where is education collected —

DR. LUCAS: But I think what Susan was saying was that we have no idea — you
know they have questions. What’s the agreement in the wording? What’s the
agreement in the response categories? None of that information is there. That’s
what she was saying. That, in and of itself, isn’t enough. That tells us that
all these surveys have a question on education, but not necessarily the same
question.

MS. GREENBERG: It doesn’t tell you what the standard is, but I think what it
tells you is at least they’re asking something on education, so moving them
towards a standard is probably more likely than if they weren’t asking anything
on education. That’s a pretty low bar. Also, I have that you don’t want minutes
or a summary of the meeting.

DR. MAYS: We definitely we need the minutes. For this meeting, the more
detailed the minutes are, the faster we can move. The extent to which we can
have detailed minutes will help the entire committee, particularly because we
have people who weren’t here, that I think it will help them to just read them
or listen to them. Can

DR. GREEN: I am going to give up on my question for now because I really
want to make it to Denver tonight. I think our next steps are clear enough that
we’re headed toward distilling findings and draft recommendations for a letter.
We have to do that with the subcommittee. We’ve got to bring other people into
this now.

That means that we’re going to need some sort of an event in April where
there is prep work for that and we reconvene the subcommittee to where we
actually have to move toward having drafted some findings and some conclusions
that galvanize that sort of discussion, from which we can then prepare a draft
to circulate to the full committee.

MR. SCANLON: This is Jim Scanlon. It sounded like it was a good two days.

MS. GREENBERG: As I mentioned in my message, we need a little clarification.
There are two big issues: what you’re expecting from the committee and through
what data collections the standards that the secretary will adopt will apply
to.

MR. SCANLON: The second answer is it’s basically HHS surveys. It will be the
same application as the first set of standards.

MS. GREENBERG: Is that only population-based surveys?

MR. SCANLON: Yes.

DR. CAIN: Are those things that go through OMB or does it also apply to
grants?

MR. SCANLON: It’s mostly not grants. There are some grants and cooperative
agreements that go to OMB, but we defined it as synonymous with agency
sponsorship. If it’s a cooperative agreement that would require OMB approval or
a grant that would require OMB approval, then yes. It’s the same as the
standards now.

DR. CAIN: But it’s the OMB approval —

MR. SCANLON: That is the definition of agency sponsorship. This would apply
to surveys conducted or sponsored by HHS. That’s literally synonymous with OMB.
It’s all of the main surveys.

DR. CAIN: For example, would that include Add Health or HRS?

MR. SCANLON: Yes, if they are sponsored by HHS. I don’t know if they are or
not. If they had to get the approval or will have to get it, yes, it applies.
That’s the only practical way. We can’t get at every grant that NIH does
because you don’t even know when you work a grant what it’s going to involve,
and there’s no way to get back at the grantees. But the place to start,
everyone agreed, was the surveys that we have direct control over. It’s the
NCHS. It’s the SAMSHA. It’s the MCBS. It’s MEPS. It’s the basic National
Immunization Survey. It’s exactly the same as the first set of standards.

MS. GREENBERG: So it doesn’t include BRFSS?

MR. SCANLON: It don’t know if that requires OMB approval or not.

DR. COOPER: It does include the National Child Longitudinal Study?

MR. SCANLON: Again, if it got OMB approval, then yes. The National
Children’s Study will be there. Our Data Council member from NIH is putting
together the list of the surveys to which he thinks it applies. I, for one,
would like to make it apply to the BRFSS, but we can’t literally make it. I
would hope the agencies would step up. But it’s a very specific set of surveys.
There’s no way to control the grant world, to be honest. We don’t even see most
of those. There’s almost no way a priori to do that. Frankly, I wouldn’t even
worry about it. I think it’s more the major surveys that people were concerned
about.

DR. MAYS: Are you wanting for us to actually have recommendations in which
we give you specific minimums, like here is education, here is what we think is
the best approach to asking? Part of what we’ve been discussing is this notion
of to tell you what we learned and what we found and give you some
recommendations about what we think next steps are in order to get you to that
minimum standard if you choose to move ahead.

MR. SCANLON: I’ve been listening. I don’t think anyone is in a position to
recommend the standards just yet. What I think would help is the first stage in
any of these endeavors is sort of what is the state of the art. What are people
collecting now? What are the strengths and limitations? Is there a lot of
variation now? Do some of these seem to be emerging as sort of the best
practice?

If you gave us what we might call an environmental scan or kind of a
baseline assessment of how these variables are being collected in federal
surveys now pretty much based on what you heard the last two days, I would say,
that would be very helpful to us. I’m not even sure of the standards territory
yet. I think we’ll need more of a collective assessment.

You’ve heard a lot of good things. You saw the way the major surveys are
collecting this information, you saw some of the newer ways, and you heard some
of the issues. I think if you just pull that together for us with a
transmittal, that would certainly meet our requirements. Then we’d have to go
into more depth about is there potential for a standard yet. I think if you did
that, that would be very helpful to us.

You don’t have to recommend the standard. I think that would be a heavy lift
now, and it would take a lot more hearings and work because you’d need to — a
standard is more than just a good question. It’s something that meets a very
high level of performance. That’s why there are so few of them.

That’s certainly one way to go, a nice transmittal, and you’re basically
summarizing and interpreting what you heard. But you don’t have to give
specific recommendations for standards. You might want to say that for the
major surveys like the CPS or census or others, there are ways of collecting
data that could be looked at. But I think the most you would say is to point a
bit of direction. I could be wrong, but I don’t think I heard anything that was
an obvious standard from the two days.

The other thing that I think would help is if you just reminded everybody
that any standards should be based upon proven methods and measurements and
sort of a concept of the core and the minimum, the constants that we’ve
employed previously and that came up again in the discussion. I think that
would be very helpful.

Then we would have to take it in HHS, along with Census, and we’d probably
have to look at what exactly — is there a standard here for education? Is
there a standard for income? Occupation? If so, then we still have to go
through a full process because the standard makes it mandatory.

This is the minimum, obviously. It’s not meant to limit research that
everybody does or to collect additional information. The goal of a standard is
a very tough thing to meet. I think if you, at one level, simply summarize,
analyze what you heard in the form of an assessment of these kinds of variables
in federal surveys and then gave us some directions, I wouldn’t expect
recommendations for standards. Does that make sense?

DR. MAYS: Yes, it does. It not only makes sense, but you just were like a
great psychologist and brought our anxiety down.

MR. SCANLON: That’s why it’s so tough. You would think it would be easy to
standardize age and sex over the years, but it hasn’t been. Everybody does it
differently. I think that would be immensely helpful to us, and we could take
it to a work group. We’ll get the Census Bureau back and all the agencies back.
They won’t have to plow over the old ground. They will be able to say let’s
look at what people think of the way CPS does it. I think some of the witnesses
you had actually compared educational-level measurement and achievement across
surveys, and I think they did the same for income. That gives us a good head
start.

DR. MAYS: I agree. I think we had some very good presentations to help us
move this along. We’re going to be mindful of your time. We really appreciate
you taking a little bit of time to let us know.

MR. SCANLON: I want to thank the committee. That was really excellent. I was
there yesterday and listened to most of today. It really turned out very
nicely. It was exactly what we hoped for, and it save us a lot of original
research and assessments that would take us a very long time to do.

DR. MAYS: I think the next two steps are, one, we need to figure out if we
have any gaps in the information that we need. If we do, I think what we talked
about is having a webinar to get that. Then I think the other thing is the
first thing that we do is schedule a conference call of all the populations.
Let’s try to do that kind of late in April. Do you have a sense of how quickly
we can get the transcripts?

MS. GREENBERG: First, we need to get the transcripts. It’s usually 10
working days. I don’t know if they can speed it up a little. Then based on the
transcript, we can have someone do the meeting summary. They have to do it from
the transcript, obviously.

DR. MAYS: What I am going to suggest then, since it’s about that long, is
that the planning group start identifying if they think there are any gaps and
identifying if there are gaps, who we want to give us the additional
information. Then we should, by the time we’ve done that, be able to talk to
you about where the transcripts are and make sure we get everybody who wasn’t
here up to speed. Then probably we’re looking at maybe by the end of April
having a call with the whole Populations Committee.

MS. GREENBERG: Is that the webinar you were thinking of, or you’re not
thinking you’d need the webinar now?

DR. MAYS: We don’t know if we need the webinar until we check and see if
there’s some depth. I know there were specific things that Susan still wanted.
I just need to find out from here if she still wants those. It may be just a
few presentations.

MS. GREENBERG: For the planning group just one of the co-chairs of the
subcommittee should be involved, Larry because he was here, but if for some
reason he can’t, Sally will do it.

DR. MAYS: Any other suggestions, recommendations, anything else that you
think we should be doing? This is like our agenda for the month of April.

DR. BREEN: I see you’ve got us until late April getting together all the
materials we need, and then assessment of if we need to get any additional
information. In April would this call be to look at a draft letter? What would
that be? I’m just thinking it might make sense to do the planning right up to
June or up to the letter.

DR. MAYS: I was a little worried because we have so many people not here.
What I would suggest is that week two of April we try to determine if we have
gaps. Week four of April would be the webinar. Then by the end of that week or
some other time in that week, we would also have the full Populations Committee
to have a conference call.

MS. JACKSON: That’s kind of late. In order to set up something if you really
want to get the products that you want in April, you kind of need to know by
the end of this month whether or not you’re going to have something set up. A
webinar, from what I understand, is a public kind of discourse. Is that what
you’re talking about in terms of getting information and hearing from people
and setting up a time and date? It’s very time-intensive, and you would need to
know by the end of this month if you’re going to even have that. I’m assuming
that you’re going to be talking among yourselves about gaps and know that by
the end of the month.

DR. MAYS: I forgot we’re in March. I was already in April. We can actually
back up and have our decisions made about if we think we need something else,
who those individuals would be, and when to do that by the end of the month. I
think that by the second week in April would be when we would have if it’s
going to be a webinar, as long as you all can do it within about two weeks
afterwards, because we would have to ask the individuals.

Somewhere near that same week or very close after in week three would be
when we would have a full Populations conference call. I think from the full
Populations conference call should be some of the marching orders of what that
letter should look like. Prior to that, as we go probably to the full
conference call, we want some sketchy idea of what we think the recommendations
would be. I think we can do that by then.

MS. JACKSON: Also, we can talk off line about the specifics of the timing,
but the subcommittee as a whole needs to be involved as early as possible.

DR. MAYS: That is what I am saying. The full Populations Committee would
give us some marching orders of how they would like to see the recommendations
— this is what we’re thinking, and then let them opine on it. Then we have
some time in May. What you have to tell us is I know there’s a step where
things have to go to the executive committee, so I’m assuming that’s mid June
or something.

MS. GREENBERG: The meeting is the 21st and the 22nd,
so at least two weeks or three weeks before then to be able to get something
into the agenda book. We’ll come back to that when we see where we are.

DR. LUCAS: Is it possible to get packets made of all the presentations for
the rest of the committee members that were not here?

DR. MAYS: I think what would help is to just maybe send an email out to the
full Populations Committee. Then I’m going to also ask that the things that are
on the little Widgeo site get also moved. Then that way, we’ll have everything.
They can read background. They can get up to speed to whatever extent that they
want.

MS. GREENBERG: I think they would like minutes, so we’ll need somebody to do
that. What’s the earliest we could get the transcript?

DR. BREEN: In terms of prep for the full subcommittee call, is everybody
expected to review the slides and read the minutes in preparation for just a
general discussion of what the recommendations would look like?

DR. MAYS: No, I think we’re back up a little bit more. I think the Planning
Committee has to step in here first before we do that and see the extent to
which it can draft some things. Then that’s, I think, what we would give them.
The Planning Committee needs to come back into this process at this point. I
think Bruce has been gone. We need to make sure that we can get them up to
speed as soon as possible. I’m hoping we can pull all these documents together.

DR. LUCAS: One of the ways that we could identify where some of the gaps are
that haven’t already been identified is to go back and revisit the questions
that we gave to the presenters within the context of the information that
actually got presented and determine whether or not we feel like we got answers
to all these questions, whether or not these are the questions that are still
relevant, or there are different questions that now need to be posed. Then I
think that will help us identify where the gaps are, because I took copious
notes.

DR. MAYS: I think that’s excellent. Before we adjourn, I want to really
thank the staff. I’m glad that you loved what happened. But in particular, to
talk about Susan and Jackie and Nancy, it’s a wonder you can’t tell by the
cauliflower ear here in terms of how much they’ve been on the phone and talking
to people nicely twisting arms. I greatly appreciate the assistance to get us
to this. They have also produced a report and some tables, which you’ll get
soon.

I want to thank Marjorie, who made sure that I had some guidance. I’m just
coming back into this, so I needed to learn kind of the best way to do it. I
also want to make sure that I thank Bob and Leslie Cooper and also Nancy for
getting the materials out. I hear we had a large number of people who were on
line listening to it. I think they were our colleagues at NCHS and NIH. I’m
very excited about that.

MS. GREENBERG: Thanks to Nicole and Debbie.

DR. MAYS: We are here sitting here being as comfortable as we are. Nicole
was just like give it to me and it was done. She took a lot of worries away,
made sure people had a place to stay. Debbie made sure that everything
happened. So I just want to say thank you to recognize that you played a role
in this. The spotlight needs to be shared. Thank you, everybody.

(Whereupon, the meeting adjourned at 3:20 p.m.)