[This Transcript is Unedited]

DEPARTMENT OF HEALTH AND HUMAN SERVICES

NATIONAL COMMITTEE ON VITAL AND HEALTH STATISTICS

SUBCOMMITTEE ON POPULATIONS

WORKSHOP ON DATA LINKAGES TO IMPROVE HEALTH OUTCOMES

September 19, 2006

Renaissance Hotel
999 9th Street, NW
Washington, D.C.

Proceedings by:
CASET Associates, Ltd.
10201 Lee Highway
Fairfax, Virginia 22030
(703)352-0091

TABLE OF CONTENTS

P R O C E E D I N G S (9:02 a.m.)

Agenda Item: Call to Order

DR. STEINWACHS: I’d like to welcome everyone to the second day of our
Workshop on Data Linkages to Improve Health Outcomes. This workshop has been
put together by the Subcommittee on Population Health of the National Committee
on Vital and Health Statistics, and this workshop is being broadcast live on
the Internet. So I would like to welcome those people who are listening on the
internet.

And I thought before we got started officially, we might just go around the
room and introduce ourselves, for those who are on the Internet.

And I am Don Steinwachs. I chair the Subcommittee on Population Health, and
I am from Johns Hopkins University.

MR. IAMS: I am Howard Iams from the Social Security Administration Research
Office.

MS. MADANS: Jennifer Madans from the National Center for Health Statistics.

MR. HARRIS-KOJETIN: Brian Harris-Kojetin from the Office of Management and
Budget.

MR. BJORKLUND: Rick Bjorklund, Office of the Assistant Deputy Under
Secretary for Health, the Veterans Health Administration.

MR. PETSKA: I am Tom Petska. I am Director of the Statistics of Income
Division of the Internal Revenue Service.

MR. CHAPMAN: Chris Chapman, U.S. Department of Education, National Center
for Education Statistics.

MS. OBENSKI: Sally Obenski, Data Integration Division, U.S. Census Bureau.

MR. PREVOST: Ron Prevost, Data Integration Division, U.S. Census Bureau.

DR. DAVERN: Michael Davern, University of Minnesota.

DR. STEUERLE: Gene Steuerle from the Urban Institute and a member of the
committee.

MR. LOCALIO: Russell Localio, University of Pennsylvania, School of
Medicine, a member of the committee.

DR. SCANLON: Bill Scanlon, Health Policy R&D and a member of the
committee.

(Additional intros around the room.)

DR. STEINWACHS: It is a pleasure to welcome everyone today.

Our first session has speakers from Internal Revenue Service, Department of
Education, Department of Veterans Affairs, and Russell Localio, who is a member
of the committee, is going to serve as facilitator.

Russ.

Agenda Item: IRS, Education and Veterans Administration

MR. LOCALIO: Good morning, everyone.

I just want to introduce our first speaker, Tom Petska, our friend from the
Internal Revenue Service.

PARTICIPANT: Is that true? Oh, of course that is true. Of course.

Agenda Item: Tom Petska, Director, Statistics of Income Division, IRS

MR. PETSKA: I have no comment on that.

Actually, I am very pleased to be here to speak on this topic, Workshop on
Data Linkages to Improve Health Outcomes. A lot of people might think what is
the IRS role in that? I hope I can say a few things in the next 10 or 15
minutes about that.

In a way I feel a little bit inadequate about speaking to this group for a
few reasons.

One is that my division is centrally located between IRS and Treasury, and
that gives me two bosses, the Director of Research Analysis and Statistics of
IRS and the Director of Tax Analysis of Treasury, and they can – they sometimes
agree – put it that way – on what we should be doing and how we should be doing
things. So that is a little awkward.

Also, because I am Director of SOI, I get occasional questions as to, Can
you tell me about Section 8267 of the Code and how that effects my small oil
and gas operation?

And the bottom line is the Internal Revenue Code is 2,000 pages long. The
supporting regs are over 10,000 pages, at last count, and I really don’t have
that encyclopedic knowledge. I am sorry.

But I do get questions like, What is the average amount of charitable
contributions for my level of income? And that is something I can look up, even
though I can’t look up what is the status of your refund.

So, that said, hopefully, I can tell you a little bit about IRS tax data
administrative statistics as a potential source for shedding some light on
health outcomes and so on.

Before I say anything further, I would like to add my disclaimer that these
are my personal views and not necessarily those of the Internal Revenue Service
or the Department of Treasury.

A little bit about my organization. IRS is a large organization, 100,000
employees, an $11-billion budget.

The SOI program is about .4 percent of that, $40 million, which sounds like
a lot of money, but, relatively speaking, it is small, and under 500 employees.

Our primary customers – We have two very intensive customers and those
being the Office of Tax Analysis of the Treasury Department and the
Congressional Joint Committee on Taxation.

I’ll be talking a little bit about data access and disclosure, and let me
just say that those organizations do have full access to all of our data, and
they have a heavy role in directing our priorities and studies.

However, we do have many other customers, many in the federal statistics
community, including the Bureau of Economic Analysis and the Census Bureau.

Okay. What kind of data do we have?

Well, I probably should have given a little bit more background to this
slide. I am going to be talking a little bit about two types of data. One are
the sample data from my organization. We sample tax returns and we have
scientifically-designed samples. We edit these samples very carefully, and we
weight these to national totals.

The content is based on our user needs, and so if the Treasury Department
comes back and says, We want multiple schedules on depreciation by different
types of class lives, we can’t add that. It is a resource issue, but it is not
a policy issue or program issue.

Now, separate from that is the relatively content poor and less edited data
in the IRS Master File system, and we don’t produce that data, but we are kind
of a gateway to other federal statistical agencies and researchers who do have
access for that.

It is also the main source of statistics at the sub-national level, because
our SOI samples are not robust for most states and certainly not below the
state level.

Okay. So those are the types of data we have.

Now, in each of these program areas – individual, corporate, partnership,
estate and gift tax, tax exempt or non-profit organizations – we have pretty
much these two sources, and, for the most part, the content-rich SOI samples
are the preferred source, just because the data are of higher quality as well
as the content much greater.

And just as a footnote, I should add that all the data that is filed on all
tax returns and schedules is not transcribed, and that is one reason for our
program is that I think there are something like nearly 100 schedules that an
individual can append to their 1040 return, and only a limited amount of that
data is transcribed. So, as far as content, we have that flexibility, although,
as I said, it is a resource issue, and, again, all of our data are preaudit.

Well, two viewpoints on SOI. Well, one is that we try to be a cooperative,
collaborative and efficient producer and user of data based on administrative
records, and I think we do a pretty good job on that, for the most part, but,
then, on the other hand, we are the – quote – tax collectors in disguise, as
survey statisticians, and I’ll talk about what that means a little bit further
down the road, but, first and foremost, we are employees of the Internal
Revenue Service, and that has certain legal issues in terms of what we do and
in terms of our relationships with others as well.

Now, we have kind of generically three opportunities for data linkages,
first being linkages involving solely IRS tax and information returns. Others
are linkages involving tax data to surveyed records, and that is a very short
topic, which I’ll tell you why, and, then, lastly, I suspected that the kind of
tone or focus of the conference would be what about microdata access by
researchers and other agencies and so on, and so I thought I would spend some
time on that, and I have brought my disclosure expert, Nick Greenia, who is
sitting in the back there, who works with me and my boss, Mark Mazor(?), on a
lot of interagency issues involving data access.

Okay. If you gave me the time, I could go on and on about this first one,
linkages involving tax and information returns, for which we have control over
the data. We have access to the data and so on, and first might be individual
returns linking in 1099s and W2s.

For instance, you see an individual return. It’s got a certain income,
$75,000. It is a joint return. There’s a husband and wife, apparently. Is it
from one income or two incomes? Well, you can’t determine that without looking
at such things as the W2s and so on. So we do linkages like that to look at the
whole picture of family economic income.

Partnerships. Partnerships are a major type of business, but they are
untaxed. People or organizations, corporations, non-profits form partnerships.
They report their financial activities on an information return, and then they
distribute their taxable shares to the partners, and those could be expenses.
They could be income and so on.

So to ascertain the total effect of partnership taxation, you have to link
in partner tax returns, and one partnership could have 20,000 partners. So it
is not a trivial matter.

Small business corporations are similar. They have the flow-through nature
of a partnership.

The estate tax has become a hot political topic once again, and among the
questions are, What happens to these bequests from large estates? The estate
return shows who gets them, but it doesn’t show the income and what happens to
those individuals over time. We can link them up by matching their 1040 returns
and following them over time on a panel basis.

There are a lot of issues in regard to consolidated corporations.
Subsidiaries may file separately. To get a combined picture of corporations,
you have to link these together, and we use the employer identification number
to do that.

And then another focus of our primary customers, Treasury and the Joint
Committee, are for panel files linking the same entity – corporation,
individual, et cetera – over time or linking within a year, and we do quite a
bit of that kind of work, which we can talk about as time allows.

Okay. What do these files have?

Well, first of all, high-quality linking variable. If you don’t have that –
We don’t do a lot of research on using other things like names and addresses to
link. If we don’t have a high-quality variable, we probably don’t have the
resources to do a high-quality study.

Fortunately, most of our studies, most of our files, the variable, the
employer identification number or Social Security number is very accurate.

Obviously, if you didn’t have overlapping samples and you try to link
records, you are going to get a few hits because of that. So we need at least
one population file or samples that are substantially overlapping.

And, lastly, what about accounting periods? We do – at a lot of our linked
studies, we want to align accounting periods. We want to take the partner and
the partnership or the corporation and its employment return and we want to be
very specific about aligning those accounting periods, or, in other cases, we
want to show a panel-like study with different periods.

Okay. Well what have we learned from all these studies over time?

Well, a few things. First of all – and I think that has been the tone of
day one of the conference is that matched files can be very rich analytically
and so on, but, on the other hand, linking data files is never probably as easy
as it seems to be. Data quality is never optimal. Even in our SOI files linking
variables, even though our Social Security numbers and EINs are high quality,
but there are times when, in some cases, that they are not as high quality as
we would like and we have non-matching and so on.

Resolving these discrepancies, if they are important – and I think they are
– often is a very labor-intensive effort, and so we try to avoid that to the
extent possible, but we don’t want to produce a file that is based solely on
matches and ignore the non-matches. I think that would be a mistake, and linked
files sometimes don’t answer all the questions, and we could talk about that
also. Okay?

But I think the one key thing is that you take this data, if you develop
high-quality linking variables and you put it in a relational database, I think
you are way ahead of the game. Okay?

Well, this, I said, was going to be a short topic, linkages of tax data to
survey records, and it is short for two reasons. One, we do very few surveys
ourselves. The only surveys we do are for corporate returns, particularly
multinationals.

As you know, just as individuals can request filing extensions,
corporations can, too, and most of them routinely do, and so if we have an
early data cutoff and we need to get that corporate return in, that major U.S.
corporation, that sample at a weight of one, we often do a survey and request
preliminary data from them, and so – or, in some cases with multinational
corporations, they’re just not completed that accurately, the presumption being
that everybody provides IRS accurate and complete data, but that is not always
the case, and these multinational returns are very, very complex and sometimes,
even despite their best efforts, they are not complete showing all their
international activities and so on.

But then the last point is we don’t do matches to other survey data,
because, again, getting back to my point earlier, we are the tax guys, so
people don’t provide us microdata. I mean, that is the other hat we wear. We
are the IRS guys and the perception would be that could these data be brought
into a compliance type situation.

Okay. Provisional microdata to other agencies. Who can get what? Well, this
is a very, very short summary and so on, but, first of all, tax administration.
By this I mean, for the most part, members of the IRS, and sometimes Treasury
as well and so on, who have a tax administration motivation. Taxpayer account
processing, audit, compliance and research functions are all internal and they
all can get access to most of these. Although, we have to be very careful in
cases where there are sample data involved and could it destroy the
accurateness and representatives of these samples.

Tax analysis. Treasury’s Office of Tax Analysis, the Congressional Joint
Committee, CBO and GAO all have roles in tax analysis and can get data, not 100
percent, but, for the most part, can get some identifiable tax data for
specific purposes as articulated in the Internal Revenue Code.

And, then, statistical use. The Bureau of Economic Analysis can get
corporate data. Census gets population data for individuals and for businesses.
When the Ag census was moved to the Department of Agriculture, NASS, a few
years ago, that data access was also enabled with them, and the CBO as well.
Okay?

We talked yesterday – and I think the Census people presented this very
well – that their goal, their mandate is to use existing data system to
the maximum extent possible, such as administrative records.

Our mandate, unfortunately, is the opposite, though. It is provide only the
federal tax information for authorized purposes and to the minimum extent
necessary. So this is just a naturally conflicting mandate that we have.

Constraints on using IRS tax data. Well, first of all, it has to be for a
use for authorized purposes, and this is defined in the Internal Revenue Code
and in supporting regulations, the 10,000 or 12,000 pages that I mentioned
earlier, and can be also further defined in separate documents, MOUs, such as
the Census-IRS criteria agreement.

Again, our mandate is to disclose only the minimum confidential federal tax
information necessary, and there are substantial penalties for unauthorized
disclosure or inspection, and publicly released data must be anonymous.
Although, we do have a public-use file. As time allows, we can talk about that,
but we do remove identifiers and sanitize records in a subsample of our
individual program, and it is used by a number of high-profile policy analyst
groups.

Okay. Briefly, the authorization process for access to tax data.

Well, first of all statistical recipients – and this came up yesterday –
need to be cited in Section 6103(j) of Title 6 (sic) of the Internal Revenue
Code.

To change that, Congress must enact legislation.

The statute authorizes access purpose and may stipulate supporting
regulations and so on. The regs – regulation detail may restrict uses as well.

And, as I said before, policy agreements can provide additional
enumeration.

In summary, access to tax data is very restricted. Some possibilities
include – and these are very limited – working as a contractor for tax
administration purposes. We have had a few of these, but they are very limited.
Working at an agency with current access, like the Treasury or like the CBO or
the Joint Committee, or accessing limited business data via Census’ Center for
Economic Studies.

And to find out more, we can talk later or drop me an email or give me a
call.

Thank you.

MR. LOCALIO: Thank you, Tom.

Do we have any questions?

MS. TUREK: (Off mike).

MR. PETSKA: Survey on Consumer Finances.

MS. TUREK: Yes. They use you as a sampling frame. They don’t get any data
items.

MR. PETSKA: I worked on that study several years ago, and where it is right
now in terms of the firewalls, I am not clear exactly, but, basically, the
high-wealth portion of their study is a list frame developed from our 1045.
That is correct, and we have done this with all years of the Survey of Consumer
Finances, going back at least to ‘83, and we did have some involvement in
the first, the Survey of Financial Characteristics of Consumers.

Nick, do you want to say something about that?

MR. GREENIA: It is true that the Federal Reserve does receive some data,
but the Federal Reserve is perceived as a 6103N contractor, which means that
the purpose of that tax-data receipt is seen as fulfilling a Treasury
tax-administration purpose.

As you may recall, when CIPSEA was submitted to Congress in 2002, there was
a companion bill, and the companion bill had the Federal Reserve in there, so
that they could receive tax data unrestricted for survey of consumer finance
purposes.

DR. STEUERLE: Tom, you might clarify a little the fact that it is possible
to apply to your agency to have data run by outsiders.

I am wondering if you might also comment the extent to which that is really
restricted by resource constraints, because I am sure everyone who comes to you
to have something run basically impinges upon your resources to some extent,
because they always need some helping hand, but that is a possibility –

MR. PETSKA: Yes, that is a very good point.

DR. STEUERLE: – with respect to health data that may not be even with
respect to any tax question, right?

MR. PETSKA: Yes, that is right. I mean, again, I have talked about
restrictions on access to micro data, but at table-level data, we have
disclosure rules to suppress cells that have fewer than three observations at
the national level and so on.

But, for the most part, if we have an existing file that will meet your
needs, we can enter into a small reimbursable contract to produce tables from
those files and so on.

The problem gets in when there’s matching required or content that we do
not currently have.

For instance, a few years ago, we talked about the idea of non-cash
charitable contributions. Could we produce some aggregate statistics to be
published on that?

We didn’t pick it up in the program back in those days, and so for us to
edit those data, build it into the sample, weight it and everything else was a
very expensive task.

Since then, we’ve gotten a push from Treasury and from Joint Committee to
include that part of the program. So, now, we have, though, so we could produce
additional – from that and so on.

So, again, we do have restrictions on staff, time and so on, but, for the
most part, a tabulation from an existing file, we really try to service those
kind of requests.

DR. DAVERN: Hi. Michael Davern from the University of Minnesota. I have a
question.

We heard from Census yesterday about matching Medicaid records to the
current population survey. Something that might be interesting as well would be
to take a look at some of the W2 information – I don’t know if they have access
to it – which would say deferred compensation for health insurance coverage,
for example, that a person paid pre-tax dollars into an account.

I was wondering if that was some kind of matching study that may be
possible to verify how well the CPS not only measures Medicaid, but how well it
measures private insurance coverage.

MR. PETSKA: That is a good question.

Nick, can you help me out there in terms of does Census get the W2 records
now?

MR. GREENIA: As a result of a regulation amendment two or three years ago,
they now get some limited data that includes deferred compensation from the W2.
So that information is in scope at the Census Bureau.

Where we get into some issues is when Social Security data might be
involved, and Social Security, as you know, has a very unique arrangement with
the Census Bureau, they essentially access tax data and Census data as special
sworn status employees.

So, for purposes of the criteria agreement, the policy agreement that Tom
was talking about that enables access to new projects by special sworn status,
we treat them as, if you will, employees for purposes of this agreement.

The sticking point is that that kind of access, especially if we are
talking about any tax data that Social Security does not have access to, even
when they are matched to Census Bureau and they do have access as far as we are
concerned, Census Bureau views them as special sworn status employees, which
means, for purposes of our agreement, they are viewed as Census employees.

So once that enters the equation, the work has to be done under the
criteria agreement, which means that it has to be predominantly for Title 13,
Chapter 5.

MR. LOCALIO: Howard, did you want to comment on that at all?

MR. IAMS: Yes, the process for doing that would be to go to the Census
Bureau and apply for permission to use these data at your Census restricted
data center and they have an application process that formally requests a
purpose, and what Nick has emphasized about the Title 13 is – has to be done,
what you have articulated would have a clear Title 13 purpose.

I think the problem you will encounter is that I do not think the code that
isolates health purpose for the deferred compensation is available. It may be
available for the last earnings year, but it was not available before 2005. So
you won’t be able to identify how the purpose of this deferred differs from,
say, 401K.

I don’t know if Ron – do you remember? We recently started coding for our
matched data the reason for the deferred – what kind of account it was –
whether it was 401K, 403B, 457, and I do not recall if the health account was
separately coded.

MR. PREVOST: Yes, I don’t believe it was, but we could check into that,
that is for sure. Certainly, the project that we are potentially discussing
here would have a clear Title 13 benefit. I mean, it is –

MR. IAMS: Oh, it would. The question is whether the data are there.

MR. PREVOST: Whether the data are available is the main question.

MR. IAMS: The deferred compensation is identified, but there is a box that
identifies what the deferred compensation reflects, and I don’t recall if
health account was one of the codes. It has only been available in the last
earnings year.

MR. LOCALIO: Well, I think we need to go on.

I want to thank you, Tom, very much for your presentation. We gotta go on
or we are going to run out of time. I am sorry.

Next, we are going to hear from Richard Bjorklund from the Veterans
Administration. While he is getting his presentation set up, I just want to say
that, yesterday, we heard several people comment about their potential dealings
with the Veterans Administration and how those dealings were cut short by that
announcement of a potential breech which never happened.

I do want to say that I got a letter from the Veterans Administration, as a
veteran, saying that there had been a potential breech, and then a letter
saying that it did not happen, and I think we would all be interested to find
out what have been the – if you have any comments about what have been the
repercussions of that from your perspective.

Thank you.

Agenda Item: Richard Bjorklund, Director, Veterans
Administration

MR. BJORKLUND: Well, just to answer that question quickly, the
repercussions have been a very tightening of all of the policies and procedures
for distributing data both within the organization and also to other federal
agencies or contractors or researchers.

In fact, researchers outside of VA who once had access with stipulations to
VA data have all but been restricted from accessing that data now.

So the security and privacy procedures have been tightened extraordinarily,
and I know some of you from CMS are here, and we have been working diligently
for months to get specific agreements in place and protocols for transferring
VHA to CMS. That is happening shortly, but it has taken a great deal of time,
too.

Well, let me launch into the presentation, the nature of my presentation is
more general in nature than what Tom was talking about.

I am going to be talking about more strategic direction that our
organization is taking regarding linked data and talking specifically about a
project that is underway.

First, we link with our internal data a number of independent activities.
We have an annual survey of enrolles where we try to identify perceptions,
interests, preferences, behaviors of veterans, and we link that to their
inpatient and outpatient clinical records.

We also do customer-satisfaction surveys of fairly excruciating detail, and
we link that also to our administrative data both clinical and cost data. So we
can get a comprehensive assessment of the performance of veterans
organizations.

Today, I want to talk specifically about a very large project that we have
been undertaking for the last 18 months, and it regards the integration of VHA,
Medicare and Medicaid data in the production of a user-friendly system, and I
am going to be talking about the opportunities that we envision for improving
healthcare outcomes, some of the barriers to implementation that we have
observed going through this 18-month process, more about what the process was
and some of the challenges going forward.

First, to put this in context, I want to spend a little bit of time talking
about the VHA organization.

First of all, VHA is a component of the Veterans Administration. It is one
of three major components. The other two being cemeteries and veterans
benefits.

We have approximately 156 hospitals, 876 outpatient clinics, nursing homes
and domiciliaries.

The VHA budget is about $35 billion and would rank it amongst the Fortune
50 organizations if we were a private-sector organization. So we are a very,
very large organization and we are a very big player in the U.S. healthcare
system.

Recently, VHA has been mentioned as providing some of the best healthcare
in the country, and it has been management’s objective to be the world-class
healthcare provider for some time, but to continue to be the world-class
provider, we need to continue to be on top of our game, and that is to identify
opportunities to improve quality, cost and access, and, in addition, we think,
by identifying these opportunities, it facilitates what we refer to as a
learning organization.

When these opportunities are identified, it challenges our employees to
think about the best solution to the opportunities, and, hence, old cliches
like not invented here are quickly becoming disassociated from the culture of
the organization and in its place is a constant search for better ways to do
things, better ways to achieve superior outcomes and looking beyond our
organization to the outside world to identify ways to improve that and
essentially becoming globally smart, and we think that integrated data provides
opportunities to facilitate our overall objectives of maintaining our
world-class status and identifying opportunities, and, specifically, the areas
that we think have the greatest potential here are for best practices, and that
is comparing VHA with the private sector along both outcomes and cost
dimensions.

So for any of our 156 hospitals, we would be able to identify where the
biggest opportunity, the biggest clinical opportunity for improvement is or
where the biggest cost opportunity is and what the tradeoffs and the metrics
linking cost and quality are.

So we, essentially, are beginning to use – we have internal resources
devoted to developing risk-adjusted outcomes models, severity-adjusted cost
models. We also have resources dedicated to identifying how veterans make
decisions when they select a VA facility versus a private-sector facility, and
we can compare things like quality, cost, access, benefits and service
characteristics of our facilities versus those in the private sector and look
at the impact of decision making.

As part of this particular effort, we were also able to identify fraudulent
billing practices that occurred where healthcare plans, physicians, offices, et
cetera were billing, double billing both VHA and CMS for the same set of
services.

We think that when more timely data are available or integrated into our
plans that physicians will be able to utilize this data online for treating
patients.

And, finally, strategic opportunity identification where we can look at –
from the corporate level, we can identify where we think the biggest
opportunities are, whether they be in cost or quality of access, and identify
corporate-level strategies that would be part of the corporate-level strategic
plan for the coming year.

In terms of barriers to implementation, we have talked about the number of
opportunities, I think, by the description that I have given. The size of these
opportunities are potential huge. So why haven’t we taken advantage of these
opportunities in the past?

These are some of the barriers that we have identified as we went through
this project. First, there were few people that have knowledge about these
three data sets and the ability to access the data.

Secondly, integration is very difficult and time consuming. Medicare and
Medicaid and VHA data are three separate databases that were developed
independently with different purposes, different data and different data
definitions, and so if you can imagine every time in the past when we have
tried to do an analysis of data where we had to integrate this, it was each
time the data had to be integrated for that single purpose.

Data sets are very large and generally require a higher level of
programming skill, and, generally, that is SASS.

Investment in hardware and storage media can be an important consideration,
depending on the number of users.

Potential users of this data have different needs, and it is – those needs
have to be carefully considered in designing a system. One system will not
satisfy the needs of all users.

The size of the potential demand for this integrated data is unknown within
our organization, and, hence, the risk of investing in a large system that is
very expensive and it may not produce a payback.

And, of course, privacy and security laws and regulations add in a very
large dimension to managing this set of data and is something that is becoming
increasingly more important, being elevated in terms of its priority in making
investment decisions.

Next, the decision makers, by and large – and I guess I am talking in
general – do not have the experience of using data that is outside of the
organization. Historically – and I think this is true within most organizations
– the focus has been on internal data, and, quite frankly, I would suspect that
– just estimating – 70 percent of the information for solving most problems
comes from internal information.

And so our managers and executives are not – do not have the extensive
experience in requesting the kinds of information that comes from the
integrated data.

And, finally, and another important consideration, is the economics of such
a database, and when we talk about the economics, we are thinking more broadly,
not specifically, at the costs, the dollars and cents numbers, but more broadly
in terms of the fixed cost and the variable costs of these activities and
whether it makes any sense to outsource those variable costs, those costs that
could be converted from fixed to variable.

So the process in this project, first, we undertook a survey of users in
our organization to try and identify both the size of demand and the timing of
demand and also customer uses of integrated data, and those are potential uses,
and we learned that demand would grow slowly, but would, over time, begin to
increase rather dramatically as people learned how to use the data and how to
access to the data.

We hired a contractor who had experience in integrating Medicare and
Medicaid data, but no experience with VHA data, and asked them to integrate the
data and develop a user-friendly system.

The user-friendly system we considered key, because it would expand,
exponentially, the number of users, and, specifically, historically, our user
base for this integrated data have been researchers and data analysts.

We wanted to expand that to what we refer to as the casual users, the
directors of hospitals, the directors of our visims(?) or regions, our chief
medical offices, et cetera, but they are not sophisticated users, and so the
user-friendly system had to be simple enough for them to access the data, and
we thought, as we expanded the user base, that we would be increasing the value
that the organization received from the data.

The pilot test that we put together consisted of data integration of the
three data sets that I have spoken about and systems design.

We had three white papers written which were basically analytical, short
analytical papers by the contractor. He worked on three issues that were top
priority issues for the organization at the time and presented some white
papers.

We also did tutorials to researchers and data analysts about the system. We
asked them to come up with a short research topic and to use this integrated
system to address the quick research questions that they came up with.

After the tutorials and research projects were completed, we conducted a
customer-satisfaction survey amongst the users to identify strengths and
weaknesses of the system and of the integrated data. We also did a data
validation study, where we validated the data that came from this integrated
system with the raw CMS data that we have in our files.

Generally, systems design, we have in mind a multiphased project. Each
phase would consist of design, use and assess. So we would be coming up with a
first phase having our users assess it, going back to the design table with the
contractor, redesigning it again and going out, having users use it and
accessing it and until we felt comfortable that the new system could be rolled
out to the entire organization, and I have talked about the three customer
groups, the researchers, data analysts and the casual users we were trying to
look at.

In terms of Phase One, we used the five-percent sample of Medicare data for
one year and 100 percent sample of VHA and Medicaid data for the same year and
merged those three data sets.

As I mentioned, we did a customer-satisfaction survey, and it pointed to
areas that were strengths of the system, but also some shortcomings in the
system. So we are prepared to make improvements should we decide to go forward.

VHA, Medicare, Medicaid data were integrated using contractor assumptions;
that is to say that VHA staff were not intimately involved at this point.

Intermediate data products, and these were basically SASS data sets that
were produced from the raw data, were compared to VHA Medicare files and no
significant differences were found.

Issues that came from the satisfaction survey, one was spending more time
learning the system and/or making the system more user friendly were raised.
There were technical questions raised about how to make the system faster, and,
for example, with bigger machines, more memory, processing one year’s worth of
data at a time.

It was thought that managing risk associated with HIPAA, the privacy act
and security regulations might be reduced via using a contractor and
contractor’s customized software, and, in the future, more involvement of FHA
staff was needed.

Question remain about whether there is sufficient demand to justify the
investment, technical and other challenges – whether technical and other
challenges can be overcome.

And some of those challenges are the user-friendly nature of the system.
Can it be made more intuitive. Reducing the learning curve of researchers, data
analysts and clearly for casual users.

We think that a reporting system linked to the output of the user-friendly
system would satisfy the needs, for the most part, of casual users.

And addressing processing time issues is also another one. Processing time
was mentioned by our technical folks. Another side of the story was mentioned
by one of our researchers who said that his time from the point where the
project was initiated and where integration of the data was called for to the
time that he received analytical results was cut almost by a fifth.

Now, at the same time, what we were hearing from some of our technical
folks that it was taking up to 24 hours to run requests for large data sets.

So there is some benchmarking that is required as we go forward and some
internal agreement as to what we are going to measure and how we are going to
benchmark.

MR. LOCALIO: Richard, we have to move on to the next speaker, if you could
conclude as quickly as possible.

MR. BJORKLUND: Okay. There are cultural differences issues that we need to
overcome. I have mentioned the economics and the importance of outsourcing,
and, finally, some organizational issues.

This has been a green-house project. We are not entirely sure whether it is
part of a – the next phase should be part of a planning and policy office.

MR. LOCALIO: Thank you very much.

I want to introduce our next speaker, and, then, while you are setting up,
maybe entertain a question.

Our next speaker is going to be Christopher Chapman of the National Center
for Education Statistics.

Do we have a quick question for Richard on his presentation?

DR. SCANLON: This is maybe a comment as much as a question.

Since you raised sort of the issue of Medicaid data again, we heard about
it before, it creates for me sort of a bigger issue, which is the quality of
administrative data, and while we are interested in terms of linkages to be
able to expand our capacity, there is a question of did we move too far in
terms of reliance upon sort of administrative data, and while averages may sort
of turn out for populations to be the same, when we do validation studies, when
we get down and we start to slice things more and more, we may be on relatively
thin ice, because the data are not good.

I raise this because of sort of prior work that I did at GAO. Medicaid data
is always suspect, and there were efforts that we had to do where we had to go
out and collect new data because the kind of information that comes to CMS is
not necessarily sort of accurate.

It becomes even more problematic as Medicaid moves more and more towards
use of managed care and the variability, in terms of what the managed-care
plans report to the state, is increasing and leaves you big gaps, and so I
guess this is – maybe there’s not an easy answer to this, but I think it is an
issue that we should be thinking about.

It is as much as sort of how much – in terms of trying to protect sort of
privacy, whether people can be identified, it should be a concern about sort of
linked-data sets should be – what is the extent of their strength? I mean, what
can they be used for and what would be pushing their limits too far in terms of
reliability and accuracy?

MR. LOCALIO: Did you want to respond quickly?

MR. BJORKLUND: Yes, in terms of the Medicaid data, yes, we agree. We have
concerns about that.

Our primary focus in our best practices is with the Medicare and the VHA
data.

At the corporate level, we try to what we say – what we refer to as dumb
down the data. So we downgrade it from ratio and interval-scaled data to
nominal and ordinal data, so as to eliminate some of that error, but, clearly,
these are concerns.

MR. LOCALIO: Thank you.

Chris, why don’t you proceed. Thank you.

Agenda Item: Chris Chapman, Program Director for Early
Childhood and Household Studies, National Center for Education Statistics

MR. CHAPMAN: Sure.

Hi. My name is Chris Chapman. I am from the National Center for Education
Statistics, which is part of the U.S. Department of Education.

My presentation is really going to focus more sort of on our experiences at
the center with using administrative record data that have been collected
already through National Center for Health Statistics.

I guess before I get too far into this I should also make a disclaimer. I
am speaking here not for the department or from organization, but more as a
data user.

That said, let me sort of jump in and discuss a little bit about the kinds
of data that we typically get at the center regarding health.

Not too surprisingly, we focus most of our data collection on trying to get
information about students and other individuals like teachers that are key to
the education system, and apart from individual-level data, we also collect
information directly from institutions themselves, in particular schools.

Most of our experience there, in terms of gathering health information, has
been at the elementary and secondary level, trying to determine which students
actually have individualized education programs which are specifically designed
to help students with disabilities get the kind of education that they are
going to need in order to function in society later on after school.

The data sources that we normally get our information from are parents,
students and school records. Okay. These are not health-system type data
collections. They are relatively general, and we rely on parents to have
relatively good information about medical evaluations of their children, and we
rely on students to be relatively knowledgeable about their health conditions,
and for school records, as I mentioned before, we really focus in on the IEP
data that schools gather and keep for their students.

However, it would be good and would be useful for us to be able to get more
information about student health linked into the school record systems.

This next couple of slides, I am going to briefly go over the types of data
collections that we’ve got in place, so that you’ll have a better understanding
of what we have available and what we usually work with.

This first slide focuses on the early-childhood longitudinal studies. These
studies are actually in my program office.

As you can see here, we’ve got two cohorts. There is a birth cohort and a
kindergarten cohort. The birth cohort focuses on a group of children who were
born in 2001 and the kindergarten cohort focuses on a cohort of students who
were in kindergarten during the 1998-1999 school year.

The reason why I want to start off with this slide is the ECLS-B, the
birth-cohort study, really is, I think, the center’s most extensive experience
using health-record systems – okay? – in particular the sample for the study
was drawn directly from the birth-certificate record systems that are available
through the National Center for Health Statistics.

And apart from using the birth-certificate data, that data set also
involves some direct health assessments. Our field interviewers actually did
data collections on birth weight, height, cognitive growth and motor-skill
development as the child progressed from birth through at least into
kindergarten now.

And then we also had, as I mentioned before, many of our data sets, parent
reports of diagnosed disabilities and overall health of the child.

The kindergarten cohort data collection had much of the same kind of health
information, except for the birth-certificate data, and those data would have
been useful to get.

We were not as experienced, I don’t think, as some of the other
organizations here with actually taking an existing data set and trying to
cross link it with administrative record systems. So we did not undertake that.

And apart from that information, the kindergarten cohort also collected
data directly from schools about IEPs for the sample children.

This next slide has some information about a high-school cohort that is
comparable to the ECLS studies in that we are tracking a group of students over
time. In this case, it is tenth graders, and we are tracking them through early
adulthood.

Again, we – the health-related information we have in this collection were
reports from the students’ schools about their IEP status and health-related
programs that the students were in.

We have also asked the parents to provide us information about diagnosed
disabilities that might not have impacted their IEPs, and then we also asked
the students themselves about their health status.

The National Household Education Survey collects data about populations
from preschool through adulthood. Here, our only data source is information
that we get directly from the parents and the students themselves.

The type of information that are on these data sets that would allow us to
cross link to the administrative record systems is relatively limited. The
sample draw for this particular data collection are telephone numbers, and we
have not, to date, found a good way to even cross link the telephone numbers
with strong, address-matching records, which prohibits our ability to link it
into some of the more detailed administrative record-data systems that are out
there.

This next slide summarizes our post-secondary data collections that we have
collected to date. Then we continue to field.

The biggest one here is the National Postsecondary Student Aid Study or
NPSAS.

All these studies, again, rely on us getting reports directly from the
students about their health status.

The NPSAS collects a lot more detailed information than many of our other
studies, but, nonetheless, is still a self-reporting system.

Okay. I wanted to get back to the ECLS-B for a moment here, because, as I
mentioned before, that is really our primary experience with this sort of
activity, where we are trying to actually link in existing administrative
record systems with our survey-data systems.

The birth cohort did this quite efficiently by starting out with the
birth-certificate data system that is available. So, as a result, we have a
very rich database available for these children right from their birth, and the
data set is actually rich enough or the birth-certificate data is actually rich
enough that we treat that initial birth-certificate collections point is
actually an initial data set, even though we didn’t do any surveys. We just
basically took the data off of the birth-certificate data and we linked it into
the student record, and we have been tracking it ever since, but that is really
our only experience to date using the statistics with our survey data.

Staying with the ECLS-B, I don’t want to minimize the health-record systems
that are out there. Without them we could not have even done this study. We
wanted to make sure that we had a representative sample of children at birth,
and the most efficient way to do that was to use the birth-certificate-record
system.

In the next few slides, I am going to go through some of the ramifications
of that, one of which is that because the administrative record data are so
rich and it is relatively easy to identify individuals using them, even in a
relatively small end-sample study, with the ECLS-B, in particular, we have gone
to a model where we do not have a public-use data set. Okay?

If researchers in the room are interested in using the data, they need to
apply to the center for license, and, then, we’ll grant you the license, and we
have a relatively stringent – procedure to make sure that the data are not
inadvertently released and that no individually-identifiable information is
ever published.

That said, much like the VA, we have done a lot of work to try to figure
out ways to make the data more user friendly to the public. You know, you can
get a restricted-use license if you are a researcher and do a lot of very
interesting analyses, but a lot of the times the types of people who want to
get access to the data are school administrators or child-care providers, who
just want to get a general snapshot of what the population looks like. So we
have been working with an online data tool that will allow people to get access
to the underlying micro data, and we have done similar studies that other
agencies have done to make sure that those reports are accurate and also to
make sure that the types of data you can get out of the systems cannot drill
down below groups that are smaller than 50 in number. Okay? That is our primary
–

Yes, 50. I know. Some people think that’s pretty big. Some people think
that’s way too small. We have gone back and forth over the years. To date, we
haven’t been able to – it with 50. We haven’t really tried to drop it down any
further, but right now, that is where we are at.

In – reports, however, we will and do produce tables that have cells that
are based on ends as low as three. If it gets below three, we go on
data-suppression mode on the cells and collapse them, so people can’t even
figure out that we only have three or fewer cases in a cell.

At the center, most of our data collections are actually done through
sample surveys, and, as I mentioned, with like the in-house – the National
Household Education Survey, the sample frames themselves are often limited to
the extent that you have identifiers that you can easily use to link into
existing administrative records systems.

So, to some extent, some of these crosswalk activities might be of relative
utility to us, but I started to try to think through, well, how could we access
these rich data sets that are out there now and help them inform our studies,
and we have some experimental articles that have been put out that look at self
reports and crosswalk them with the administrative record data on like health
records or medical records, and those studies have been relatively useful, in
terms of us improving our survey items.

One way that we can use the administrative record system is to do
relatively small-end studies whereby we actually can sort of cross link and
purposefully design our study to cross link the survey data with the
administrative record data to see just how accurate self reports are and to try
to figure out ways to improve those self reports.

And another area of research that we probably should consider is taking a
look at linking the administrative record data that are out there on health
statistics with our own school-based administrative records systems.

We have relatively extensive administrative records systems that we have
for both elementary and secondary schools, and also our postsecondary schools,
and, right now, the type of data that we have really focus on disabilities, and
that has to do with some legal requirements that the department has to help
service students with disabilities, but thinking beyond that, we could, if the
data were available, use health statistics and health data to try to figure
out, well, are there students who are not necessarily of disabled status who
could benefit from additional services? And, right now, we don’t have any way
to really get those data, and I think we could use the administrative records
systems to do that.

I have a feeling that this first bullet was pretty extensively covered
yesterday, but one of the key issues that we have and would have with trying to
do some more crosswalks with records systems is actually getting the correct
identifiers in our surveys.

In order to do that, we have to have a really good understanding of what
information is available in the different records systems that are out there
that we might link into, so that we are not collecting data to crosswalk that
only will crosswalk with one record system and not another. We don’t want to
waste resources there.

And then we also need to have some help interpreting the data that we do
get from the health databases.

I just heard a little bit of a back and forth here about what exactly is in
the Medicaid data sets. We’d have to get a better understanding of the
strengths and weaknesses of the administrative records systems to use them
properly. We don’t have that kind of expertise in house right now. I mean, we
really focus on education types of issues.

I think that is it. So not too far over.

MR. LOCALIO: Well, thank you, Chris, and we have time for a couple of quick
questions before we have to take a break.

Don.

DR. STEINWACHS: It would help me to get a little bit better idea, on your
longitudinal studies you are picking up information on individual children.

MR. CHAPMAN: Um-hum.

DR. STEINWACHS: I guess linking in some of those information on the schools
and those resources.

MR. CHAPMAN: That is right.

DR. STEINWACHS: Are there other national data collections that the
Department of Education does that tracks children or are they all these – are
they special studies or is there sort of a statistical system that –

MR. CHAPMAN: There is an ongoing collection effort, basically, where we
start a new cohort every so many years of a particular population, so – Then
that ranges all the way from high school up to college.

DR. STEINWACHS: And those would be nationally representative and –

MR. CHAPMAN: They are nationally representative data. That’s right. So
drilling down below the national level really isn’t an option. The cost gets
prohibitive very quickly with these types of collections.

DR. STEINWACHS: Because, on the health side, there are some interesting
issues these days as people are very concerned about the amount of drugs and
medications being given to children, whether it is for Attention Deficit
Disorder or other problems, antidepressants. There are anti-psychotics now
being given to small kids, so on, and, in concept, you might think creatively
about or we might think creatively, I guess, about are there ways in which you
could take information like out of Medicaid or other sources that could tell
you geographically populations that are getting high rates of these and link
them to school districts or things that would tell you something, and I was
just wondering whether or not it was possible with the kind of national data,
and I guess maybe it’s probably a longer discussion –

MR. CHAPMAN: Right. The first answer is maybe. The second answer is we
start those – Especially with the studies that we do of children who are
already in elementary and secondary school system or who are already in
college, we start that sample design with basically school frames. So if there
is some way that we could link a student ID, especially once they are in
college, when we start getting Social Security numbers, then, the linking
process becomes relatively straightforward.

But for the younger children, as you were talking about, we might be able
to do some linking activities through the addresses that the schools would have
for the students and then crosswalk those into the databases that are out there
on health statistics. I mean, we do think about that stuff.

DR. STEINWACHS: Thank you.

MR. CHAPMAN: Yes.

DR. STEUERLE: This question is a question I am going to ask later, when we
get to our final session. So you might just answer briefly, if you have some
answer, but I am involved in so many projects within particular silos of
government organizations, but within the research community itself. So I am
involved with one group that is trying to study children’s outcomes that pretty
much is now focusing on early childhood education and even beyond, and has sort
of at least taken up, whether correctly or not, this model that the earlier we
intervene with children the more the return on the investment, and then I am
involved with groups like this, which is interested in healthcare, and then, at
Urban Institute, we have another group that is helping to work with some of
these longitudinal studies you are creating at Education, and sometimes I
wonder how much do the health, the economic and the education researchers
really get together when they design some of these samples.

I guess it is probably unfair to throw this all on you, except that it’s so
many cases where we are talking about outcomes and opportunities and mobility
and issues like that. People always seem to come back both to early
intervention and to education, and sometimes I don’t know how to link them.

Give you a common example. For instance, some people now think we really
should start at minus nine months, starting to measure what is happening to the
well being of a child, because it could be that drugs and alcohol, depression,
whatever other illnesses within pregnancy could have strong educational
outcomes down the road.

So the question for you is to the extent you get together, you start
designing these models, how easy is it to bring in somebody from HHS and how
easy is it for them to come, and how easy is it to bring in people from some of
these very different worlds and try to really design the models and the
longitudinal studies you have?

MR. CHAPMAN: Okay. I think at the beginning I made a disclaimer that I am
speaking for myself. I am going to do that again here. Not speaking for the
agency.

The ease that I have experienced so far has been great. It is relatively
straightforward to contact Health and Human Services or the National Center for
Vital Health Statistics and say, We are developing this study of children from
birth, and we are going to track them through the first couple of grades or at
least through kindergarten. Can we start to have some meetings with staff in
your agency that might be interested in related topics?

And for those early childhood longitudinal studies, we have had a lot of
input from Health and Human Services, and, obviously, we had to get the birth
certificate data for the – cohort, but we have also done some work with – in
our own Office on Special Education – to get better measures and to think
through measures in health.

I think what we run into isn’t always necessarily a coordination problem.
Although, those certainly do exist. We also run into just response-burden
problems, and it is good to see OMB in the room, because we can only spend so
much time with the student in a school setting or so much time with a child in
their house or with the parent in their house without running up to really
serious burden issues.

And they are good issues. I mean, we need to consider them, because, from
our perspective, we want to get as much data as we can on education. So we
focus on assessments and we focus on educational resources in the household and
in the school, and, then, we know, from a lot of research, that there’s health
issues that relate strongly to educational development. So then say, Okay.
Well, we better make sure we get some of those health statistics in there, but
it is rarely the case that we can make it the primary focus of the study. So we
are limited to the types of data we can collect, even with really good
collaboration.

Does that answer your question?

MS. GENSER: I am Jenny Genser from Food Nutrition Service.

I wanted to ask if your surveys contain information on receipt of school
lunch and breakfast, and also if you have obesity data, because that is a big,
big health-related issue that wouldn’t show up in an educational plan.

MR. CHAPMAN: Right. It varies across our studies. I like to keep going back
to the longitudinal study that we have for the early-childhood populations,
because that is the one I work on day to day.

In that particular study, we actually work with USDA to make sure that we
had proper items in there to ask about weight, but apart from that, we also
have some direct assessments where we actually weigh the child and we do a
body-mass index measurement that is part of the data collection.

That said, I don’t want to say that that happens regularly and across the
board in all of our collections. I mean, it is actually relatively unique to
our early-childhood studies.

MS. GENSER: (Off mike)?

MR. CHAPMAN: We don’t ask the parents regularly whether or not children are
in free and reduced-price lunch programs for similar reasons that are causing
problems for the CPS item on lunch receipt.

In order to really get at that, you need a good 10 questions, and we don’t
always have time to nail that down, but, at the school level, we do, in our
school surveys, ask what – how many students in the school are actually getting
free or reduced-price lunch and whether or not the program exists in the
school.

DR. STEINWACHS: I want to thank the panel speakers very much and hope
you’ll stay with us and continue this dialogue. We are at the point where we
promised you a break. We will deliver a break at this time, but what I always
say is five minutes and figure that it probably stretches a little longer than
that. So please take a break and come back in about 10 minutes or so.

(Break).

Agenda Item: Maximizing the Benefits from Linked Data:
Access for Research and Related Issues

DR. STEINWACHS: We are sorry that Dr. Citro is not going to be with us
today. She is ill and sent her regrets, very much, that she couldn’t be here.

So we have just about an hour-and-a-half and split it among three speakers,
and very happy to have Joan Turek, who deserves a very large part of the credit
for bringing together this group, and we did ascertain that Joan is the one who
knows everyone, and so certainly the right person to know, and so over the next
hour-and–a-half, we’ll be hearing from three key speakers on areas of
Maximizing the Benefits from Linked Data: Access for Research and Related
Issues.

Joan.

MS. TUREK: Thank you.

When they were setting it up, I asked to facilitate this section, because I
am a major data user, and I am probably very obnoxious about wanting access to
my data and not wanting anything to happen that could limit that.

It is unfortunate that Connie couldn’t be here. There is a disc available
either from Richard Sussman at NIA or from her, that has all of the reports of
the studies that they have done on data sharing.

The first one, in 1985, was called Sharing Research Data. The last one, in
2005, was Expanding Access to Research Data. So it looks like, over the last 20
years, we haven’t solved all the problems.

But we have three very good speakers. We are going to start with Brian
Harris-Kojetin from OMB, who is going to tell us about their activities to
improve access to data, and then we are going to talk to two major users about
what they really want to get.

I think it is important that we have the data available and we have high
quality data, but it is equally important that it is available to the people
who wanted to use it, and I think that the value of the data is really
dependent on our ability to get access. If you just collect it and stick it in
a box somewhere, you may as well save the money.

Brian.

Agenda Item: Brian Harris-Kojetin, OMB

MR. BRIAN HARRIS-KOJETIN: Good morning.

As the other speakers said, I have a similar kind of disclaimer. In fact,
if I say anything inappropriate, the Director of OMB may well disavow that I
even work there.

But I have been listening here for the past day and – well, yesterday and
this morning, and I am not sure what I have to contribute to your discussion. I
don’t have any data sets. I don’t link data. I don’t directly use any of the
data sets, but I’ll share with you some work our office does in terms of – kind
of related to this in terms of the confidentiality issues and some of the legal
issues.

Another disclaimer is, of course, I am not a lawyer. I do play one on TV
every once in a while, but – and I fake it in my job a fair amount, and, as
you’ll see, but, again, I have to disavow any of those kinds of things that I
might say that sound like I actually can make such a policy.

There’s a few laws – there’s a couple of issues on the charge questions
here that I thought I could say something about and see if it is at all helpful
to the committee.

There are several laws that have some impact on data sharing. Here are the
ones that I am familiar with.

One of the things that came out in the presentations by the folks from
Census yesterday, you heard Title 13 mentioned quite a number of times.
Certainly, a primary consideration is whatever statute – whatever authority –
legal authority the agency has that is originally collecting the information,
whether that is being gathered for statistical purposes or whether it is being
gathered for administrative purposes, the agency has to have some defining
authority to gather that information, and, oftentimes, their statutes will
specify what are appropriate uses for the information and if there are any
confidentiality provisions for that, and sometimes these are very vague and,
you know, Go out and gather data on health or on the economy and do good works
and disseminate it. Other times, it is very specific.

One administrative data set that many of you may be well aware of is the
National Directory of New Hires, and those of you familiar with it know that
there are – Well, with the exception, I guess, of SSA, every other agency that
has access to the data has access to it for a specific purpose that is very
carefully specified, and this is – So that is one example of what you are
allowed to access the data for, how it is allowed to be used, and it can be
specified by each agency that may be allowed access to it, and if you are not
explicitly allowed access to it, then, even though you have the grandest
intentions, you can’t have access to it.

Folks yesterday also mentioned a couple of broader laws that apply across
government agencies. The Privacy Act and some routine uses of information came
up. The Paperwork Reduction Act was also mentioned by a couple of folks.

Those of you not as familiar with it may be interested to know that under
the functions of the statistical policy and coordination functions that are
codified in the Paperwork Reduction Act, the director actually specifically
authorizes the chief statistician to promote sharing of information collected
for statistical purposes, but consistent with the privacy rights and
confidentiality pledges.

What I am mostly going to talk about this morning, the thing that I mostly
know about, so this is why I assume I got invited, was CIPSEA, and which many
of you are, I believe, aware of and maybe you know everything I am going to
say.

CIPSEA is the Confidential Information Protection and Statistical
Efficiency Act of 2002, which is why we call it CIPSEA instead of all of that.

It is composed of two major titles. First one is confidential information
protection, which was – which you can see the purposes here is really to
strengthen public trust in pledges of confidentiality, prohibit disclosure in
identifiable form, control access to in uses made of statistical information
and ensure that information is used exclusively for statistical purposes.

CIPSEA provides a nice statutory floor for the use of information collected
exclusively for statistical purposes.

The second part of CIPSEA is the statistical efficiency part that applies
only to three designated statistical agencies – the Bureau of Labor Statistics,
the Census Bureau and the Bureau of Economic Analysis.

The goal of this subtitle was to reduce paperwork burden on businesses,
improve the comparability of economic statistics, specifically mentioning BLS
and Census, comparing their establishment – and increasing the understanding of
the economy.

Quite a number of you, I am sure, are intimately familiar with the history
behind this, why CIPSEA was sought after for literally decades. This went
through many evolutions. There were many bills that came so close, some that
passed the House and then died, and we finally got CIPSEA in 2002.

As Nick noted earlier, if he is still here. I thought I saw him, but maybe
he stepped out, there was a companion bill – a Treasury companion bill for
CIPSEA that – for amending 6103J – that did not – I don’t know if it was ever
even introduced, but has not been passed that is really key to some of the
data-sharing provisions within CIPSEA, but that has not gone forward yet.

But one of the reasons why this has been a very important law is that there
is a real patchwork among many, many different agencies that do some kinds of
statistical activities. Now, there’s 10 agencies that are often referred to as
principal statistical agencies that that is really their sole mission is to do
statistics, but there are many others representative in this room and outside
this room that do some kinds of statistical activities as part of their other
mission that may be regulatory or providing services.

And there have been many attempts over the past number of years to
strengthen and try and standardize the statutory protections for the
confidentiality of individually-identifiable data.

As I was saying before, every agency has their own specific statutes and
has variations in confidentiality protection. Some, like Title 13, are
extraordinarily strong. Other agencies, prior to CIPSEA, like BLS, had
practically no authority whatsoever to – legal statutory authority – to base a
promise of confidentiality upon, and so this really – CIPSEA provides a
ground-level kind of foundation of protection for information gathered for
exclusively statistical purposes under a pledge of confidentiality. So I have
been saying this, now, uniform protection. It covers all data that an agency
collects for statistical purposes under a pledge of confidentiality. There’s
very strong penalties. This is similar to penalties that some agencies already
had, like Census under Title 13, and CES, under their statute, a $250,00 fine
and/or five years in prison.

It also specifically says that FOIA requests are exempt, because it defines
them as a non-statistical purpose.

When we talk about – there’s a few key distinctions that are important in
CIPSEA. One is between statistical and non-statistical agencies. CIPSEA
provides a definition of statistical agencies, those that are predominant –
whose activities are predominantly the collection compilation processing or
analysis of information for statistical purposes.

Statistical agencies have – are given some special privileges, and also
some requirements, extra requirements under CIPSEA. Specifically, one area that
I think many of you have been interested in is this ability to designate
agents, which may be a contractor or an external researcher or – this is
similar to Census’ authority to have special-status employees. CIPSEA
specifically provides this authority only for statistical agencies, not all
federal agencies.

The other key distinction that we talked a fair amount about in the
discussion yesterday is this statistical versus non-statistical purposes.
CIPSEA really puts in statute this functional separation between statistical
purpose, defined here, and a non-statistical purpose and draws a bright line
between these two uses; that is, any information collected for statistical
purposes cannot be used for a non-statistical purpose, and a non-statistical
purpose being using the information in identifiable form that effects the
rights, privileges or benefits of a respondent, that we talked a little bit
about yesterday afternoon.

So for the Census Bureau to get administrative data that were used for a
program that were used to effect – were originally gathered for non-statistical
purposes, to take that across the firewall there and say, Now, we will use it
for exclusively statistical purposes. However, it does not go back. CIPSEA is
drawing that same bright line there. Even if your intention was to get – give
it to Census, get better race codes and say, then, Can we have it back in our
administrative records? Sorry.

So what requirements does CIPSEA impose on agencies? Inform the public,
basically, that CIPSEA has to – can only really take effect here is – You are
only collecting something under CIPSEA if you are adequately informing the
respondents that you are going to use the information for exclusively
statistical purposes and keep that information confidential, and we’ve got some
forthcoming guidance that talks about this specifically as a CIPSEA pledge,
and, of course, safeguard the information and protect it. CIPSEA is law for
protecting the information. Honor that pledge that you make to respondents.

In terms of data sharing, as I said, and as many of you are well aware, the
provisions are very specific. Only business data are covered. Only three
designated statistical agencies are authorized for this business data sharing.
So that is all that CIPSEA is itself authorizing. It is important to point out
that CIPSEA is not altering existing laws that may permit other data sharing
among federal agencies, but CIPSEA did not itself authorize any further data
sharing than this business data sharing and – between BLS, BEA and Census.

So some implications here, which I think is – which you are most interested
in that you may, again, be well aware of already.

For federal agencies that are acquiring and protecting confidential
statistical information, CIPSEA may offer some new protections for those
agencies that didn’t have strong legislative protection already.

It does not – It specifically does not restrict or diminish any existing
protections. So if the Census Bureau is gathering information under Title 13,
for example, and they can only use that information for a Title 13 purpose,
CIPSEA does not restrict or diminish that at all. CIPSEA does not say, Oh, but
why can’t you do something that – CIPSEA would let me do any kind of
statistical purpose. It does not effect that is how the lawyers, I understand,
are interpreting it now.

For federal agencies providing access to confidential statistical
information, CIPSEA does permit statistical agencies – remember, only
statistical agencies – to designate agents, to perform exclusively statistical
activities. This is not a requirement for statistical agencies to do this. It
is a may. They may, if they so choose, do this.

This will require from them policies and procedures for access and control,
responsibilities for providing security and employee training and these things
take resources, as everyone in the room is well aware. RDCs are not cheap, any
other means of – Chris Chapman, if he is still here, can tell you licenses are
not free, and –

Implications for researchers, just to kind of wrap up here. One of the
things I wanted to make clear was that some people had viewed the language in
the law regarding agents as opening up researcher access to all data collected
by statistical agencies, which it really does not do. It does provide a means
for statistical agencies to designate agents, but encloses stringent
requirements on those agencies and the agents to protect the confidentiality of
the data.

It is important to remember, as I was saying, that CIPSEA doesn’t diminish
any of these existing protections, and so it is not going to remove some of the
barriers that currently exists, and it also doesn’t provide a right of access
to federal statistical data. Researchers who obtain authorization to access the
confidential data, for exclusively statistical purposes, have to share that
responsibility to maintain and uphold the confidentiality of any data they
access.

And, as you all are well aware, different agencies have – vary in the
sensitivity of the information that they have and they may not be able to
provide access to their data or all of their data or may have to do so under
varying circumstances or in different ways, and so researchers seeking access
to those data will have to conform to the agency requirements and respect those
confidentiality provisions, even if those are more limiting and restrictive
than those that they may have for – in their own institutions or they may
encounter with some other data sets, but I think that was – I had to share with
you this morning.

MR. LOCALIO: Thank you for your presentation.

I just have to say that one of the problems that I mentioned yesterday, if
you were here, is that people refer to, The lawyers have done this, and the
lawyers have done that, but the lawyers are not here, and they do not
understand the problems that we are discussing. In fact, I am not sure that
they care about the problems that we are discussing.

MR. HARRIS-KOJETIN: I disagree with you, and some of the lawyers we work
with, but they are very, very well informed on this issue.

MR. LOCALIO: What I find, and I have reviewed some of the legislation, I
have a copy of CIPSEA here that I have been carrying around with me for the
last three years, and I have a copy of the statute that NCHS uses, and they are
in conflict. They are vague. Some of the things are vague, and it doesn’t seem
there has been an effort to reconcile them.

Is there any further effort, post-CIPSEA, to say how these statutes are
going to work in practice? Is there any effort to evaluate the implications of
these statutes, in terms of who gets what, when, where or do they just pass it
and then they say, Well, this is going to work?

What is the evaluation component of – Does OMB have an evaluation component
to figure out whether this is working? And I am not talking about the second
provision, the data sharing among the three agencies. I am talking about
essentially the first one.

Thank you.

MR. HARRIS-KOJETIN: Very pertinent question and I don’t know that the
evaluation is strictly in the law, but I think that is something we do care
deeply about or I do, since I am speaking for myself.

We have got guidance forthcoming on CIPSEA. Some of you may have heard me
say this for the past two or three years. Nick Greenia, in the room, I think
has just given up asking me when it is coming out.

It is actually coming out very soon, but you can’t really trust whatever I
say on that issue, since I have said it has been coming out soon now for two or
three years.

So we do have some fairly lengthy, in terms of relative to the statute,
like about 30 pages of guidance on implementing CIPSEA that we will be issuing
that will help agencies that are using that.

We have gone through quite an intensive interagency process to help develop
and inform this. Again, if you folks in the room were on an interagency team
that has helped give OMB input into doing that, and so as that gets out there,
as agencies are going forward, we have had a number of questions that have come
up from agencies in terms of what impact it has on their programs, how they
need to – how it will effect their operations. We have been dealing with those
on an ongoing basis, and, again, have used that to help inform the broader
guidance.

So it is an evolving process and it is something that every agency is going
to struggle with a little bit in terms of what does this – what kind of changes
and what kinds of things does this do for us, and we have some reporting
requirements for agencies to get back to us on how they are using CIPSEA, how
they are using specifically the – the statistical agencies are using the
agent’s provisions, and so we can evaluate and monitor this and see where
things are working.

Many folks are, of course, very interested in the data-sharing portion of
that and hoping that we could go back to Congress after some time, after we
prove how good of a job that the three designated agencies have done sharing
business data, to see if we could explicitly expand that authority, which is
where I thought you might go, even though you pulled back from that.

DR. STEINWACHS: I think, just clarify that. There was a reference in there
to business data, and so that was the sharing among BLS and the others of
information on employers in business activities in the U.S.?

MR. HARRIS-KOJETIN: Exactly. Yes, economic data, and so the business lists
between Census and any of the economic surveys that Census and BLS did.

DR. STEINWACHS: Thank you.

MR. PETSKA: Can I make a comment also?

Again, speaking from my own personal view, I wish that CIPSEA would have
gone further and kept in the tax component of that, because when CIPSEA, when
the early discussions of CIPSEA were unfolding with Brian’s boss, Kathy
Wallman, at OMB and other representatives of the federal statistical agencies,
there was a lot of valid research purposes that were articulated for sharing of
data, including tax data and so on, and once CIPSEA started to be formulated,
it became very clear that the – one of the more controversial aspects was
tax-data sharing, and the question is – I don’t know if it is because of
congressional committees. Clearly, there was not strong support on the Hill.

Randy Krosner(?), in the Council of Economic Advisors office, was pushing
this and so on, but when he left the administration, it seemed like that piece
had no possibilities at all.

I spoke to him a couple of months ago at a conference in Cambridge and his
comment was, CIPSEA, as it was, was the best deal we could get, that we would
have liked to have expanded data sharing to more agencies, including the tax
component, but it was clear that that bill would be dead on arrival, which is
unfortunate.

MR. GREENIA: I am Nick Greenia from IRS and I just wanted to add a couple
of things.

First of all, I wanted to address the previous question in terms of the
evaluation, because the guidance that is, we hope, going to be coming out next
month – Brian, is that right? – in the Federal Register –

MR. HARRIS-KOJETIN: Absolutely.

MR. GREENIA: The guidance is actually – I was on the other agency committee
for that. So I can speak to that a little bit.

I think the outlook is unknown. I think the answer to your question is it
is not clear, and one of the reasons I say that is because there’s a lot of
flexibility in terms of how agencies can safeguard the data, how they make the
data accessible to researchers, and I think what may come about is, if you
will, a de-facto evaluation, which is that researchers and Congress, if there
is another data-sharing effort, are going to look at the experience and they
are going to see that, You know what, it depends on the sensitivity of the
data, in terms of what sort of safeguards are prescribed, and procedures, and –
are on, and there is going to be a lot of flexibility, and there is going to be
a lot of variability in terms of how the data are accessible and protected.

And I just wanted to add something to what Tom Petska said, since Connie
Citro is not here, as you may know, since that report on the data-sharing
workshops was released last Friday, and if you would like to get a – I am doing
a little plug here for Senstat(?), of course – but if you want to get an idea
of some of the difficulties, including the recommendations facing tax data for
purposes of data sharing, I would highly recommend you read the article
coauthored by Mark Mazor and myself on tax data and some of the many, many
issues that have to go into that.

And picking up on what Tom said regarding why the tax-amendment bill did
not go anywhere, as you know, the tax-amendment bill accompanied CIPSEA in July
of 2002 to Congress, and CIPSEA proceeded to the floor of Congress, and the
J-bill, the amendment to the tax bill, went to Joint Tax Committee, and there
were a number of reasons, we think, that the tax bill foundered. Tom has put
his finger on one of them, which is the leadership vacuum, but there are some
other lessons that we think are valuable as well, including freezing the items
in the statute itself, as opposed to allowing infinite regulations to stipulate
item content in the future.

So I highly recommend you take a look at that article for tax data.

MS. TUREK: Thank you.

I have one question also.

We have been talking here about new forms of data, that would be the survey
and administrative data linked together.

Has OMB begun to look at the implications for users and to look at whether
or not we would need any kind of new statutory language to permit this kind of
data to be shared? Because, I mean, it would not be identifiable, presumably,
but there was the risk-disclosure issues.

And will OMB take a position or will it look at what should be done to help
users get access to this data once it is available?

If you do a SIPP that’s got a match to administrative records, will you
look at whether or not we could have a public-use tape?

MR. HARRIS-KOJETIN: I thought David was going to look at whether you can
have a public-use tape or not.

You’ll have a public-use tape.

MS. TUREK: Thank you.

I mean, I just wondered if OMB had a role in this or –

MR. HARRIS-KOJETIN: We could have a role if we need to have a role, if it
seems that that would be helpful.

Obviously, when you are bringing in the linked data, there are other things
that go along with it. The example I was giving before, an agency certainly can
promise confidentiality to a statistical – Well, a statistical agency can
promise confidentiality to another agency, and so BLS, for example, does this
all the time when they gather information from states. They take data that
states may not consider confidential, but say I will use this for exclusively
statistical purposes and I will keep it confidential, and once it goes back
there, then BLS intermingles it with their other information, and, in essence,
they have elevated the level of protection required. Just like whenever
anything gets intermingled with IRS tax data, it gets elevated to that status
of protection.

MR IAMS: Could I make a comment?

I am Howard Iams from Social Security.

I really think Tom is correct and Nick is correct. You have to have
legislative authority to permit a broader sharing than currently exists. The
agencies do not have the authority to pass data to other agencies for use at
those other agencies for statistical purposes, and this is separate from
disclosure and confidentiality. The agencies just cannot pass it and cannot use
it without some sort of legislative authorization, and Brian can – I don’t know
– perhaps disagree on some instances, but I don’t think that the agencies that
are interested in this can go further than what they are doing now, and a lot
of the limitations that you are hearing about are created by this legislation.

Now, the disclosure raises a – Well, let me finish – My train of thought
would be that if you are a CIPSEA-authorized or a CIPSEA-compatible agency,
which I think I am in – We are a statistical outfit in a big administrative
organization, but we just do statistics and policy research.

There ought to be, ideally, permission to share confidential agency data
with such a group for them to use it however they wish for whatever purpose
they want, not just Title 13, not just Title XYZ, but that it is a
statistical-analysis function that might have policy implications, might not,
but it is for a statistical purpose. It is not going to go and administer
somebody’s benefits or effect some individual’s rights and whatever, as CIPSEA
is defined. That would open up a whole lot of sharing that outfits could do and
a whole lot further analysis with this type of information than is currently
possible.

Once you bring in the disclosure, confidentiality issues, you raise a whole
lot of other things, and my only comment would be, the thing that I think
undermines almost any public-user file is geography.

We put a copy of our new beneficiary survey – It is sitting on the web.
It’s got two surveys. It’s got earnings from our tax records. It has hospital
records from the Medars(?) file. It has our benefit information.

We cleared this through – with Pete Saylor’s(?) help at IRS through their
requirements. It meets all their confidentiality requirements. It meets
Medicare’s, CMS’s – at that point it was HICFA. It meets SSA’s. The key is
there is no geography. It is a big country out there. There are a whole lot of
characteristics that you could say are unique, but you really can’t tell,
because it is a big country out there.

If you know what state someone is in, it is over. It is not a big country.
It is a small state.

Now, we have a national program. For Social Security, it doesn’t matter
what state or what locality you are in, but if you are dealing with TANIF(?),
you want to know the state.

I, being selfish, think that they ought to put out all this administrative
data on a national file with no geography, and if you want to use geography,
you should have to go to these research data centers. That is what the
University of Michigan does with their Health and Retirement Survey. There are
restrictions. Users can use it in University X in Alaska, Hawaii, whatever. The
only place you can do geography is in Ann Arbor, Michigan. That is the price
you pay to have geography with those data. If you’ve gotta have geography, you
will never have a public file with confidential data. It will not be possible
in today’s age. My judgment.

MS. TUREK: I find that fascinating.

I think we ought to go to our last two speakers who are both users, and
actually will be from a very different perspective, I think.

Heather Boushey is an economist with the Center for Economic Policy
Research and Dr. Deb Schrag is – I guess you are on the staff of the Memorial
Sloan-Kettering Cancer Center, and so we have here an economist and a medical
doctor who are both heavy data users. So I think it’ll be really interesting.

Heather.

Agenda Item: Heather Boushey, Economist, Center for
Economic and Policy Research

MS. BOUSHEY: Great. Thank you. Thank you so much, Joan. Thank you for
inviting me to speak here today. It is a pleasure to have the opportunity to
talk to you about the way that we use data.

Before I talk about the main points that I want to make, I want to tell you
just a little bit about myself and my organization and what we do, because my
understanding is is that is why I have been invited here to speak today, to
talk about how we use the data that agencies make available.

I am an economist. I work at a think tank here in town called the Center
for Economic and Policy Research. We are a very small shop. We have four
economists, about a staff of 15, and we do research on economic issues facing
people here in the United States.

We are very heavy users of the CPS and the – the Current Population Survey
– and the SIPP, the Survey of Income Program Participation – although, I don’t
know, in this audience, if I need to spell out those acronyms, but I am so used
to doing it.

We make use of this data in a very timely manner, both to effect media
debates and policy debates around pressing policy issues.

We work on very short time frames, and, because of that, we have spent the
past five years taking both the SIPP data and the CPS data, both – we are
working on the March(?), but we have done this for the org – and creating what
we call Uniform Data Files.

As many of you know, if you use survey data, they do these fabulous things
at Census and BLS where they’ll change the name from year to year or they’ll do
little things that mean that you can’t just write one piece of code and pull
out the data from every year.

So you have to invest a lot of time if you want to know what has gone on
between 1973 and today or ‘79 and today. If you want to have a time
series, you have to sort of make this huge up-front investment.

So we do that, and we have made all of this publicly available on our
website, all of our code and our uniform extracts, but we have made this
investment so that when there is a debate on the Hill or when the media says
something that is inaccurate about what is going on in the economy, we have a
data set that is up and running, that is on our desktops, and we can then
comment on it in days or weeks, rather than months or years, and I know all of
you know just how complex this work is, and being able to do the work up front
and have it available for timely analysis is a critical part of our mission and
how we use the data.

Which is not to say that we don’t have longer-term research projects. We do
projects that take years, but it builds on this uniform data file and it is
always with this goal of policy work.

We do not do any linking with administrative data, because it is certainly
beyond the kinds of timely work that we could do.

I do have experience working with administrative data in another life, when
I was at the New York City Housing Authority. So I do have some understanding
of just how complex some of those issues are, but I have not matched it, just
so that – I don’t want any questions about that. I have never done it. Don’t
want to. Sounds complicated.

So being able to have data on our desktops that we can use quickly and
accurately and that we have confidence in because we have already done all the
background work has been critical to our success, and it is that major point
that I want to relate to the main points I want to make to you here today.

We are concerned about timeliness of the data, and we are concerned about
our access to it. The question that Joan said about public-use files is
critical to what we need to know.

And we are also concerned about – and I don’t know how germane this is to
this topic, but we are concerned about maintaining access to survey data. I
think – and I’ll talk about that just for a few seconds at the end.

So my first two concerns about timeliness and accuracy of the privacy
issues, of course, they are linked, and so the major questions that we have are
will administrative data that has – survey data that has administrative data
matches lead to delays in releasing the data? Will it be less timely than it is
now?

And, second, will it require new security measures, kinds of like the
things that have already been talked about requiring us to go to special
locations or special sites to use the data, and that will significantly both
delay our ability to use it, but not just delay it by days or months, but could
delay it by years, because we wouldn’t be able to have it sort of up and
running and ready to go when we have an issue.

Now, I have not been doing this kind of work for maybe as long as many of
you have, but I have heard tales from people who got their Ph.D.s in the
‘80s and before that it used to be the case that if you used survey data,
you had to go to special computers, because you couldn’t do it on your laptop,
and I hear they had these things called cards and that it was very time
consuming, and what I find – the point here is that I think that the way that
we are able to use data now has transformed the way that we are able to engage
in policy debates, both at the national level and the state level.

The fact that we have access to this data, not just on our desktops, but I
have access to it on my laptop at home, I can do it on the train, that we are
able to do these very complex kinds of work in a much faster way than we had
been able to in the past.

I think you can – There is some correlation between that and the rise of
think tanks like mine and private research organizations that are effecting
policy debates, both at the state and local and national level, and I think
that that is an important accomplishment that technology has given us, and we
don’t want to sort of move backward in any way. So I think that that is just a
very critical, critical point.

To give you a couple of examples, when I have been asked to testify in
front of Congress, both in the House and the Senate side, I had no more than
two weeks’ notice, and, in one case, I had just five days’ notice.

These are very short time lines, where, if you want a specific number from
your data, you need to just be able to go to your computer. You don’t have time
to go and to sign up and to wait.

But the second kind of concern we have about timeliness is that we often
only have a few months lead time to know what the issues actually are. We spend
a lot of time thinking about what kinds of policy issues are going to come up,
what do we need to prepare for over the next year or two, but we may not know
whether or not Congress is going to vote on minimum wage this session or next
session until a few months out.

So having to go through application processes is simply just not viable for
those of us engaged in policy debates. It is perfect, it is fabulous for
academics, and we build on their work and use it, but we need things that are,
of course, shorter.

And on this issue, I might add that this is something that our organization
is on sort of the more progressive end of the political spectrum, that we have
been working very closely with the Heritage Foundation on these issues about
access and timeliness, because they are just as concerned as we are, and it is
certainly a point that transcends, I think, political boundaries and sort of
goes beyond left and right, which I think is a very important point, and
especially this issue about having independent organizations be able to access
government data to discuss pressing policy issues is one that we all, both on
left and right, agree on.

I basically am making the same point over and over again. So I won’t be
that much longer here. We need access to timely data. So, hopefully, that has
gotten through here.

So the final question in the set of questions that we were given ahead of
time to focus on was what are the potential costs to the public from failure to
take advantage of these opportunities, and I have a couple of comments on that.

First of all, one of the largest projects I am engaged in right now, we are
looking at take-up or effective coverage of benefit programs in 10 states. We
are doing this for advocacy purposes, for public policy, and we are doing it
using the SIPP and the CPS and the National Survey of American Families.

Now, we know that each of these data sets has significant problems with how
people report their benefits, that there is under-reporting of benefits. It
would be absolutely fabulous to be able to have data that is matched, so that
you can look at eligibility for public programs and then get the numerator to
be an actual estimate of coverage. That would be a significant improvement,
and, right now, we are working on this project, because, in the states, many of
the state groups that we work with are very concerned about take-up of public
programs, and because there is no real place that makes a lot of this
accessible across a wide range of programs, because the eligibility rules, on
the one hand, are so complicated, if you want to look at eligibility, you have
to use survey data, and, quite frankly, the SIPP is the only survey that I have
used that has enough questions to really get at the complexity of eligibility,
which, of course, as Mr. Iams said, this is all at the state level, this game
is all at the state level, but we really don’t have a good numerator, because
we don’t have administrative matching. Being able to have that matched data
does have – I mean, significant policy implications that we could be using
right now. So that would be wonderful.

Of course, all these issues about privacy, I leave it to you all to sort
that out, but we would love to have access to it.

But I do have a couple of concerns about sort of the move to the matching –
particularly, and, of course, my perspective is thinking about this matching
either SIPP or CPS or ACS, one of the surveys that I have used, with
administrative data.

My first concern is, with my limited experience using administrative data,
I think that there are concerns about accuracy and how high the – what the gold
standard is.

It seems to me that we need three kinds of data. We need administrative
data that tells us one thing, but there are biases and there are – obviously,
there’s problems and there’s errors in that data as well as there is with
survey data. The biases run in different directions.

We need survey data to tell us about the wide populations, the full scope,
and I think we also need qualitative data to tell us some of the why questions,
but that is a whole different group of people.

This question about whether or not the administrative data is always
perfect and what we are going to gain from matching and how we talk about that,
especially with the public and especially with the people that are trying to
convince how important this is. I think it is an important note to just note
that some of the caveats and some of the potential problems with that, in terms
of accuracy.

Second, administrative data is clearly no substitute for survey data, and
this cannot come at the expense of these surveys.

Again, looking in the issue that I look at, eligibility for public
benefits, this is something – and we need to know how many folks are eligible
for Medicaid and who aren’t receiving it or who are receiving it. The only way
we can do this is through surveys that ask a ton of questions of people at a
subannual, a monthly level, because this is the way that people access these
programs.

I mean, and just to go off on that just for a second, one of the things we
learned from our work with the SIPP – and looking at take-up – is that people
are – People move up – their incomes move up and down month to month, and when
they access the system is not necessarily the month that they don’t have any
income, because it takes months – weeks or months for them to even make it to
the office or get on line or get all their papers together to receive Food
Stamps or another benefit program. You need access to the survey data that
provides you with those dynamics. They are not substitutes.

And then my final point, which is going back to my theme here, if the cost
of matching is that we lose in terms of timeliness or the ability for the
public to access public-use files, then I think that is a serious concern and
one that we should spend a lot of time focusing on.

And I think, having said that message about 12 times here, hopefully, it
has come through, and I think I will stop there. So thank you very much for
allowing me to speak to you.

Agenda Item: Deb Schrag, Memorial Sloan-Kettering
Cancer Center

DR. SCHRAG: So I am Deb Schrag. I am a physician and health-services
researcher at a big cancer center in New York City, Memorial Sloan-Kettering
Cancer Center, and I also appreciate the opportunity to speak to this group.

Unlike Heather, I don’t do anything in days to weeks. I am more
representing the perspective of academics, and we do everything on a months to
years time frame.

I am going to – I guess, I have to say that I had a completely different
talk when I came here yesterday, and perhaps it is being towards the end of the
workshop, I revised my slides and got rid of most of the data examples and
slides that showed the results of various linkage projects I have been involved
in, and put in what I’ll call more philosophical slides, because I think some
sort of more conceptual framework – Maybe it is just – at the end of a
workshop, it feels like a conceptual framework tying together all these
different enormously complex issues that we have heard about for the past two
days is sort of in order and maybe there’ll be some discussion in that regard
at the end.

Again, I represent an end user, not my institution or any agency.

So types of research questions, examples of linkage attempts, challenges
that we have encountered and, of course, a wish list to add to Heather’s.

So I guess I am here representing academic health services researchers, and
we examine, obviously, relationships between need, demand, supply, delivery and
outcomes of healthcare.

The big topics for us, I would say, over the – since – in this decade – and
I think that these are going to remain front and central on people’s research
agendas – are disparities in healthcare, access and barriers, technology
dissemination. Quality measurement is a big one, and, ultimately, efficiency of
healthcare delivery. So that includes all the cost issues.

We talk about data – and I think that this is an underlying theme of many
of the presentations that we have heard in this workshop – is that these data
are layered.

We start out with source populations, basically, United States citizens,
who have IRS data. They work. They don’t work. They do save for retirement.
They don’t. They exist in specific geographic regions of the country, and on
top of the source populations are diseased populations. I happen to work in
cancer. Other people work in mental illness or psychiatric disease or
malnutrition, all kinds of examples of what one consider a diseased population
or a population with a health concern of interest.

On top of that are providers. Typically, these are physicians, but other –
nurses, other types of healthcare providers as well, and, on top of that, are
healthcare delivery units, facilities, whether they are clinics, hospitals.

The issue with federal data is that federal data is better at the bottom of
the pyramid. Federal data has a lot of information about source populations and
some – For example, Dr. Breen is here from the National Cancer Institute. A lot
of information about populations who get cancer. It tends to be a lot less rich
as you go up to the top of that pyramid and have a lot less information about
providers and facilities.

So when we try to link, very often, in my experience, what health services
researchers are trying to link is the rich, rich government data at the bottom
of the pyramid with more granular detailed data about providers and facilities
at the top of the pyramid that often resides outside the public domain, if you
will, and I think we have heard allusions – you have heard references to some
of these data sources from the speakers yesterday. AHA data was mentioned, AMA
data, and I’ll give you some examples.

We talked about evaluating the quality of healthcare. Really, we are
interested in health outcomes, and the main ones – main health outcome we get
out of big federal databases are typically just very basic things like who
lives, who dies and who gets particular diseases. So mortality and incidence,
basically.

And the inputs that we want to go – that we want to relate to health
outcomes are community attributes; person attributes; health risks and
behaviors, which come from the big surveys; the structure of delivery systems
and the processes of care, processes of care. I would probably put Medicare
data in that bucket. Medicare data is what exactly are we doing to these people
– all of us – that lead to these health outcomes.

As we think about linking data, I think it is helpful to think about what
the frameworks are for putting data in different buckets, and, now, obviously,
some data belong in multiple buckets. Medicare data also has mortality, which
is an outcome, but I think that sort of as we conceptualize these linkage
exercises, it is helpful to think about in what domain the data sets belong.

The other thing that I think is helpful to think about, and I always think
about when I am contemplating any sort of linkage project is where it lies
along the spectrum of pure population-based data.

So federal-agency data are best for big, broad population-based analyses.
So a cancer example is I want to know something about everyone in New York
State with lung cancer, and I can go to the registry and Census data, but, very
often, increasingly, we are interested in quasi-population-based data, where we
want everyone in New York State with lung cancer who is covered by a particular
private insurance provider – Oxford Insurance Plan – and so I think it is
really also important when we talk about linkages to be clear. Is this true,
pure population-based data? Are we trying to link federal agency or state
agency data with some external data source that resides elsewhere? We need some
kind of nomenclature for where those boundaries are.

And, then, of course, there is non-population-based data. It may be that –
You know, my research institution is always coming to me and saying, Well, you
link these data and you work with these large population data sets. We want to
know why cancer patients in New York State are not all coming to get their
treatment from us, since we are the best center, and you have all these data.
Why can’t you do that?

And I say, guys, that is marketing. That is not health-services research.
That is not an appropriate way to use these data, but explaining sort of
analyses at the population, the quasi-population and the non-population data,
we need some kind of taxonomy, and I think simply having that taxonomy or the
government help develop standard taxonomy for these types of activities would
be a really helpful place to start and would educate the end-user community.

Obviously, the health services research strategy – I mean, we just all want
to get our hands on as much data as we possibly can as quickly as possible, and
we want to juxtapose all these various data sources.

So the kinds of things that we work on, really, I would say the focus and
theme of our research is to look at what we call the implementation gap, which
is the difference between clinical efficacy and effectiveness.

So most healthcare – What works in healthcare is discovered and described
in these very neatly, nice-packaged little clinical trials where we take 100
people and give them blue pills and 100 people and give them red pills, and we
decide that the red pills are better.

What we get out of that is efficacy. These red pills work, but that really
doesn’t tell us anything about what happens when we unleash red pills on the
population.

When we unleash red pills on the population, we are trying to measure
effectiveness, that gap between efficacy and effectiveness – me and others call
the Implementation Gap, and we are really trying to get at what the reasons are
for those gaps and trying to identify important sources of variation and
particularly those that we can do something about, and we want to know whether
the reasons for the gap are endogenous to patients, doctors, healthcare
systems, background population. So that is really the unifying theme of the
research and why access to these data are so incredibly important.

We told you a little bit about this data source yesterday. Very simple
example, a kind of chemotherapy given after an operation for a particular stage
of colon cancer, and this is a big deal. Fifty-thousand Americans get this
condition a year.

Do patients, in the Medicare population, who are insured and this kind of
chemotherapy is covered. Do they actually receive these treatments?

Well, we went to SEER-Medicare, and, very quickly, within a week, were able
to answer the question.

Now, of course, it took a while to get the data. Once we had the data, the
analyses took a week. So, again, I spend 90 percent of my time trying to get
data, manage permissions and so on, and actually analyzing it is a lot less
time consuming.

But we identified a very simple finding, which is that there is a very
steep gradient, and that, although we treat most young Medicare beneficiaries
with this kind of chemotherapy, we really don’t treat the older folks.

Well, this very simple finding, made possible by linked data, really
sparked a whole set of subsequent more detailed analyses to go back to
physicians and patients and to conduct interview studies, to really hone in on
what the reasons are that underlie this important healthcare-delivery pattern.
Okay?

Now, one of the problems here is that the older patients were never
included in the randomized trials, those little efficacy studies. So doctors
don’t know what to do. So there is uncertainty, and this is what actually
happens.

Okay. So there is a real circularity where we do these population-based
data analyses with linked data, and that, basically, catalyzes subsequent
studies to really get at the underlying reasons.

So we have this nice linked-data set, but we really wanted to know – We
said, Look, these people are not getting a kind of chemotherapy that they
really ought to. What is going on? Are they just not being referred to medical
oncologists who have the therapy? Are they refusing the therapy, even after
they go to medical oncologists and are old people just saying, Thanks, but no
thanks?

Well, to do that – Sorry. This is Census data that just shows that we also
– Census data can be helpful because we see that if you are married, you are
much more likely to get the right kind of treatment, chemotherapy, than if you
are widowed or single, and these are all adjusted for age, and these are very,
very strong findings. So having Census data can be very, very important, and we
can figure out who is at risk for getting inappropriate medical care.

So why doesn’t everybody get chemotherapy? Do people refuse? Do they see a
medical oncologist? We could use the UPINs on CMS claims, and CMS claims have
access to some specialty-code information about the types of doctors patients
receive, but that data is not particularly complete, updated or accurate. Maybe
Gerry can comment on it. There are other better data sources for figuring out
provider characteristics.

So when we use just the specialty datas, we could see that among the
patients who got chemotherapy is represented by the green bar. Most people saw
an oncologist. The top 20 percent there in the chemo bar, those people got
chemotherapy, but, apparently, did not see an oncologist. All those people saw
internists. Well, that is just because CMS doesn’t know the difference between
an oncologist and an internist, because the data is not coded well, but we
wanted to get our hands on better data sources.

The people who didn’t get chemotherapy, most of them made those decisions
without seeing an oncologist.

So we are able to basically do these kinds of analysis to say, We, in the
healthcare system have a healthcare delivery problem, patients are not making
informed decisions not to get a treatment. They are making uninformed
decisions, because they are not even going to see the relevant providers.

People look at this data and it wasn’t all that compelling because they
say, You can’t even figure out who is the medical oncologist. So, then, we
wanted to get AMA data.

It took essentially 18 months to get the AMA data, which is much more
complete, to be able to do the linkage to really prove that to a higher level
of satisfaction. Very complicated to do.

Ultimately, successful, and, then, the green bar went all the way up 99
percent, and the green bar in the no-chemo group went up to about 40 percent,
and our conclusion was, essentially, that the mechanism was people were not
appropriately being referred.

So a wish list from an end user would be linkage of UPINs on claims data to
files that describe position characteristics.

AMA data is better than CMS data. Data from the American Board of Internal
Medicine, the American College of Surgeons, all the specialty societies, is
still better than AMA data, and the untapped resource is state-level data. That
is the most complete and the most difficult to obtain.

So I have a license to practice medicine in the State of New York. They
know lots about me. They know whether I have ever committed a felony, been in
jail, all the tests I have taken. They maintain it. I pay them $250 every two
years to update that license.

So we don’t have that data to link to federal. So with respect to health,
it is really critical to know physicians, physicians’ characteristics,
distribution of physicians.

Next on the wish list is pharmacy claims. The analysis I showed you, we
want to know were people not getting intravenous chemotherapy because they are
getting oral chemotherapy? Are people not getting supportive medications? Are
they sticking to their therapies? Are they getting appropriate pain control?

So the wish list would be Part D data. We heard about that yesterday.
Medicaid data for pharmacy claims, and private claims data sets. There are
enormous pharmacy-data clearinghouses that are very – that have not been widely
linked, but are very important for health-services research.

So, again, the example I just gave you involved taking federal data set and
trying to link it to external data sets that are not federal. So I think
developing some kind of taxonomy or framework for what these linkages are – Are
you linking federal to federal? Are you linking federal data to state data? Are
you linking federal data to private data with a broad public relevance?

And I put AMA data, AHA data. Those are private data maintained by
non-profit – Well, yes, AMA is a for-profit organization, but they are big
organizations that have – really control, monopolies on important data sets
that pertain to health that have broad relevance for many researchers in and
outside of government.

And, then, there are custom data, where you have your own personal data set
about the patients in a particular region or with a very specific set of
disease that you have that you then want to link to.

And I think developing some kind of taxonomy to understand what the
activities are would help frame development of coherent policies and rules that
researchers could understand.

So an example of a study I am working on – and Gerry Riley has been
extremely helpful to us here – are to look at capacity to deliver mammography
in the United States.

Women in the United States, age 40 to 80, need mammograms. A lot of them
are unscreened. There are big racial disparities, and the lack of available
facilities, mammography screening centers, and lack of radiologists are
potential explanations for suboptimal use.

So the question here is does lack of capacity explain geographic variation
and racial disparities? And does capacity predict breast-cancer incidence and
mortality?

Well, these kinds of analyses require geo coding. They require knowledge of
where the facility is, and you can get that from FDA accreditation data. Where
are the radiologists? Again, that is physician data. Where are women
unscreened? BRFSS, Medicare data are informative there. And where are there
high rates of breast cancer? SEER.

To do these kinds of analyses, we want data ideally at the Census tract
level, but if we can’t get it, we’ll go less granular to the Zip Code or county
level.

Trying to do a project like that and figure out where to start to go to
obtain the permissions is extremely complex. Approval from one agency or many.
So some kind of central clearinghouse and more clearly delineated set of
procedures would help.

To get these kinds of projects done, we are really dependent on personal
relationships with key individuals who sit in specific agencies.

For example, Gerry, in this case, helped us get access to the FDA
accreditation data for mammography facilities, but people who don’t know Gerry
can’t do this, and that doesn’t really seem fair. Although, we are very happy
we know him.

Finally, I want to talk a little bit about area versus person-level data.

Access to granule-level data helps most health-services researchers, and
privacy security concerns obviously involve less risk when you are talking
about area as opposed to person-level data.

I think, again, in all these discussions, it is really clear it really
would be helpful if we delineated between what we are talking about, and I
think Howard was alluding to this before.

So, again, area-level data at the bottom, state, county, Zip Code, Census
tract. On top of that, you often have anonymized patient data, which can be
linked to unit area, and, on top of that, what is the most vulnerable is
individual patient data, and I think one thing to think about, in terms of
linking federal-agency data sets, is to release them – again, I am not
pro-public-use data – but to make them available to the research community with
appropriate bars and jumps and hoops you’ve got to jump through is to make them
– make area-level data available without making individual patient-level data
and have discussion among the agencies of what the hoops are to get state,
county, Zip-Code, Census-tract level data with higher bars the more granular
you go.

So, again, this would really help us with very repetitive common tasks that
we end up performing again and again when we try and make maps and figure out
where are the patients, where are the providers, where are the disparities and
where are the mortality rates.

Wish list in that regard would be access to chloropleth maps by various
geographic units, very useful for common-data elements and Census and
survey-data results. It could be a shared resource for investigators, and ARC,
GIS and other software packages that have really just become available in the
last three or four years have catapulted the possibilities in the ease – how
easy it is to do this light years ahead. I would say even in the last three
years. Others might wish to disagree.

I think I am going to skip this and talk, finally, about Medicaid, and if I
had to put something at the top of my wish list it would be for the federal
agencies to help us figure out how to tap in more effectively to Medicaid.

I think the states just don’t have the organizational capacity to get this
going, but the federal government does, but the states really care. The largest
component of their budget, talking about healthcare, I guess, at this workshop,
Medicaid funds healthcare for the poorest, sickest members of society. It is
really an untapped resource. I think CMS has done an enormous amount of work
over the past decade trying to get the data into some common file structures
and make it a little bit easier to work with, but it is still an untapped
resource.

I think when we talk about Medicaid data, we want to distinguish between
two things. One is enrollment. Who are poor people enrolled in Medicaid versus
the administrative data which is what is done to those people, are they getting
EKGs or chest X-rays or what units of healthcare are they actually consuming,
and it may not be so complete for the latter, but very informative for the
former, and I think we really have to not let the perfect be the enemy of the
good when we talk about Medicaid data.

So this is just an example of a study where we tried to link
cancer-registry data from the entire State of California to Medi-Cal, which is
Medicaid claims for the State of California, because we wanted to know about
delivery for cancer care to poor patients, and we were wondering what would the
yield be from a linkage of California cancer registry data and Medi-Cal.

So what we did is we started, and you’ll note on the left, with all
incident cancer cases reported to the California Cancer Registry, which is very
complete, and we took 98 cases, and if you look at the cervical-cancer example,
there were 1,690 women diagnosed with cervical cancer in the State of
California in that year. About 80 percent of them were 18 to 64 at diagnosis.
We don’t care about the older ones, because we have them in Medicare, right? So
that is about 1,350 cases.

What proportion of them were enrolled in Medicaid? About 21 percent. If you
look down at hepatoma, it is about 35 percent. So these are cancers that are
associated with infectious diseases. These are important health problems,
Hepatitis B and C and the HPV virus can be prevented. That links, therefore,
back to health surveys.

You know, we think that 21 percent of the population of a state who has a
particular cancer or 34 percent is a meaningful fraction and that figuring out
how these people are cared for and whether they are getting antecedent
appropriate care and appropriate care subsequent to diagnosis is important and
that these kinds of linkages could be leveraged further.

A lot of problems were encountered. So for the example of Medicaid, we have
big problems figuring out the duration of Medicaid enrollment. So we had
enrollment files for two years, and we looked over two years – in this case,
‘97 and ‘98 – only about half of the cohorts were enrolled for the
whole 24-month period, and a lot of people were in and out, and the way these
denominator files are maintained is very confusing.

We had about 74 percent of patients who were enrolled during the month of
diagnosis. Some were first enrolled after and some before.

When we looked inside the claims to see how often a diagnostic and
procedure code inside the claim corroborated the diagnosis that we found in the
California Cancer Registry, the answer was about 80 percent of the time. We had
access to a year of data. So if we looked over the whole year, it was about 70
percent. If we restricted the analysis to people diagnosed in the first half of
the year, it was 80 percent, because it can take a while for these codes to
catch up and appear in the claims.

Is this perfect? No. Is this useful? Absolutely.

So SEER-Medicaid, we attempted a link in California. It took us two years
more to obtain the data sets.

The denominator file structure limits our ability to identify cohorts of
the chronically poor easily. Challenges are retroactive enrollment, chronic
versus episodic poverty, spend-downs – that is, people for whom illness
precipitates enrollment – variation in state thresholds in generosity when we
try to do this with other states, and I won’t go into it, but, believe me, are
trying, and definition of an HMO or managed care. Just some of the challenges.

But we think that sort of a coordinated federal approach that is helping
will help the states and some of the states would undoubtedly be receptive.
This is only going to get done if federal agencies get involved.

So our wish list would be consistent definitions in the Medicaid enrollment
files – What does managed care mean? When are claims itemized? When aren’t
they? – Linkages of Medicaid data files to state discharge abstracts – So most
of the states maintain hospital discharge registries. You can’t link that
information – geocoding of where Medicaid beneficiaries reside. Linkage to
pharmacy data, and linkage to Census tract and socioeconomic variables.

Priorities. Coordination of procedures for obtaining access to data and the
review process – and I am not in favor of public use, but a series of hoops and
whether you have to actually go sit in Ann Arbor or you just have to describe
how secure your computers are, whatever. There need to be various stages and
procedures.

Standardization of reporting rules. So SEER, which I work with a lot, has –
SEER-Medicare – you can’t put in any cell N less than are equal to five to
protect patient privacy, but other agencies have different rules and have to be
less than 10. Some standardization, and, I guess, harmonization would be
helpful.

Central clearinghouse.

Anything the federal government can do to help us federate state data would
be fantastic.

Making chloropleth maps available for common tasks that we do again and
again in a common-base-type system, and working with the states to fulfill
analyses of Medicaid enrollment and claims files. Those really would be an
end-user’s wish list.

Very ambitious. You can ask, right? So that’s – I’ll stop there.

MR. LOCALIO: Thank you both for comments. I certainly can relate to both of
you in your tasks.

Heather, I just want to say to you that your particular point about having
access to data and why you need access to data is something we do understand.
It is something that I have raised previously at our committee meetings and our
subcommittee meetings, although, in terms that are somewhat more colorful.

And I do want to stress a point that you made, that we have been talking
about technical issues of access. We have been talking about protecting
privacy, but the point that you brought up is there is the other issue, that it
is important for organizations, other than government, to have access to data,
because there are opinions other than the government’s opinion about data. Let
the data speak for themselves.

Unfortunately, people who work in government are not always free to say
what they want and report what they want, and the example that I have raised at
committee meetings before has to do with HHS, has to do with Medicare Part D,
has to do with an analyst named Foster, who wanted to release information about
the cost of Medicare Part D, and he was told by Mr. Scully(?) that he was going
to be fired if he did. So he did not release those data. That is well known. I
got that information from the press.

Now, I, on the other hand, if I had that information, I would have just
said, this is it, and nobody would have done anything to my job. In fact, I am
encouraged to report things like that if they are of interest.

Now, you may have a particular point of view in your organization, but
there are others that you mentioned on the opposite end of the political
spectrum that have their points of view, but I think we have to stress that it
is important in this entire discussion to know that the data need to be told.
They need to be told, and even though we have to bear in mind it is not just
researchers getting access to data. It is people getting access to data, so
that they can let the data speak, and I just want to emphasize that do not feel
that your point has not already been raised or has not been heard in this
committee.

Thank you.

DR. STEINWACHS: Probably my education, Deb, chloropleth?

DR. SCHRAG: Oh –

DR. STEINWACHS: I thought this might have some surgical, medical procedure,
and then I decided, no, it didn’t quite sound like it, because –

DR. SCHRAG: No. It is the technical term for – You have all seen them. You
see them in the newspaper all the time. They are, essentially – I wish I could
explain the derivation of the word. Unfortunately, I can’t.

Essentially, they are those maps that you look at that have, typically –
they can get very fancy, but the typically have population density. So, for
example, you might take all the zip codes in the United States and rank them by
anything. It could be number of graduates from high school. It could be number
of foreign-born persons in the Zip, the Census-level data. Right? So you
essentially – and you can get fancy, so you can look at the relationship
between the number of foreign-born persons and the incidence of stomach cancer.
Can look at the relationship between not speaking English in the home and
mortality from a particular cancer, engagement in a particular – Right?

And, sometimes, the maps are shaded, you know. So, typically, they are
shaded red to blue, and then they have dots on them. Those are chloropleth
maps, and standard data files are really helpful to make them.

DR. STEINWACHS: Helping my education.

Let me take you to another thing. You raised the idea of having data
clearinghouses, and there is a kind of function in the private sector that many
times corporations use where their claims data, pharmacy data and so on go into
a place that standardizes it, makes it into something that is analyzable. CMS,
in the past, did some things with Medicaid data, in the old days –

DR. SCHRAG: Resdac. Resdac. We work with Resdac, which is a research data
clearinghouse center. Maybe Gerry could probably talk about it, but they don’t
have everything that we want. Okay? We love Resdac. They are on our speed dial,
but – We know them by name. They know us, but there is a lot that they don’t
have. They are chronically under-funded, et cetera, et cetera.

DR. STEINWACHS: So I guess there were – and you are getting at it – sort of
two questions in here.

One is what would you see a clearinghouse doing? And the reason I was –
Sometimes, it is just you have the data there. The other is you actually make
the data more useable, and so one of the issues across the states on Medicaid
data – and you were pointing this out and so on – is that –

DR. SCHRAG: What would you do?

DR. STEINWACHS: – make it more useable for researchers, there may be a
variety of things you would do that is not necessary for the state.

So maybe getting both of you to sort of comment, what would a clearinghouse
do, and are there some examples that you think are ones that could be looked at
as exemplary?

DR. SCHRAG: So I’ll give you a specific example of the kinds of things
Resdac, which already does a good job, could do even better if they had a
broader mandate – I think they just need a broader mandate – is they could say,
If you want access to the Medicaid enrollment files, these are the hoops you
want to – you need to jump through. If you – an anonymized version of that. If
you want the actual unencrypted data file, you need to jump through a few more
hoops. If you want the enrollment files plus the claims files to figure out who
had obesity surgery, you need a few more hoops. Really laying that out in a
very clear way.

So one hoop just to get the enrollment file. That way, I can figure out how
many poor people there are in a particular area. Maybe that is all I need.

A few more hoops, a little bit more difficult, and I do think you need to
set up barriers in researchers’ ways, so that they are really clear about what
they are going to do with the data and what they need it for.

If you make it too easy for me to get it, I’ll just say, Give me
everything, when I often don’t need everything. Sometimes, I just need little
bits of it. So they can really help researchers figure out what it is that they
need at what level and setting up a progressive gradient of barriers.

And Resdac does some of it, but I think that they could do more. Gerry
probably –

MR. RILEY: (Off mike) – that Dave Gibson talked about yesterday, and they
are trying to get the data assembled into a format that is easier for
researchers to use, to pre-identify people with certain kinds of conditions –
searching through the claims forever to sort of identify people with common
conditions and things like that. So that might be one example of where this
data base will go beyond what Resdac normally does for people.

As far as the Medicaid data goes, there has been a great deal of effort put
into just trying to get very basic – you know. So I haven’t worked with
Medicaid myself, but there’s been a lot of work in our office, particularly by
Dave Bah(?) and other people to try and just get consistent measures of
enrollment and claims data and so forth, and they made a great deal of
progress, but I think the data are just starting to be used now on a much more
wide-scale basis than they have in the past. Can’t speak too much about that.

MS. BOUSHEY: I would just add one quick comment. I mean, I think, in terms
of many of the surveys that we use, using the administrative data capacity to
correct and amend many of the program participation elements would be
wonderful, and I could imagine you could do that without – one could imagine
just doing that to the public-use files, rather than – and making those
available in the same way that you do now, rather than – and sort of
eliminating that step, so that researchers didn’t have to, which then means we
never have to see the Social Security numbers or whatever.

MR. PETSKA: Can I just comment on that?

I thought some of that is done already using the tax data to edit the CPS
and the SIPP, which would go into public-use files and so on. I believe that
has already been done.

DR. STEUERLE: Just a quick comment and a question. Now, my 2-1/2 minutes is
down to 1-1/2.

Heather, I just want to say I – like Russell, I fully identify with your
statements about timeliness, particularly in the policy process. I mean, so
much research is oriented towards developing the status of things two years ago
or three years ago when Congress is constantly in the midst of making changes
that have enormous impacts, minor – not a minor example – an example being the
recent drug benefit that spent over – depending on how you do your present
value calculations, over $1 trillion, and often with almost no data input
relative to even some of the other things research.

But I guess there is one gap in our data that has always bothered me for a
long time. Deb, you are the only – really the only speaker who has referred to
it, and so I am going to put this question to you and you may not have an
answer. You can tell us afterwards, that is to do with trying to integrate in
the provider data.

Because my background is largely in areas like budget, I know the way or at
least I study a lot the way that economic systems work, and I know that part of
what goes on has to do with the cost of these systems, and part of what you are
talking about in the way some parts of the country provide benefits, some
don’t, largely relate to cost, and, sometimes, even the incomes within those
particular geographic communities.

But an important side of this is, if we don’t get at some of who is getting
these benefits, these costs, we are not very far in our – While we do develop
modest data on expenditures and what we are buying, we develop almost no data
on who is getting the money.

So, for instance, HHS doesn’t even do what I would call simple – not so
simple – quite a few people who do it – an input-output analysis, so when costs
rise by 10 percent, we know who is getting it. Is it 20 percent more to doctors
or do we have 10 percent more doctors and 20 percent more practical nurses, and
how much of it is going for administrative costs? How much is going to the
pharmaceutical –

And we don’t even have that, and you are the only one who really mentioned
the provider data, and I am just curious whether you have any suggestions for
ways of getting at some of the cost side of this equation by looking at the
provider data.

DR. SCHRAG: I think you can get at the cost side, because you have the
UPINs and you know which UPINs are doing what to whom. So I think that you can
get that.

If you want to look at things like physician income, that you need separate
survey data, but, procedurally, in terms of access to data, the biggest thing
that I think needs to be fixed and doesn’t seem to me to be that complicated to
fix is that detailed information about providers is not within the government –
the government doesn’t have that organized at all.

The AMA makes, I think, $50 million, some humongous number, and most of it
is sold to private companies, but they sell the AMA database, and people use
that AMA database, which profiles all physicians in the United States, and they
sell it again and again and again.

The government needs to do it itself. The states know who is a physician
and what their characteristics are, but the federal government doesn’t – I
don’t know. It doesn’t suck up that information from the states. Maybe there
are complicated reasons why not, but that would really help a lot, and that
would be a good place to start, and I think it would help the states, because
they have interests in these kinds of data for fraud and other things that they
really care about, and cost-related issues.

DR. SCANLON: We don’t know enough about providers. We don’t know enough
about cost, but we do know an awful lot about both. I mean, and maybe the
analysis doesn’t get presented sort of widely enough, but it is known. I mean –
what Russell mentioned, in terms of Rick Foster, the actuary’s office really
does a lot of work on the issue of cost, and we do know sort of where Medicare
costs are going. We do have the Medicare Payment Advisory Commission, which is
looking sort of at the cost reports that the providers file, and, in fact,
this, in some respects is much better data than what the private sector will
have.

You do not want to know what VHA response rate to particular items on their
surveys are, okay? As opposed to, if you are a Medicare-participating hospital
you have to turn in your clinical report.

This is the kind of thing, I think, where we need to move incredibly sort
of forward in terms of improving the comprehensiveness and the quality of the
data we have, but it is not that we have been sort of completely sort of static
here or sort of – or have ignored the problems. We really – we know a lot.

There is a large group sort of, over at GAO. There is a large group sort of
in the Office of the Actuary. There is a group sort of at MEDPAC. That are all
doing these kinds of analyses of exactly what I think you are talking about.

And the idea of using propriety data, these organizations don’t necessarily
want to share sort of the data at the level at which we would like to have to
be able to access their qualities and be able to be confident about them.

One of the things that I worry about here – and this is a democracy, and,
as Russell said, we would like to sort of have the information out there.

Information can be used for good and bad purposes – okay? – and I think one
of the things that people within federal agencies probably would think about in
terms of release of data — is this going to potentially cause harm, because it
is not very good data? Would they release data where there was only a
30-percent response rate on their survey? That would be astounding.

When I was at GAO, our response rates, the requirements were that we would
be in the 60, 70, 80 percent range before we would use a number. That is not
true of data that are coming out of private surveys. Private surveys, they are
happy to be able to say, We did a survey. Here are the results. Okay? And,
then, they can turn around and sell those because there is a market for them.

DR. SCHRAG: But the point is is that the states – I completely agree with
you that the private data – The AMA data are terrible. Absolutely. Nobody
should use it. The states have good data. It is just not accessible.

DR. SCANLON: You are right. It is not accessible, and it is potentially not
uniform. I mean, that is the other key thing about starting down the path of
saying we are going to go to the states. We’ve got the 50 states plus the
District of Columbia. If we are talking about Medicaid, we’ve also got five
territories that run programs. Try and assemble a consistent database from
those. It is incredibly challenging.

MS. TUREK: That is easy compared to TANIF, or, in some states, the data is
collected at the county level.

MS. BOUSHEY: Yes, or childcare subsidies.

MS. TUREK: What?

MS. BOUSHEY: Childcare issues, county level.

MS. TUREK: Thank you all very much. I imagine after our last session we can
continue talking about users’ needs forever.

Anyway, for now, everybody go and have a good lunch.

(Whereupon, the workshop recessed for lunch.)

A F T E R N O O N S E S S I O N (1:18 P.M.)

Agenda Item: A Broader Perspective on the Role of
Linkages

Fritz Scheuren, Vice President for Statistics, NORC
and 2005 President of ASA

DR. SCHEUREN: I am going to talk about the past because I am an advocate of
Deming, and Deming says you should only talk about something you know about.
Well, I don’t know about the future, and I probably will know it when I see it,
but then it’ll be the past. So I will talk a little bit about the past.

This is a very interesting subject, record linkage. Deeply connected to
health, quite expanded as these days were on other topics. Very deeply
important subject, and so let’s see how people thought about it 40, 50, 60
years ago or more. I call it the Once and Future King. So you may have seen
that line before. I don’t know who – Somebody may have used that.

The Book of Life. This is a concept that we are putting together a book of
people’s lives. That is what we do with linkage. It could be contemporaneously
this way, but it also could be this way, and all of those variations in
dimensions have occurred in these meetings. Great idea.

Started out – Dunne(?), I think is the one who used this phrase. Started
out right around the ‘30s and ‘40s that this phrase began to appear,
and just as we had the ability to do the kind of large-scale record linkage
that we all now do.

I am going to put this one up. This is the Chalk River Nuclear Power Plant
in Canada. We were talking about Gil Beebe(?) a little while ago, with Nancy.
Gil was an advocate of record linkage, because he’s an epidemiologist. This is
really where things started, and Howard Newcombe, who did an awful lot of the
fundamental work on linkage worked at Chalk River and it was all
epidemiological.

Social Security did a massive amount of linkages, epidemiological linkages,
to look for various carcinogens in various industrial processes. A massive
amount of work done. Great stuff. Not talked about anymore. Great stuff. Still
can be done, I believe.

Most of our problems of that kind have been seen and acted on. We don’t
have the kind of problems they have in China when they do a cancer map in
China. They find that most of the cancer problems in China have to do with
differences in the way food is prepared in China. So we don’t have that
problem. We all eat McDonald’s, which is to say we all have a higher level, but
–

The theory showed up with Ivan Fellegi. Howard and Ivan are Canadians. It
is not an accident that I am going to talk about Canada. An awful lot of the
best work on record linkage has been done at Statistics Canada. Fantastic
tradition. Ivan is still Chief Statistician at Statistics Canada. Great man.

And we ran a conference in the names of these two people here in Washington
in 1997 on linkage, an international conference.

We had done an earlier conference, which I think is the – one of the things
that Jean associates with me. I was organizing both these conferences, was in
‘85 – which is more a local conference on record linkage.

Both of those are at the FCMS website, but I think they are huge PDS. You
have to really be serious if you want them, but there are people here – Listen,
at this time of the day, on the second day, I figured there’s six people here.
I was wrong. Now, don’t all leave. Wait for him.

This wonderful piece of work by Marks – that is the same Marks. Carol
Krotki and Bill Seltzer. Bill Seltzer is still with us. He was the Chief
Statistician at the UN for a long time. Now, he does a lot of important work in
other areas, including human rights.

And here is a tremendous book, Bishop, Feinberg and Holland. These are all
contingency-table books. They are really valuable, and they are worth knowing
about in order to understand the error patterns in the data search you are
using. If you do not understand the error patterns – non-sampling error
patterns – you really haven’t grasped the whole thing.

When you are doing linkage, you actually really can improve considerably
the quality of your data in many, many ways, and that has been talked about
quite well here, but you there is no free lunch, not that there are many
economists in the room. Gene took his badge and put it away today. He is not an
economist today, right, Gene?

One of the things you can do with these systems is you have three or more,
you can do multiple systems estimation. If you have two, you are doing dual
systems estimation. This is the traditional pattern that has been used around
the world in evaluating censuses. It is very modeled appended(?), and the
bureau has moved carefully away from that towards multiple systems, okay? Very
carefully. I wish they went a little bit faster, but, anyway, very important
work.

What about content – A lot of work has been talked about here about that. I
am going to give some names to you. This is Mitsuo Ono, who used to be the head
of the Income Branch at the Census Bureau. He did some very important early
work on matching income-tax returns to the CPS, okay. I think it was the 1970
CPS. Yes. I think it was – Maybe it was ‘72. I thought it was the ‘70
census.

Dorothy Rice. If you don’t know Dorothy Rice, you better leave right now.
Just leave right now. Major hero of mine. She used to be at Social Security.
That is when I used to work for her, and then she was the head of the National
Center of Health Statistics. She is a tremendous individual. A great force.

I interviewed her a year ago. The article appears in the September issue of
AMSTAT(?) a year ago. Not this – well, it is two years ago now. This is
September. Two years ago. Worth reading. I interviewed her, but listen to what
she said. Don’t pay attention to what I said.

Joe Steinberg is the person who started the linkages at Social Security
that we have been talking about the last few days, including getting the Social
Security number question on the October 1962 Current Population Survey. That is
when it was first put on the survey. Of course, it is not on there anymore. It
was taken off. He did a great deal of work, very good work. I came on after him
and tried to finish what he did. He went on to be Assistant Commissioner at the
BLS.

And Ben Bridges was my boss at Social Security. I needed to put his name on
here because he used to invite me to his house a lot. So good guy. Really good
guy. Well, he’s a really good guy, and he had the patience to read everything I
wrote and fix it. Since then, it has been bad.

Then we ended up producing a whole series of products called the
Interagency Data Linkage Series. Very dated now, but full of mathematical and
statistical ideas that are still valuable and have not been recaptured anywhere
else. I am sort of proud of that, pleased with it.

There are some other things that happened as a result of that work.

I am going a little too slowly. I can tell that from the three people in
the back who have woken up now and said, Where is the next speaker.

One of the key champions of augmenting survey data with administrative
records is Gene Steuerle who is here. Okay?

Another one is Howard – Howard Iams. I hope you listened to what he said
yesterday. Dead on. Dead on. Dead on.

And Julia Lane. I happened to come in – I know Julia from other worlds,
when she tried to recapture the essence of the Continuous Work History Sample
state by state. Amazing individual. She gave a great talk yesterday.

Two more things. Gene Rogot, whom you don’t know. He was at the National –

PARTICIPANT: (Off mike).

DR. SCHEUREN: Pardon me?

PARTICIPANT: (Off mike).

DR. SCHEUREN: Yes, he convinced people, and CHS and the Census Bureau to
match the CPS’s to the National Death Index. I asked yesterday if that was
continuing. It is continuing. Okay? That is a really interesting process, and
that has been published. You published it, didn’t you? It was published. A
Million Deaths. It was a publication a few years ago.

It is a marvelous piece of work, because you can look at social-economic
differentials in mortality with that, which is my interest in mortality, by the
way, social-economic differentials. I am not going to go down that road today,
because that is just too much fun.

And Joe Peckman, a lot of people know Joe here, at least Gene does. The
idea of – matching data when you can’t do it exactly is a valuable idea. It is
a heuristic that I urge you to use with great caution. I have written
considerably – I know. I have written considerably on its weaknesses. Okay? It
is not always weak, but if you are desperate, do it. If you are not desperate,
wait for the real thing.

Let’s talk about optimizing of systems. That is – What we have been doing
is taking the existing systems and making them better, but what if we were
optimizing systems, what would we do?

Well, this is not a real long list. What we want to do is we want to
prevent the survey errors from occurring – all right? – to begin with, if we
can. Very hard in a day when non-response – item and unit non-response are so
high. We want to build in a detection system so we know there is an error and
we need to fix them – repair them, and, of course, one of the best ways to do
those – the last step is to replace them with data from a better source. So the
linkage is an enormously important quality-improvement step.

Now, it was said by somebody this morning the quality of the thing you link
to may not be perfect, okay? It certainly wouldn’t be, anything I have ever
done.

One of the things that’s going on is that there is a tradeoff between
response variance and response bias. Administrative records are typically
biased. They are not measuring the right thing, and if you think it is
something that it isn’t, you’ve got a bias, even if it was perfect – okay? –
but they do get rid of a lot of the response variance, which is very
characteristic of surveys.

If you have ever done any linkage yourselves and you have compared
essentially equivalent – never exactly – equivalent concept from an
administrative source or an operating source in a more general world, which is
the one I am in now, with a survey, you see this enormous variation in the
survey results. Rounding errors. All kinds of things going on in the data.
Maybe the right signal is in there, but an awful lot of noise.

Playing with Matches is a book. This is a plug. Three people in the back,
the one who has fallen asleep, you know, you should wake up now, because you
have to buy this book when it comes out. It’ll come out next year. Okay?

It is about data quality. It talks about all the traditional ways we have
looked at data quality. Everyone in this room who has done – handled data has
used these techniques in various ways, and it talks about linkage, all of the
aspects of linkage, most of which have not been talked about today, but,
fundamentally, linkage, in my opinion, is really to replace one data source
with a better one. Okay?

If you want to study error patterns, that is good. That was done a lot. I
don’t think that got us very far, frankly. What got us far was replacing bad
data with good data or better data. There is no good data.

Okay. A couple of more slides. I guess I got four. Three. I’ll make it
three.

Privacy and confidentiality. One of the big problems if you are in an
administrative agency or a statistical agency is you really don’t understand
the language the same way. There is a conflict of principles, really, between
the two.

If you are at the Census Bureau or NCHS, you are a part of a culture of
confidentiality. If you are at the IRS or Social Security – Social Security is
a bit on both sides – you have the culture of privacy. You focus on the privacy
of the person. You are – that person’s data is sacred to you.

Those two values do have an intersection, but it is sometimes very hard to
find, depending on the setting you are in.

I think more work like this meeting, more joint work, designed work for
joint goals can help deal with some of this, but it has existed for all the
decades that I know about and have heard about in all of these different
processes, and I don’t think it is going to go away any time soon.

I want to make a comment about – I am using an industry coding example. The
statistical agencies say to the administrative agencies, We can’t give you back
the data after we clean it, okay? Well, because we would violate the trust we
made, and so I think, though the statistical agencies need to look at the point
of intervention where they get the data, and look at whether they could get the
data at a different point and thereby aid the administrative agency – and the
industry coding example, which I am not going to cover, is a perfect example of
a great deal of waste with Census coding things and the IRS coding things and
the BLS coding things and the states coding things – okay? – that are none of
them done well. Okay? All of them might be done better, if we were to fix the
system.

Legal and bureaucratic. Lot of discussion about law and practice links. You
are all discouraged by it. Get a lawyer in the room. I mean, get some lawyers
here. Really – need to do this again, and experiment, and I believe in the need
to do continuous measurement of what is going on.

That is one of the reasons I really like this group. I didn’t realize that
you don’t meet as much as you might, and, of course, I hope – I mean, there’s
great ideas here these last two days. I hope you are stealing each other’s
practice, best practice. Don’t steal each other’s worst practice. Steal each
other’s best practice. I don’t think I have to tell you that, but, sometimes,
you are not necessarily sure you know where it is.

One of the things that NCHS does, which I absolutely think is fantastic, is
they have an IRB. All right? Every agency here who does linkage should have an
IRB. If they don’t, there is a real issue there. Okay? I really, really think
that is what we should do, and I am not going to name names, because I know
some of the agencies here who don’t, but they should. It is very important that
you be held accountable by your peers, okay. Should be held accountable by
other stakeholders, too, but by your peers, because your peers can help you fix
some problems.

I have been subjected to IRBs in lots of settings, in private settings.
Somebody was talking about using the National Survey of America’s Families this
morning, which is a survey I worked on for a couple of years. Doesn’t do any
linkage, of course. One of the problems with it.

Let’s talk about learning linkages. Let’s think about – it as a learning
system. We have been doing it forever, but we haven’t thought about it as a
learning system. We could have. Just didn’t.

We need to continue these conferences. Fundamentally, keep talking. Keep
listening. Collect and publish a summary. I don’t mean 100 pages. Forget that.
Two pages. Five key points. Okay? Four contacts with people. Okay? Really.

If you got into somebody’s remarks the last two days, talk to them on the
phone. Get going, okay? Only do the things you are interested in. Don’t do
anything else. Doesn’t matter. There’s enough interest in this room so a lot of
good things will happen.

I have said that.

I want to see diagnostics developed for linkage. I am a big fan of
diagnostics. I have learned a lot about diagnostics from regression. Many of
you who are economists do regression, log, linear, logistic regression, if you
are in epidemiology world or standard regression if you are in some of the
other worlds in this room.

Build diagnostics. We need to do this. This is very important. Not hard.
Fun, actually, and then you can get rid of some of the other errors in here and
get some of the misleading things out.

And here is one that you won’t agree with. I put match in first, because I
wanted you to find people at equivalent levels and swap staff. Two, three
months. Right? As long as you can stand it in another agency. It is
fundamental, really fundamental. We are not learning fast enough. This is a
shame. All these smart people and we are not – We are all living in our various
stovepipes – okay? – smoking something. No. No. No. Wrong generation. You never
did that, did you?

Okay. What’s happened here? Somebody has taken over here my computer. They
say, End of show, here. That is what they told me, End of show. I am almost
done.

Thanks for the memories. I have gone back to something I used to do, and
some of it I still do, and best of fun on our road. I am saying – including
myself – our road ahead.

Thank you.

DR. STEINWACHS: Thank you very much.

DR. SCHEUREN: You are welcome. Sorry for too much past.

DR. STEINWACHS: So, Mike, did Fritz set the stage for you?

Agenda Item: Michael Davern, Assistant Professor,
University of Minnesota

DR. DAVERN: He certainly did. Yes, it is very hard to follow Fritz, of
course. Now, everybody can really go back to sleep back there. Right.

He had most of what I had to say here. I’ll give some examples, I suppose,
or some of what I had to say. I don’t really know the names or the history
quite as well. I almost left the room, unfortunately, when he said, If you
don’t know this person, why don’t you leave the room?

DR. SCHEUREN: Dorothy Rice?

DR. DAVERN: Yes, I know –

DR. SCHEUREN: You know Dorothy.

DR. DAVERN: So, first of all, I would like to thank Joan for inviting me
here to be a part of this. I think it is really important, and I am looking
forward to giving you my thoughts after a couple of days here or at the end of
it, and I get to be the last speaker, so everybody is eagerly anticipating the
last slide. So I have it duly marked, so you’ll know when it is coming.

Basically, administrative data and survey data are really kind of collected
for different purposes, right? We all know this. Survey data are collected for
research, for the most part, and research file, administrative data are to
administer our program, and we want to put these two things that are sort of at
odds together to do really good health research, health-outcomes research.

So I am going to do some ramblings and musings. I don’t really know much
about administrative data, other than what I have learned from people who are
in this room. I know Dave Ball was here yesterday. He taught me an awful lot
about administrative data from – on the Medicaid side, and so if I get
something wrong, feel free to correct me, but I am just going to give you my
impressions are of what is going on.

So I am going to stick with what I know, which is survey data, and I know
survey data fairly well and been working with it for a long time, and, then, I
am going to see how administrative data is sort of like survey data, in some
ways, and see if that can be a useful exercise, and talk about much of what –
People have already brought up a lot of the issues I am going to talk about.

But here is what I have: I have several concerns with survey data for
health research, and then I am wondering how administrative data compare on
these issues, and then I have issues in merging the two sets of data or
matching the two sets of data, and then work left to do to fulfill the great
potential, I think, of these merged and matched data sets.

I won’t spend much time on the data stewardship, privacy, confidentiality
stuff. It has been covered well, I think quite well, elsewhere, and it is
extremely important.

So I am going to start at the end just in case I don’t make it to the end.
This is what I want people to understand from at least my point of view.

There is great potential to health research to be done with these linked
survey and administrative data files.

Survey micro data are in the public domain, and that is really important.
Heather talked about it quite a bit here today, and the importance of having
that out in the public domain for policy research.

There is also the importance of having it in the public domain for the
strengths, and especially the limitations of this data are extremely well
known. Sometimes that is thought is a weakness of the survey data, that we know
a lot about it and we know a lot about its limitations.

We don’t, unfortunately, have that same kind of information about the
administrative data, because, obviously, it is on – it is not in the public
domain, and researchers can’t do research on its quality.

So because these data are not in the public domain, it is really imperative
that the limitations be thoroughly investigated by the people who are entrusted
with these data, more so – You know, a lot of it is going on, but if we are
going to put these data sets together, we really need to have this information.

And there needs to be documentation and research on these linked files, in
other words, metadata – you know, data – information on the actual data
elements themselves, how they were collected, how they got in the data file,
where they came from, and a lot of information on the process of how that
information was produced. That needs to be put out into the public domain, so
if I am reviewing an article for a journal and someone is using this
linked-data file, I know how that variable was produced.

If I don’t know how that variable was produced in the administrative data
file, it makes it hard for me to review an article and know if that correlation
or regression coefficient they found is actual or just something that was
created as a part of the administrative process.

Certainly, NCHS, Census, NCI, Social Security, AHRQ, everybody here has
these agreements in place and ideological people to start producing this work,
both the documentation and the research on the data itself. I think that is
really key.

We need to have that research done. We need to get all this information out
into the public domain, and these are the people who can do the work at the
moment, because it has taken a couple of years for our – the project I have
with Census to really hit the ground running, just because of all the
agreements that have to be in place, and it is impossible for researchers on
the outside to do this kind of work.

Survey data have extremely well-known limitations. Okay? Just to give you
some of the highlights – or low lights, as you may see – survey data concerns
that we have are sample frame coverage error. We have talked about that quite a
bit, and Fritz brought it back up again. We have sampling error and variance
estimation. You have non-response error, both item, non-response and unit
non-response. It is becoming worse, certainly, with both of those.

We have measurement error, things like collecting data from mixed modes.
There’s a lot of people who study whether or not if a piece of information was
collected through a self-administered questionnaire. It is different than if it
was collected through an interviewer.

They have data processing amputation editing, and there is always need for
better documentation of metadata on the survey data side of things. There is no
doubt about that.

And all these things, I think, are extremely well known about surveys.
Survey data are dirty, messy and not for the timid, and I highly recommend that
– you know.

So when I am talking about administrative data, as a survey researcher, I
know that survey data are messy. I have made a living off of pointing out to
people that the survey data are messy. It is something I publish quite a bit
on. So, in general, I think that knowing the survey data are messy is a good
thing.

And so how are the administrative data unlike or like survey data with
respect to these main issues that we are concerned with survey data?

And there is, of course, a great variety of administrative data, and I am
kind of throwing it all in a bucket here at the moment.

Sample frame and frame coverage, not really a problem, obviously. Survey
data, you know, it covers the entire enrolled population.

Certainly need survey data, as pointed out, to know about the unenrolled
and potentially enrollable populations and take-up rates and all that kind of
stuff. So the survey data provides you with that, but there isn’t really much
of a problem from an administrative data point of view as far as a frame of the
population being covered.

It is important to note, though, that I did work – I was working with the
Veterans Administration in Minneapolis on a study that they were doing of
Post-Traumatic Stress Disorder, and I was doing a non-response analysis on
their survey, where they had sent out a questionnaire to people about PTSD, and
what we found was that we had a response bias when we looked at the
administrative data, that it seemed that people who didn’t have PTSD were much
less likely to respond. So we thought this was something quite interesting, and
then it turned out, when we dove into it, that it was the quality of the
contact information that was really producing this.

People who have PTSD received cash payments for having PTSD, and, as a
result, the quality of their contact information was very, very good. We had
good addresses. We had good phone numbers. We didn’t on the others, and that
was what explained the difference. There was very little difference after we
controlled for that factor.

So when you are using these things as administrative data survey sampling
frames in a way to mix – as another way to mix the two, we need to be careful
of those kinds of things like contact information.

Sampling error, of course, is not much of a problem. Could be if you are
drawing samples from the administrative records to use for research, not a big
deal.

Non-response error is a bigger deal or missing data. Certainly, I know the
Medicaid data the best that I have been working with. Item non-response on
those is a major issue, largely because, I think, item non-response here isn’t
the same as it is in surveys. The mechanism that produces it in surveys is
someone doesn’t give you a piece of information or they don’t know. So they
refuse or they don’t know, you know that.

I think it is maybe more systematically missing in administrative data if
it is missing for some reason, and that is an important thing to keep in mind
when you are using these data for research purposes. It is probably more likely
to be not missing at random versus missing at random in kind of the statistical
ease of things.

Age, program codes, race and ethnicity, we have been back and forth with –
The CMS people have been wonderfully open about problems with their data with
us, and, as a result of that collaboration, we have learned a lot about the
data, but it is important to know that this is – it seems to largely be
systematically missing. You know, some states are missing race and ethnicity
information, and others are – have very good race and ethnicity information,
or, at least, it is filled – the variable values are filled in.

So some of this data is missing systematically, TNF flagged by county, and
we found out – we wanted to use a TNF flag on the MSIS, the Medicated
Statistical Information System, and found out that it wasn’t really all that
good, because it was systematically missing. Race, ethnicity by state were
systematically missing in the MSIS, and some states had much more missing data
than others.

Identifying data can also – ID data can also be missing systematically,
which is really important for doing linking, obviously, and we need to really
do a good job and figure out where these data are missing in systematic ways,
and it can be a large source of sample loss for the merged data if ID data are
missing systematically.

So administrative data have important information for health research that
is missing. I mean, that is the bottom line of the missing error, and I think
it tends to be missing more systematically, perhaps, in this survey, and so
maybe some of the techniques, like – imputation or things are not as possible
with the administrative data.

There are certainly measurement issues with administrative data. Certainly,
administrative data are, as we heard from Social Security yesterday, are the
standard for knowing whether someone is enrolled in a program and how much
someone received in benefits. There is no doubt about that. That is right on.

However, there’s other administrative data that is desired for research
that is on these files that may not be as linked to the program as – and may
not be as well measured, and there’s probably a lot of error associated with
these things, as Fritz brought up and other people have talked about.

Administrative data can be collected through many modes during more than
one wave of interviewing with several instruments used, and it is all kind of
mushed back together, okay?

You have interview – and the survey researchers will tell you all this
stuff matters – okay – that they do a lot of research into interviewer effects
and to self-administered questionnaires versus non-administered questionnaires,
and so you have all this stuff going on.

You have people completely filling out, where the interviewer actually
fills it out completely for the enrollee and then submits it and just has them
sign it, and, certainly, I have done that for tax information for people who
don’t speak English. I helped out quite a bit in doing that over the years, and
so I fill out the form completely for them and just have them sign it and send
it in, after they have given me whatever W2s that they have.

So you have all these kinds of things that are going on, and so it is
important to try to track that, as best we can, to try to figure out what the
source of the information is and create this metadata and do the analysis on
this kinds of stuff.

And interviewers have a wide variety of training and skills. For example,
you can have a tax – If you have an accountant fill out your taxes or you do it
yourself or other kinds of things, there may be quality-of-data issues involved
with that, and so it is important to be able to – You know, when you are
beginning to link this stuff, it is important to know about that when you are
trying to use these data for research purposes.

Medicaid enrollment data can be drawn from a wide variety of sources,
including county level, state level. You know, it is coming from all over the
place, and you have no idea how that variable got to where it is when it is on
the MSIS. It has gone through a lot of hands before it is into that MSIS, and
it is really important for people to understand that that is an incredibly
different situation than when the Census Bureau goes out and collects a survey
with an English version of the instrument and a Spanish version of the
instrument and so those are things that could be going on that are being drawn
from all over.

Administrative data forms. So the forms you actually fill out, are
generally not as user-friendly. I am always frustrated by them. That is one
thing that I think administrative people could learn – data people could learn
from survey people is how to get a form that is actually user-friendly and has
layout in an easy way for people to see the race and ethnicity information, and
they can circle more than one or they can fill in the box for more than one, so
that it’ll be comparable on the survey data and things of that nature.

So research is really needed into the mode effects and longitudinal panel
conditioning. As data is collected over time and things change, instrumentation
effects and all that kind of stuff can certainly creep into the data and it is
something that I think we really need to go in and take a look at. Survey
research has a long history of this kind of work.

I know administrative data has done it, but the thing about survey data is
you have these wonderful journals, outlets that get out to the public. You have
Public Opinion Quarterly, Journal of Official Statistics, and this work gets
out there. You have all these – and, certainly, America’s Fiscal(?)
Association, JSM meetings you have all these – you know, historical record of
the problems with these surveys that have been created, and it would be nice to
see – Certainly, some of that work is done on the administrative side, but
seeing this kind of work, looking at these kinds of questions, I don’t see very
often in the administrative data, and it would be interesting to see, looking
at like interviewer and mode effects and if there’s reasons to suspect that the
data may not be consistent.

Also, it is important to remember that people have different motivations
for filling out administrative data than survey data. Okay? I think that is
really key. You might want to have one income for your tax record. You might
report another one to your CPS interviewer. You might report another one to the
Medicaid agency, so you can get enrolled in Medicaid. There’s a variety of
these things. You can think of a creative caseworker in Medicaid being just
probably as good as a tax accountant at hiding income and knowing how to put
the family formation together, and so these are things we should be thinking
about as far as motivations of people for filling out these data when we begin
to look and see and cross classify by this stuff.

Also, if there is data that is not accepted in some administrative data
systems, so do data-entry folks just enter it, pass by that screen, even though
they didn’t ask or collect that information? So you are always curious about
that. So it is always good to check out if you can source out who put that
piece of data and how it got into that system. It is really a key thing to be
able to do research on.

So that is it, and, also, data editing and imputation. This is something
that is absolutely essential, I think. When we are putting together these
linked-data files, we need to have incredible metadata on them for researchers
to be able to use them well. There is very little documentation in the public
domain regarding the collection, editing, imputation procedures of
administrative data and enrollment data relative to survey data, and I think
that that really needs to – If we are going to link these things up and create
these files, the first thing we need to do, for researchers to use them
effectively, is to write the documentation. I know it is not anybody’s idea of
fun.

What I am doing, as a research project with the Census Bureau, where we
want to get to the answer, but we have put together a huge team of 20-30
researchers who have put a lot of effort into a project over two years, and all
that will sort of be left by the wayside, all the knowledge we collected along
the way, and we’ll just get a research paper out of it that gives the technical
results of what we are looking at, but we won’t have created the metadata, I
think, at the end of it. This is what is typically left. I mean, as
researchers, we just want to get to the results, and we want to pass along –
and not do this really tough part of documenting and writing this stuff up.

So putting these linked-data files together also means we have to create
the metadata to go with them. I think that is essential and needs to be done,
and all this kind of research needs to be taken – take place so we can do that.

So, basically, how does administrative data compare to survey data for
research purposes? Survey data, micro data and research into critical sources
of error are all in the public domain. Survey data are very strong, because
there are so many known problems. Okay. I have already talked about that, but I
do think it is an extremely important point that is often missed by
researchers, and similar research needs to be done on administrative data.

Certainly, the quality of administrative data will vary greatly from
centralized data collections or more centralized, like SSA, IRS or Medicare to
Medicaid or state-based programs, and so, certainly, I have been throwing them
all into one pile, but I expect that there will be great variation with respect
to some of those things.

The issues with the linked-data files that we have certainly been dealing
with over the last two years in the project that Ron described yesterday with
Census is there’s universe issues and measurement error on both sides of – the
administrative data side and the survey data side, and it is essential to
understand the differences and concordance between these data sources.

The universe issues, when there is missing linking information, that is not
good, right? So we need to really figure out why it is missing and who is
missing and how it could impact our analysis.

Do we have differential sample loss – missing ID – because someone refuses
to give their Social Security number on the CPS, for example, or could not – we
couldn’t find their Social Security number, so it couldn’t be validated? So we
have differential sample loss. There’s two real sources there. Ron showed it
was about 27 percent of the cases, I recall, total that couldn’t be linked.

Administrative data has missing linking key information, and it is
differential. He showed – I’ll show you here in a second that it was
differential, and it was systematic, and so there needs to be – when we build a
common universe, we have to do it carefully and figure out what was going on.

As you can see, here is a systematic. The red and black aren’t good for ID
information.

This is one of those maps, I believe, that Deb was talking about earlier,
right?

DR. SCHRAG: Yes.

DR. DAVERN: So here we are, and I actually didn’t know that was the name of
them either, but I have one in my presentation.

As you see, California and Montana, if you were doing an analysis, you
would have systematically missing data that could impact your analysis,
depending on how you are using those data. So it is key that you find out who
is missing and why when you are working with these linked-data files.

Developing these linked universes, there’s all kinds of reasons. This is a
slide Ron had, too. There’s not a valid record. They refused to have their data
linked, and then you have the big group of – Most people are in the big group
in both the MSIS universe and the CPS sampling-frame universe, but there is
also not a valid record. There’s people in group quarters. There’s people who
have died before the CPS interviewer gets to them, but they were enrolled in
Medicaid that year, and those kinds of things. Enrolled in more than one state
is another issue. Also, on the CPS side, you have births, people who were born
not in the calendar year that the data were collected for, but are included in
the CPS interview.

So you have measurement error here going on. There’s conceptual
differences. For example, a person can be on Medicaid, but not receiving full
benefits, so is the person really insured? That is an important question we
need to ask, and, in some cases, we determine that, yes, they kind of look like
they are getting a full range of benefits, some of these people who are partial
benefits. Others, aren’t. So you do have conceptual differences when you are
linking these files to think about. So they are on the MSIS, but do they
actually have health insurance as we think of it?

You have misreporting in surveys – person is on Medicaid, but reports some
other type of coverage or reports that they are uninsured, and, certainly, Ron
showed that yesterday.

You have misclassification of administrative data. Race data are often
missing from the Medicaid file, and are important for, of course, disparities
research, and when they are there, they may not be collected systematically in
every state the same way.

You also have systematically missing variables, as I have talked about.

So the potential for the merged data is great. We have talked about a lot
of these things. You have improving the accuracy of survey data using
enrollment data. You can improve the accuracy of sample frames.

One thing we haven’t talked about is the Census Master Address Files,
greatly improved by the delivery-sequence file, and, then, their relationship
with the U.S. Postal Service, and so those are great ways to improve Census’
sampling frames or anybody’s sampling frame is by looking to the administrative
data.

Using merged data to create small area estimates was covered. Incredible
potential, I think, there.

Improved administrative data race and ethnicity information needs to be
done, especially for health disparities work.

There is great benefit to using information on imputation models and
editing from these linked-data sets that both the administrative data site and
the survey data site can use. So survey data can do better imputation and so
can the administrative data. Even if they can’t get the merged or linked data
back, they can learn a lot about their own data and the patterns of missing
this data.

So this stuff will greatly improve health policy simulation and health
research, and engage our errors. Don’t be afraid of them. Engage. Go out there.
Do the research and document them and try to get the best stuff out there.

And just to wrap this all up, there’s a couple of – We talked a lot about
problems. Other people have brought these up with recency and those kinds of
things.

Certainly, these agreements all do their duty, and they restrict access,
and, as a result, we are dealing with old data, in a lot of respects. So that
is one of the limitations of this stuff. Hopefully, that will pick up and be a
little bit more timely.

Data are not in the public domain, and ability to conduct research into
quality of administrative data for research purposes is limited for people like
me, who like to do data-quality analysis.

So it is imperative that the agencies entrusted with those data really do a
good job for looking at data quality for research purposes not for
administrative data purposes, which are two different things.

So – and be careful, of course, about reaching conclusions based on
asymmetrical verification. Jill Colon(?) from AHRQ had this at the session we
were in, the joint statistical meetings about a month ago.

The example here is that we compare Medicaid enrollees who are linked in
the CPS to the MSIS. We know that they have Medicaid, but, in the CPS, they
don’t report that they have Medicaid or they report that they are uninsured.

We had 15 percent of the CPS data reporting that they were uninsured –
which is a problem. They actually had health-insurance coverage, according to
the MSIS in the past year.

Well, if you multiply that times the 40 million, simply, who are on MSIS,
you come up with six million, and you say, Well, obviously, there’s not 46
million uninsured in the United States. There’s 40 million. That is a dangerous
conclusion for a couple of reasons.

What is going on here is it doesn’t allow us to verify – the linked-data
files don’t allow us to verify if uninsured people report that they have
coverage. So we have only verified – we are only able to verify one piece of
the puzzle. We are not able to verify the others.

So we know if someone on Medicaid said they were uninsured. We don’t know
if someone who is insured says that they were – or someone who is uninsured
said they were insured, and I think it is likely to happen, given that there’s
10 questions that ask what type of health-insurance coverage you have, you
eventually give in and say, Oh, sure, and especially by the time they give you
the last one, which says, Are you sure that you are uninsured? So there’s
likely some of that going on on the other side.

So it is important not to jump to simple conclusions when you are doing
these analysis and when you are working with these linked-data files, and
taking a look at the sample loss is really going to be key.

So I am going to start back to where I finished, which is I think the
strength of the survey data is that it is in the public domain. There’s lots of
researchers taking a look at it for research purposes. We really know its
limitations.

This is not true for the administrative data. So it is really imperative
that people who are entrusted with these linked-data files put together both
the documentation and the research on them, so that researchers can reasonably
understand the data that other researchers are using to inform the debate, as
well as the research that are used – you know, the data that they are using.

There are certainly all kinds of standards out there, data documentation
initiatives, one that I am familiar with, DDI standards for survey data.
Perhaps something similar for administrative data would be a good thing.
Research into sample losses is really key on these linked-data files, and
understanding the measure error, so –

And that is it.

DR. STEINWACHS: Thank you very, very much.

Agenda Item: Wrap-Up

DR. STEINWACHS: We have time, now, both for questions and comments to the
speakers as well as to go into a broader discussion.

I wanted to invite people who are sitting back in the audience, we have
seats up here, and I would very much welcome you to join, and that way, also,
in making comments, you have a speaker in front of you, instead of having to
look for a wandering microphone. So please come up and join us.

Comments and questions to the speakers?

Is anyone ready to take the quiz to see if they know all of Friz’s friends?

MR. DENBALY: I have a question for Mike.

I am just checking to see if I understood you correctly. If administrative
data are used to guide the agency to run their program, why should the
evaluation of the data be for research purposes or did I misunderstand you?

DR. DAVERN: I think that the administrative data have been evaluated quite
well for their programmatic purposes, and so the programmatic data, I think, is
very good and solid.

There’s all kinds of other information that gets carried on these files,
like – that gets collected at enrollment time or other things that I think
should be evaluated, and those are the things researchers are really interested
in a lot of times.

The health-disparities researchers want to know about the race and
ethnicity data, but the people who administer Medicaid don’t necessarily care
all that much about that data.

As a matter of fact, when you enroll, some states, on their forms –
enrollment form say, This is a completely optional question. Like I think New
York has one, and so they have incredibly missing data on their Medicaid files
on race, ethnicity.

Other states just have a blank that says, Race, fill it in, and how that
gets recoded into the MSIS, I have no idea.

So those kinds of things are, as far as the quality of – So those are the
things I am talking about for research purposes.

The administrative data are very good at administering programs. There is
no doubt about it. They have been properly evaluated, but I do think it is
something different to think about from a research perspective – from a health
research perspective. They need to be evaluated for that purpose as well.

MS. GENSER: Hi. Jenny Genser, again.

I wanted to give a case study on some admin data that I work very much
with, and that is the Food Stamp Quality Control Data, which I have been
working with for about 15 years, and what I have found with the Food Stamp
Quality Control Data, what it is is administrative data. It is a sample of Food
Stamp recipients that we use to measure payment error, and we found
consistently that the data that are required to determine whether a person is
receiving the correct benefit is generally quite accurate, but other variables
that we have collected – such as race, ethnicity, citizenship status – may not
necessarily be as adequate, because it doesn’t effect the QC error
determination.

And our office has worked a lot with the program to make sure that these
data are higher quality, because we use it so much for our analysis for a cost
estimation for researchers, and, in fact, it is data that is on the public
domain with documentation.

Example might be with citizenship status is a few years ago we were finding
a lot of people who were – the race was coded as a Native American. They lived
in Oklahoma, and their citizenship was naturalized citizen, which we knew that
it didn’t make sense.

So that is just a case example you might want to just – if you are
interested in looking at quality of admin data that the Food and Nutrition
Service has done a lot of work with the Food Stamp Quality Control Data.

Now, in terms of these overall Food Stamp data, that is run and
administered by states and counties. So we don’t know that quality.

DR. BREEN: Just to build on that, when I was talking with the Social
Security Administration Quality Control person a number of years ago about the
possibility of using that data and matching it with the SEER registry data, she
was – We were discussing – I thought it would be good to use IRS data to get
the tax information for information on income and then the Social Security
information to get information on earnings and that that would give us a pretty
decent picture of people’s economic well being, which we had nothing of on the
SEER data.

So we were discussing that and she had been there a long time and mentioned
some of the issues, but one of the things she said was that before Watergate,
when their data were routinely analyzed, especially by people at the Bureau of
Economic Analysis, she said, Our data was in much better shape, because the
researchers would analyze the data and they would come back to us and they
would say, Well, you’ve got a problem here because this data or that data is
not very accurate.

So, in fact, the administrative data – and I want to just put this on
record for the committee in terms of our recommendation and cost to the public
of not doing this – the administrative data gets better by researchers outside
the agency using it, and it gets better even for administrative purposes.

And I don’t know if Deb wants to mention this, because I know she has been
using, and Gerry, too, the SEER Medicare data, which has improved over the
years as a result of researchers working on this data. It is better for
research. It is better for administration.

MR. LOCALIO: Michael, I commend you for your comments about metadata. That
is not often discussed.

I just want to let people know that it is not just an issue for the type of
data we have been talking about here, whether it is administrative data from
various agencies – It is a terrible problem with – and clinical data that you
get on people’s health.

Different organizations collect the same data in very different ways, and
it means different things, and if you think you can use a nationwide
health-information network and aggregate data that way and everything’ll be
defined the same, you have much to learn.

So I think – I wanted to comment that the metadata issue, the lack of
documentation, the lack of understanding of how things are collected,
generated, is a real problem, no matter what the source of data is, survey,
administrative data from agencies or whether it is clinical data.

MS. TUREK: I’ve got a question, I guess, more or less.

There really are kind of two types of state data. One is the data that the
state collects – chooses to collect itself to serve some function it is doing,
and the second is the kind of data we feds tell the states that they have to
collect to get benefits from a program like TANIF.

Many years ago, I used to work with school-district data, and Department of
Education used to ask for variables. They didn’t get a lot of what they wanted
for political reasons, and, I mean, the guy who was the head of the Council of
Chief State School Officers told me that any time the feds tried to collect
data to valuate them, they were going to be up on the Hill getting that data
out of the system.

So I would suspect that the data that a state collects because it wants to
and it is serving some kind of state function, like the Vital Statistics,
probably has a better chance of being better than Medicaid data, and it might
even have a more rational set of variables, and I would like to hear somebody
who knows more about this than me talk about the differences.

DR. STEINWACHS: Anyone care to respond to that?

Makes sense to me, but be nice to have –

MS. TUREK: Have you got any ideas, Jennifer?

MS. MADANS: Well, I guess I would agree with the general premise, but maybe
Vital Statistics isn’t the greatest example, because states vary very, very
much in how they use the information, other than the registration information.

I think so much of what Mike was saying that the -if you look generically
at the vital registration system, the registration information is very good,
because that is really why it is collected, and there is a lot of emphasis on
that, and that is what states use.

Some states use the other information that tends to be more of an interest
to researchers, partners and to some states, because they have more programs
that deal with that, and so there is variability in the quality of that.

I think the bottom line is don’t assume any data are good until you have
done an evaluation and then publish that evaluation.

So I completely agree on the metadata and spending the time to do the
evaluations.

MR. IAMS: I am Howard Iams from Social Security.

I have worked with AFDC Quality Control Data and Social Security data, and
my conclusion is if it is used to administer the program to calculate a
benefit, it is usually pretty good, and if it is not, you are using it at risk.

And the QC system that I worked in for six years – three years – states
would code whatever they wanted in the fields they didn’t care about.
Regardless of what the coding form said you were supposed to put there, they
would collect some data item they wanted and just stick it in the field.

And at Social Security, a lot of care is given in calculating eligibility
and a benefit, but some of the other information is not going to be as
carefully done.

In terms of mortality, in Social Security, we have several sources that are
reporting someone is dead, and when we pay a small amount for a death
certificate, and the funeral director is sending in the report to get the money
that we pay, because it is signed over to the funeral director, that is pretty
accurate.

When the states are sending in reports, which we pay for as well, they are
not quite as accurate, and, actually, it costs us money when we stop paying
that small amount of money that goes to the funeral director, because someone
doesn’t report a death and we keep paying benefits, and Congress, to save
money, will come along and eliminate that benefit. Why pay $250 to report a
death? And it will – They did it back in the ‘80s, and it cost a lot of
money, and they went back and started paying it again.

MR. RILEY: I guess one – situation with the Medicare data, on the claims
data, the conventional wisdom is that if it doesn’t – if their data are not
reported on the claims to set a proper payment rate, then you probably
shouldn’t trust it, but, on the other hand, data that are reported for payment
purposes can be gamed as well.

The provider has an incentive to try and upcode and do things like that
that might increase their payment amount. So you have different incentives at
work as to how seriously people take the information and if they are trying to
game it to some advantage, so that you definitely need to consider the
incentives of the person who is providing the administrative data when you go
to use it.

MS. TUREK: I am going to ask a very naive question, then. If, in fact, we
can only trust the payment data, what do we gain from putting the
administrative data in if we don’t know if it is very accurate? I can see the
address files, but on the MSIS data, the enrollment, yes, but is that all we
should use off of these administrative data files is the enrollment data?

MR. RILEY: Well, there has been work to try and validate some. I mean, you
have different levels of accuracy for various elements that appear on the
claims and enrollment files.

For example, there has been work to look at diagnostic information that is
reported on physician claims, and that is not used directly for payment
purposes, but I think the consensus of the validation studies on that variable
is that it tends to be accurate enough to be useful, but not anything you’d
want to make a life-or-death decision on.

So, I mean, there’s levels of –

MS. MADANS: The hospital diagnosis –

MR. RILEY: That is used for payment purposes. Again, we have noticed, over
time, there is a certain creep.

DR. SCANLON: I was just going to agree with his point that, I mean, it
really comes down to the quality of the data at the individual variable or
field level. I mean, because – and this is my conventional wisdom, too, for
years, that if it wasn’t for payment, it was potentially sort of unreliable,
and, then, as you pointed out, we have created incentives for how you report
things for payment, to the point where I would say it is more than conventional
wisdom. We have an empirical test.

I mean, the introduction of the prospective payment system for hospitals in
1983, when it suddenly became sort of such significance for payment, we saw
this dramatic shift in diagnosing reporting, and the story was, well, they had
become better at it, in terms of capturing all the diagnoses that sort of were
associated with an individual, or, potentially, as Gerry says, they recognized
the value in sort of reporting sort of things that maybe were marginal to begin
with.

In terms of if this – we have this problem, I think, this is kind of, I
guess, my sort of where I come out is what is the purpose – I mean, what is the
use we are going to apply this to, and sort of how much data error can we
tolerate? Because, in some respects, we are worse off operating with no
information. We have to think about sort of there’s errors sort of within this
information, but we still may sort of improve our decision making if we use
that data and we take that error into account.

Policymaking sort of always has to recognize that it is operating on a set
of information that is flawed, but it needs to still move forward.

DR. STEUERE: I have a comment, then a question.

The comment is, Fritz, they say is what age is one’s memories turn to
myths. So I appreciate the fact that your memory of my involvement had turned
to a myth as to what I had actually achieved, but thank you anyway. It was a
nice myth.

My question had to do with incentives, which is where we were going just a
minute ago.

As an economist, any time I see something where we feel like we are doing
something incompletely or not as well as we can, I am always brought back to
the question of what are the incentives of the system, and I wonder if the two
of you – either of you – have some recommendations on how we actually change
the incentives of our various systems to reach some of the goals that you
mentioned, and I’ll mention one, but you could go into other areas.

My sense is that when it comes to data, we of social science have it
totally upside down. If Madam Curie tried to publish an article or had tried to
publish an article in any of our social-science journals, she would be
immediately rejected because all she did was gather some data and run some
simple correlations.

And my sense is that the people within our statistical agencies who would
gather data, put it together, document it, do all the things that we have
talked about receive almost no rewards in our academic settings, even though
one could argue that they are the ones doing the basic research, and those of
us out there who are using the data sets are really the ones doing the applied
research, which actually gets the – especially in academia or in our journals –
tends to get the credit.

So I guess that is one area to go, but I guess within the agencies, too, I
am just curious whether either of you have given any thought to are there
recommendations that we could think of as a committee that says to HHS or other
agencies says, Here’s at least a couple of areas where you really need to think
about incentives of the system?

I think one you mentioned was bringing in outsiders occasionally to
constantly review the data sets as they are developed or however we do it for
whatever reasons, because they are doing research and that is their incentives
or because –

But I am just curious whether – just thinking about incentives, do you have
any recommendations?

DR. SCHEUREN: Well, you heard yesterday from Ed Sondeck(?) about two things
he thought out to be done, and I’ll repeat them again, if I got them right.

Despite the fact that I have aged considerably between yesterday and today,
Gene.

The first of them was this notion of having a small fund to get analysis
done.

And the second one – which could be done by an outsider – and the second
one –

DR. STEUERE: Could you just expand on a small fund for whom to do what?

DR. SCHEUREN: Well, let me repeat – Let me –

DR. STEUERE: (Off mike).

DR. SCHEUREN: He was talking about for NCHS, right? That he would be given
a small amount of money that he could spend at his discretion – okay? – to
bring in outsiders to work on the data, and he would also be given resources to
have internal analysts work on the data, which is what I think you were
alluding to just a moment ago.

That would be a fundamental, and it would be earmarked, and it might not
come from the agency. The problem with it coming from the agency is it gets in
the base, and then they squeeze you again. So it has to come from an NSF or
some place outside that, outside the base. Otherwise, it, in the end, doesn’t
matter more than a year or two.

That is a very good idea. I thought it was an excellent point. He made it
yesterday afternoon, and I would like to see that followed up. Talk to Ed and
develop it more. You asked me to develop it, but you were sitting next to him
when he said it, so – I know you don’t remember it, but – I am teasing you now,
Gene.

You don’t remember a lot of the good things you did, Gene. That is the
problem, because you are such an humble guy, and I am a student, so I try to
write down what you say, so that I can repeat it or recall it at various
points, and I hope I did a pretty good job of recalling it.

I think that – I want to make a point.

DR. STEUERE: You said you had two things –

DR. SCHEUREN: And staff, and increase the staff to do analysis. Earmarked
staff to do analysis –

DR. STEUERE: Earmarked – (off mike).

DR. SCHEUREN: That’s right. They have to be earmarked, because you get a
crisis – And if you are a producing organization, there are always crises. I
mean, it is just unbelievable. There is always a crisis, because the systems
are so structured that they are always at the point of a landslide, if you know
your complexity theory. They are always just at the edge of one more thing
wrong – okay? – and the whole thing goes. Okay? And it is really true. Right,
Ron? I mean, it happened to me a lot.

I have a story about one of the censuses in which the IRS was supposed to
produce a piece of information – it was an industry(?) code – and it wasn’t
being used for administrative purposes, and so one really smart service center
director who was trying to save money, says, We won’t key that.

Well, we didn’t catch it for a while. That was a big mistake. Big, big
mistake. Shame on us. Shame on me, in fact, because I was supposed to have seen
it done, provided some oversight, and it was too late when we found out about
it. Too late. Too bad.

We didn’t have the right systems to monitor it, and everyone was trying to
save every penny they could and beat the other guy in the next service center
over, and bad mistake. Bad mistake made. It really wrecked the Census Bureau. I
apologize again publicly. I apologized to many people at the bureau. I’ll
apologize again publicly to the bureau people here.

I want to make another point that was underlying something that my good
colleague said about metadata and about a subset of metadata called
paradata(?), which is about the process of the metadata.

Actually have written about this and given talks on this, and I think it is
fundamental to making the linkages learning systems. If you don’t do this –
okay? – then, you are gone.

Now, there is really good software out there to do this. I don’t know of
any statistical agency in the U.S. Federal Government that is using it. The
Canadians are doing it, okay? Some private-sector firms – I think RTI is doing
it. We are doing it at Newark(?), although not in a very large way. It is being
done at Brigham Young University. They run exit polls since ‘82 or
‘84, and they do a wonderful job of documenting things historically.

I mean, the real problem with documentation is when the person who did it
isn’t there anymore, if you can’t understand it anymore, if you don’t have the
Rosetta Stone somehow, it is gone. Very important, and when Howard talks about
these long-term systems where you add years and years of data – okay? – and the
initial starting point – however good it may be, wherever it came from – the
Census Bureau or somewhere else – is not well documented, you really have a
serious problem, and, moreover, if you are documenting these administrative
systems well – okay – if you are documenting them well, which means you have to
have money to do that and time, then, you will discover errors quicker.

If we had in place the system that I would be recommending today, we
probably wouldn’t have made that mistake for you those years ago. We’d have
made some other mistakes for you, but not that one.

MS. GENZER: I wanted to go back to the Food Stamp quality-control example,
because Howard Iams was talking about problems which I have seen, too. If it is
not connected to the eligibility benefit, it is not as accurate as one would
like, but agencies don’t have to stand helplessly and wring their hands because
you can improve the accuracy, even of those data.

We have a contractor, Mathematica, who, each year, takes the
quality-control data and edits it, and does data assessments.

What our agency does, then, is meet with the program staff, interagency,
and says, Look, Houston, we have a problem. We’ve got naturalized citizens who
are Native Americans in Oklahoma. Does that make sense to you? It doesn’t to
us.

So, then, what we did – and this was in ‘01, ‘02, ‘03 time
period – we had meetings with the regions and then with the states, talking
about these are all the areas that we have problems with. We were not getting
good information about vehicles.

Well, one of the reasons, we found out, was that states that have exempt
all vehicles or exempt one per household from the vehicle test from the Food
Stamp Research Test, they weren’t collecting vehicle data, if they didn’t need
to.

We realized in our office that, since we couldn’t rely on that data, we
said, Okay. We’ll drop this vehicle in exchange for having an assurance that
the states will make some improvement on the quality of the immigration data,
which we have to have to do our analyses and our reform simulations.

So you can work with the states who are putting together the data to get
higher quality admin database. So there are steps that you can do to improve
the quality of your admin data.

So our agency tries very hard to make sure it is available, and part of the
reason is that it is in the public domain, and if you could go home tomorrow
and download the Food Stamp Quality Control Data from the Internet, since 1996,
so –

DR. SCHEUREN: Compliments to you.

MS. GENZER: Yes. And the documentation, too.

DR. SCHEUREN: Yes.

DR. DAVERN: I just wanted to – The incentive question was really
interesting, and, certainly, what has helped us get this project underway was a
combination of things.

First, it was the Assistant Secretary for Planning and Evaluation, Mike
O’Grady, was very interested in our project with Census, so that got the HHS
side of people interested, and then Robert Wood Johnson kicked in some money,
but – and so it was money and it was power, basically, both, that brought that
together, and brought together a team of a lot of people working on this
particular project. So that was the incentives there.

But I do think that the best way to improve the quality of the data is to
use it for research, and as soon as it starts getting used for research, these
things are going to get done. So the more – Certainly, through access – Because
the data-collection agencies, the administration data-collection – the
administrative agencies are going to – They don’t like to make mistakes, and if
you can show them or document the mistake, they are going to fix it. That is
what –

DR. SCHEUREN: They don’t like to be embarrassed publicly.

DR. DAVERN: Well, right. Well, they don’t like to be embarrassed publicly,
and if you work with them and you are reasonable about it and understand that
mistakes happen and you go in and you improve the system. That is what we are
trying to do. You constantly want to work on improving the system.

So, in some way, whether it be through – Certainly, NIH could fund
researchers to do this stuff, perhaps giving subcontracts to Census or NCHS.
You have to actually work with the data, because we can’t see it or if it can
be done at a research data center where we can actually get in and have access
to it, and a Title 13 benefit, for example, would be producing metadata.

So if you go in and work and do the research with this stuff, you’ll
provide it, too. You’ll have to provide a piece of that metadata.

DR. SCHEUREN: Good. I like that. That is very good.

DR. STEINWACHS: Very good.

Other things?

MR. PREVOST: Well, I was musing. There is probably smoke coming out of my
ears, this late on the second day of a conference.

But one of the things I was thinking about – and, often, as researchers, we
don’t think in this way – is that as you are looking at data from either your
own organization or another organization, you compare them and you say, Hum,
something is different here. It is not working, and somebody is – quote/unquote
– wrong, whatever that means.

When you do find that things aren’t matching up and you suggest a change to
one of those organizations, we need to think about not just a statistical
environment, but we need to think about how can we measure that agency’s or
that entity’s return on investment, okay?

If we can start showing that, yes, when Researcher X accessed File Y and
completed this research that they were able to reduce the operating costs or to
improve the measures quantitatively that an agency is producing, I think it
would be a huge benefit to be able to continue this type of research and to be
continuing this type of linkage activity.

DR. SCHEUREN: It is important, though, to have this candid tradeoff which
the Food Stamp example was beautiful. I am going to give up something. I have
two bad things. I am going to keep one and make it better and give up the other
one. That is the right –

I mean, sometimes, you can improve the operating efficiency of an
organization by simply focusing on a weakness, and that certainly happens, but,
often, it is weak because you are just not spending resources on it.

MR. PREVOST: Well, yes, and just to add to that, I mean, how often have you
worked with an agency and they say, I don’t know why we are collecting this
piece of information. It has been on the form for the last 25 years, and you
finally say, I really don’t need this. You have automatically improved their
efficiency, but you need to capture that, so that when you go back again or to
another agency, you can start using these benchmarks to show them that, yes, it
is effective to do this work.

MS. MADANS: Kind of expanding on what Ed was talking about yesterday and
what has been brought up, I think it is true that there is not a lot of glory
in doing some of this basic work, telling people what is wrong, and it doesn’t
make you popular, and it certainly doesn’t get you articles in JAMA.

But I think a lot of the agencies are trying to incorporate more
methodologic program, and what they need to hear is that that is useful, and
useful to users, useful to the people who are advising the Secretary.

I mean, there has to be some reinforcement, because, in order to do it, if
you are not going to get new money – which, of course, we would all rather have
new money – then, you are going to be not doing something else, and so you also
then don’t want to get bitten by saying, Well, why are you not doing X? Well,
because we are doing quality stuff, and I think we are moving a long way to
providing metadata, especially the Web. It is much easier to do it.

We used to try to do these reports. They were hard to do. Well, now, we
have put all the metadata, all the quality stuff up on the Website, it is so
much easier.

But there has to be some acknowledgment from someone that this is an
important thing to do.

We would like – I think – what Ed was really talking about was a grant
program, which is a nice thing to do, because you can bring the outside – the
community in.

We like to have IPAs come, because, then, they can have access to things
that they can’t have other places. We have a senior fellowship through ASA.
These are the kind of things that we would like to have external input on, but
it is a cost, and if there is no changing that incentive, if somebody is not
saying, This is a good thing to do, people are going to say, Well, it is not
considered important. It is more important for me to go out and –

It is more expensive to figure out that I don’t need that item that I have
been collecting for 25 years and no one is looking at it. There is a cost to
doing that. I might as well just keep collecting it and everyone will be happy.

DR. SCHEUREN: There might be a stakeholder, like the Census Bureau, that
needed it, too.

MS. MADANS: Well, I’ll tell you, every time we take something off that no
one has been using for 25 years, somehow –

DR. SCHEUREN: Somebody needs it.

MS. MADANS: – the person who wanted it, has my home phone number.

DR. STEINWACHS: They may be resurrected from the graveyard, right, Fritz?
Some of those people coming back.

DR. SCHEUREN: That is another project I worked on, yes. We won’t do that
now.

DR. STEINWACHS: We have talked a little bit about what might be possible
areas in which the National Committee on Vital Health Statistics could make
recommendations, and, as you know, those recommendations are made to the
Secretary of DHHS, but those recommendation letters also go more broadly, in
terms of being shared, and so some of those recommendation letters in the past
have made it clear that the value and the idea that DHHS might take the
leadership of bringing together multiagency groups to try and address specific
issues, and so in areas of consolidated health informatics and some other
things, there have really been multiagency activities that have gone outside of
DHHS.

So I just wanted to have a chance here, if there are other areas that you
would like to identify where you think the capacity of this committee to make
recommendations would be valuable.

Deb.

DR. SCHRAG: I just want to bring up the issue of death certificates. So
mortality is a great outcome, and we all love mortality, and it is measured
pretty well.

DR. STEINWACHS: I don’t like that outcome whatsoever.

DR. SCHRAG: Okay. Well, I mean, dead is usually dead, but we all pick up
the newspaper, and, not to mention journals, and look at issues of
cause-specific mortality, which is sort of one of the key vital statistics, and
death certificates in most states basically have proximate cause, contributing
cause and underlying cause, and it is one of the most misused statistics are
cause-specific attribution. It is an area where there is total anarchy, very
little guidance, very little in terms of metadata files, very little in terms
of training the individual – I mean, this is something that resides at the
level of the individual providers.

I will speak as a physician. I code death-certificate data. I started doing
it the first week I graduated from medical school. Those are the people who
code death certificates for deaths in hospitals.

You are called to see a dead person. You don’t know anything about them. It
is two o’clock in the morning. You know what the easiest thing to do it? It is
to write down, Proximate Cause: Heart Attack. Contributing Cause: Edema,
pulmonary embolism. Underlying Cause: Cancer, or – based on what floor the
patient – It is done terribly. It is done terribly all over the United States.

It wouldn’t take that much – There are not even documents. There is no
training for physicians.

MS. MADANS: There is. There is. There’s a lot of –

DR. SCHRAG: There is documentation, but there is no training at the level
of the people who are filling them out.

MS. MADANS: We should talk. There is training. We develop a huge amount of
training.

I am agreeing with you. There is a lot to be done and there is a huge
literature on death certificates and the problems with death certificates and
cause of death, especially in the –

DR. SCHEUREN: Your mike is not on.

MS. MADANS: Yes, it is on.

DR. SCHEUREN: It is? Okay.

MS. MADANS: I mean, I think it is a good example where there is
information. We try to – This is a state issue, because states are in charge of
death and birth and marriages and divorces, but it is how do you get – when the
source of the data – you have no control over – How do you improve quality? And
that is happening in all of these administrative systems, because you have no
control of the person who is actually filling it out, and, I think, in birth
certificates and death certificates, there actually is a lot of work with
coroners. We go to conferences. We do the training, but if it is not in medical
schools, the medical schools don’t want to hear about it. They are too busy to
worry about death.

It is very hard to get that dialogue going, and unless – There has to be
kind of this joint agreement that this is worth time and effort, and how you do
that, I don’t know.

DR. SCHRAG: And I guess – I think – Yes, it may be at coroners, but it is
not at medical schools or medical providers, and I think the way to get them to
pay attention is to incentivize them by providing them with data back.

I don’t think it has to be monetary incentive, but they actually care about
the data, and if they see what the data can do for them, then, all of a sudden,
I think that you would see even more in terms of input into training, because
it is an absence. Maybe some states are really good at it.

DR. STEUERLE: This last discussion, let me say, I do think that is an area,
not just the subcommittee, but the full committee has recently decided we may
need to engage. We did have some discussion on vital statistics and even the
issue of whether – this is also an issue for us – is whether we need something
like, I am thinking like the multistate tax commission, where it is not tax,
but whether the state-level attempts to develop protocols and standards and
stuff like that are adequate.

I don’t know enough about it to say what could be done, but I think it is
something we should probably proceed to do. Although, I am somewhat a follower
of Woody Allen, who said, I don’t mind dying. I just don’t want to be thee when
it happens.

The question I have is, with respect to ways of – I mean, getting back a
little bit to incentives. I am thinking of a couple of gaps, and I don’t know
fully how to approach it, but let me base one on experience like Fritz and I
have had at IRS, and that is with respect to data that is in one agency that is
not related to their mission that no other agency really has a strong incentive
to get at.

So I am thinking this may sound minor to people here, but IRS has data on
health savings accounts.

Now, I don’t know how much effort they are or are not putting into it.
There are people, for health reasons, who have a very different interest in
developing data on health-savings accounts and whether people who use
health-savings accounts get the same healthcare as others, which would be very
different than IRS’s concern, which might be whether people are paying their
taxes adequately or IRS’s data on the use of deductions. I don’t know whether
deductions, which are probably concentrated on things like nursing-home care,
would be related to somebody doing research on nursing-home care.

Certainly has, indirectly, data on – to the extent it is listed on
employers – payment for health insurance.

So there are these data sources with someplace like IRS, but my sense is
there is very little incentive for HHS to say, Gee, we’ll provide 10 percent of
our health research funding to IRS to go do this data, and yet it could be
within IRS there is some really, really valuable information, and I used IRS,
because I used it as an example, but I am sure this is true across a lot of
agencies. It could be that adding a health variable to one of these educational
longitudinal studies could really help us to understand the extent to which
education outcomes are good or bad.

To the extent you can talk the agency into doing it, it is one thing. To
the extent you actually provide funding to other – interagency funding is
another, and I know there’s been minor examples of BEA, through – a lot of
pressures funding, say, IRS to get data, but that is because IRS has the core
data to develop national income statistics. So you are at that level, sometimes
you can get interagency funding.

So I was just curious – So that is one area where I wonder whether we need
to examine incentives.

And the other one has to do with whether we do come back and do
self-criticism enough.

You know, you mentioned several times the improved quality of Food Stamp
data, but I know for certain debates I am in, the quality of the data is very
misleading. So I am in this debate all the time of whether income credit has
higher error rates than Food Stamps, and there’s always these – Well, Food
Stamp error rates are down 5 or 10 percent.

Well, the reason Food Stamp error rates are 5 or 10 percent, they have a
monthly income-accounting system. They essentially have abandoned that and
said, Well, we are going to assume that, for compliance purposes, if you
reported your income right one month, it is right for the next six months.

It is also accurate mainly if you are talking about people who don’t have
earnings, so people who aren’t in the workforce at all, sometimes your earnings
– your accuracy of your earnings record is correct, but it is very inaccurate
if you get to the question about the three-million missing people in the Census
who are often in these Food Stamp households, depending how you measure
households.

So there is this argument that – this is a big policy debate – the
earned-income credit is a much worse measure than Food Stamps, but I think
actually – at least I look at the data, I don’t think it is, but for the health
data, there is probably the same thing. How do you get outside researchers to
come in and be critical – I mean, the agency has very little incentive to have
people come in and be directly critical about the way they are doing things,
and I am just curious the extent to which we need to raise that issue as well.

So I give two examples, to summarize. One is how do we get outsiders to
come in and do critiques? How do we pay outsiders to come in and do the
necessary critiques, when the outsiders don’t have a strong incentive system?

And, secondly, how do we get agencies to do more cross funding, where it
may be very useful – another agency has the data they really like?

DR. SCANLON: I wanted to just sort of comment, because of an experience
with the – potential experience with the IRS that never sort of happened, and
kind of with the principle that – recognizing that the unique role of the IRS
as the tax collector should be taken into account, and that one needs to sort
of approach that question, because that is a controversial role, in terms of
the public.

In HIPAA, one of the features of HIPAA was the medical savings account,
which was the predecessor to the health savings account, and it was actually
the thing that held up HIPAA for a long time, and there was finally a
compromise allowing for a demonstration program to go forward with 750,000 sort
of people that were going to enroll and an evaluation to be done or be
contracted for sort of by GAO.

And when we thought about that evaluation, we realized that the only way we
were going to identify sort of the people easily that identified people with a
medical savings account was to go to the IRS, and the issue was sort of what
were the sort of patterns of use among sort of people with medical savings
accounts versus other parts of the population, because the people that oppose
medical savings accounts say that people are going to forego sort of important
needed services because of the high deductibles and we are going to get sort of
poor health sort of as a result.

We rejected that sort of as an approach because we couldn’t imagine sort of
having it leaked that the idea of, Hello, I am from here – from the federal
government to ask you about your health. I got your name from the IRS, and that
is the kind of thing, I think, that we – it was not a good application.

Now, I think there are other applications that are potentially good. I
mean, it is a question of whether, within the context of information that the
IRS has, there is more that can be gathered in terms of useful insight for
healthcare, but, at the same time, there’s limitations on what we can do, and I
think what we need to do as we move forward is to think about sort of how do we
give advice about how you maximize sort of the benefit without incurring the
risk.

We were worried about a backlash in that regard. This was 1996, okay? The
climate was even sort of more sort of, I guess, fractionated, sort of then than
it is today. There was a very sort of strong sort of political sort of
atmosphere that – and we were worried sort of about what would be the response
to that.

And so I think that we need to sort of find – I mean, as we give advice to
the Secretary, find sort of the directions where there’s some safety in the
path, I mean, because we can sort of make some progress forward, but if there
is going to be a response that is too negative, we are going to be set back for
too long and it takes too long to recover from those setbacks.

MR. PREVOST: Just an idea, and perhaps this is coming from someone who has
been head down in the trenches for too long.

As I look around, one of the things – I love coming to these conferences or
conferences like this because I find out about all sorts of data that I never
even knew existed before, and I find that as a concern as a person who had
operated as a statistician – now, I am just a manager – but – and had to solve
real-world problems and really didn’t know that these resources existed.

And, furthermore, are the data available? What can be done with that data,
and is there some possibility that there could be an interagency working group
that could be looking at both data availability and linkages and measuring the
quality of those linkages at the working-person level, not a bunch of us
managers around, but those people who are statisticians who have to solve
real-world problems that each agency, frankly, has?

And the incentive here is is that they would be working together to – as a
problem comes up with Agency A, maybe you’ve got an interagency group that
could be focusing on that, looking at specific files, so that each agency that
is participating in it could see that as a shared resource upon which that they
could tap.

Now, this may be a utopian view of the world, but it is just a thought,
something for you guys to ponder.

MS. OBENSKI: I guess Ron and I have been working together too long, because
I was kind of converging in the same direction, and that is, I guess, that I
have never actually been to a conference like this, and I think it has been one
of the most remarkable things that I have experienced, in terms of making
progress in record linkage.

And I guess where I was thinking was trying to use kind of like the
Medicaid undercount models, like what are the big problems you are trying to
solve that could warrant record linkage, and then building on what Ron said,
bring the right groups together, and something like what we did in the SHAY
DACK(?) or the Medicaid undercount model, because that is where the incentives
are.

The incentives are what is a big problem and what are the pieces of it and
what are the different views of the different agencies and what do they have to
gain from it. Because we’ve got states involved that are willing to be involved
in this project. We’ve got ASPI(?). We’ve got CMS, and, as I said yesterday,
what is remarkable is everybody is coming at it from a different – for a
different reason, a different agenda.

Ours is improved statistics. Mike’s is improving the CPS, so he has better
data to do his research, and so I think that that would be a tremendous outcome
of this group.

DR. STEINWACHS: I have recorded it. It has been fully recorded.

Other ideas, suggestions?

Sally, you were mentioning states, too, and I think one of the themes that
has come up over and over again is are there better ways to work with states
than we are doing now or maybe more integrated ways that bring multiple
agencies together to work with the states?

MS. OBENSKI: I think that our experience, albeit limited, is that I think
that whoever brought up the question about incentives, I do think that there
appears – and this is just an observation – my observation, not the Census
Bureau’s – is that there seems to be, from what we have experienced in working
both with state folks and with federal folks on different projects, is that
there are competing agendas in terms of what the states’ incentives are in
administering their program and what the federal incentives are, and that is
very, very important to understand how to bridge that, because big federal
programs that allocate to the states need the state’s help in administering the
program, but that is just an observation that I think needs to be addressed
before we are really going to make these two pieces fit.

DR. SCANLON: I have been here thinking that we potentially need some kind
of process for reconciliation.

I mean, I am guilty sort of, I think, over the last two days of being kind
of the nay sayer in terms of quality of information and sort of saying we have
to worry about the quality of information, and I’ll tell Deb that, almost on a
weekly basis, I do say, Don’t let sort of the perfect be the enemy of the good,
and I still believe that even though what I have said here over the last couple
of days.

But it seems to me that sort of with respect to quality of data, with
respect to privacy, there is an issue of the tradeoff between that and the
social benefit we might get sort of from sort of moving forward, but I guess I
am concerned that we don’t have – necessarily have a situation where the
decision maker is weighing those two things.

The decision maker may be approaching this from sort of one perspective,
okay? Their job is that, under statute, they are to protect the privacy of sort
of all of the respondents that are in this data set, and they are moving toward
the point where the probability of disclosure is .000001 – okay? – and is that
sort of where we want to be from a social perspective? And the answer is,
potentially, not.

It would not be unreasonable, potentially for a government. It would not be
sort of an arbitrary and capricious thing for government to do to increase the
percentage of disclosure to sort of have three fewer zeros sort of in front of
that one.

And the question is how are we going to get there? How do we get there in
terms of weighing sort of these benefits versus the risks or the – of either
sort of privacy or of poor sort of data sort of leading to erroneous
conclusions?

And I don’t know whether the IRB model, which was suggested here, whether
there is some variant of it that we could think about sort of in government or
whether we should think about somebody who does – in some respects, serves as
the arbitrator, hears both sides of this and comes to some conclusion as to
what is the appropriate sort of tradeoff, because all the things that we do
with respect to statutes, all the things we do with respect to guidelines are
still going to have subjectivity in them. Somebody is going to have to make an
interpretation saying, This is reasonably consistent with that statute,
reasonably consistent with that guideline. It is never going to be black and
white. It is going to be something where you can say, Okay. Clear case. The
clear case is not to do anything, which we know has incredible social –

So that in trying to atone for my negativity about sort of quality, I
wanted to sort of offer that as a process way of dealing with moving forward
for the future.

DR. STEINWACHS: You have atoned well.

MR. DENBALY: In terms of process, and in the context of everything that we
have been talking here, suggestions that are unmade, working groups are being
put together, and perhaps one project can be picked up and used as examples.

One that I have in mind is one that Ron listed as the first one on his
slide, high-valued future research project, and that is the connection of
NHANES(?) to WIC and Food Stamp data, in the context of health.

I think what we eat is probably one of the most important things. We need
to understand why we eat, where we eat it, how we decide how much to eat and so
on, and, in particular, we are spending over $20 billion on Food Stamp and
other programs as such. We need to understand the consequences of these
programs and even the administration of these programs. So studies, as such are
highly policy relevant.

So we have this group in here and a lot of the issues that we have been
dealing with is what you have been talking about, states and providing the data
and linking up with NHANES is very tough.

So I am suggesting that this group is definitely needed to give leadership
to addressing some of these issues that we are – in a broader view, that we are
dealing with, and perhaps this group and groups as such can work together to
bring it to the table to say, Here is the kind of problems that we are dealing
with. What should we do with it? How do we address it?

MR. IAMS: My hope would be that you would do something that would make it
possible to use better data in policy analysis and evaluation by permitting a
greater exchange of linked survey data across agencies with less pain and
suffering or prohibition.

For example, Gene Steuerle yesterday said, Gee, if you had earnings records
tied to Medicare records, you could look at something connected with Medicare
expenditures and lifetime benefits and lifetime earnings.

That would not be possible to happen in the current legal context. Our
administrative data – Well, maybe it could, but I doubt it, because you
wouldn’t have any authorization for the Internal Revenue Service to permit this
to happen.

My ultimate goal would be if it is for statistical purposes in a safe,
secure environment that is not going to violate confidentiality that any
exchange should be possible with government data or government-linked survey
data to other agencies.

We have some linked survey data that, if ASPI had it, their decision making
would be a heck of a lot better than it is now. Rather than making it up on the
back of an envelope, you would have something that is closer to relationships
that exist. I won’t claim that it is perfection, but it’ll be closer.

The Title 13 constraint of Census data linkage is very, very limiting, in
terms of this kind of thing. A straight policy analysis on something connected
with Medicare or whatever might have nothing to do with data quality. It is a
question of having information connected with a decision on what the agency
supports or doesn’t support.

That kind of exchange in matching up data that currently don’t ever meet
each other isn’t going to happen without legislative changes that make the
statistical purpose and a legitimate or the policy analysis of statistical data
in a secure environment a legitimate activity that the federal agencies can
exchange and share data amongst each other, and I don’t know the prospects of
that.

Brian Harris-Kojetin agreed, when I mentioned it to him this morning, that
we really need legislation that allows data that is tied up in one agency to be
shared for statistical purposes at another.

Of course, you’d want to make sure that it wasn’t going to be used for
administering benefits or taking sanctions or anything of that nature. So you
need some sort of protections like CIPSEA has offered. I don’t know if CIPSEA
is the proper place. One person pointed out if you opened CIPSEA up, you could
have bad things happen to CIPSEA.

But we – I think the exchange of linked data and linking more things would
lead to wiser policy analyses and wiser decisions, and the agencies need to be
able to share more than they can share.

MS. TUREK: (Off mike) – the statistical enclave ASPI would not qualify,
because we are basically a policy office in the Office of the Secretary, but we
are probably among the most intensive data users in HHS, and we have the
broadest data needs, because we really are analyzing all the federal programs,
and if they do open this up, I hope they can figure out a way for us to have
access to the data, too, clearly, under controls, but none of the sharing
agreements I have ever seen would include us.

DR. STEINWACHS: Joan, we have to send you to a secure data center and not
let you out.

MS. TUREK: (Off mike).

MR. IAMS: Well, but Joan couldn’t come and use data at my secure data
center. I am three blocks away from her. We would have to have a special
agreement and you’d have to figure out some Title 13 purpose with Census to
permit that to happen.

So it is not just sending someone to a secure data center. If they are not
from the appropriate agency with – the appropriate legislative requirements,
they can’t –

MS TUREK: (Off mike) – our travel budget.

DR. STEINWACHS: I’ll get Ron in just a second.

I thought you were saying, Howard, that if we could change the legislation,
it would be, in a sense, to create something where someone could, like Joan, go
to a secure data center, do something for ASPI, which otherwise couldn’t be
done if you were saying transmit the data to ASPI, and so on, and that was –
Okay – Ron.

MR. PREVOST: Yes, thank you.

I think I mentioned this in the first day. I just want to reiterate it. I
mean, absent legislation being conducted, I think one of the things that would
help all of us is if there were standardized agreements that – particularly – I
am going to suggest OMB. I don’t know. They may – they wouldn’t – I think the
appropriate person had blessed that said these are the way that we are going to
share data between federal agencies, and they had standardized components and
all the lawyers understand what these standardized components are, and so we
don’t spend years and years and years working on agreements between the
agencies.

If it was just getting rid of all the – I am not a lawyer. So there is
language I look at and I go, The word is the –. What is the question? But they
interpret it differently. Okay?

So in looking at this, if we had these standardized agreements with a
blessing and a set of procedures that says, Yes, if Agency A and Agency B want
to share data and they both believe they have the right to share data, how can
we cut the time that it takes to do this?

If we could do it in two months, rather than two years, it would be a huge
advantage to the entire federal government.

MS. MADANS: We have kind of had two – several parallel conversations going
on the two days, one of which is about access and confidentiality and the other
is kind of the ease of the linkage.

I think we absolutely need to do what you just said, because it is – It
won’t solve the problem that Howard brought up. I mean, if the legislation says
you can’t do it, then you can’t do it. So we need to fix that, and we need to
figure out how to do it quicker, but a lot of these things are done with the
understanding that there will be very good confidentiality protection.

And Fritz brought up the IRB, and I can tell you that our IRB, which is
very well schooled in statistical uses – because that is basically all we do –
much of what they allow us to do, without getting very explicit consent – I
mean, really going through every possible bad thing that could happen to you –
is because we will protect the confidentiality of the data.

And so while many people really are pushing for more access, the more you
expand that access or there is the perception of that access, the less you
probably are going to be able to do in terms of the linkage, and it shouldn’t
be an either/or, but, at some point, it will be, and maybe we have been – We
have always dealt with it is either public use or it is confidential. There is
nothing in between. You know, it is either in the data center or it is on the
web, and we are changing that. I think we are trying to think through –
especially now that we have the sworn agent – what kinds of risk, what kinds of
– this is kind of the – what would you call it? – the portfolio idea.

But it is going to take a lot to work that out, and while we are figuring
out how to do these standardized agreements, it would be nice to be able to
figure out what are the primary criteria, how you determine what kind of data
you can put where for what kind of access, and, for certain things, I am sorry,
you are going to have to come to a data center, and when – I think it was
Heather, when she said she remembers the punch cards and going to the computer
center at three o’clock in the morning, because it was cheaper and all those
good old days, that we are now in – we have been in a position where data
access has been very easy. We went through a period where – just gave you the
CD-ROM. We put it on the Web.

I have a feeling that was a very, very nice golden age in terms of data
access. That is not going to be the main access route in the future, that there
is going to be more of a range, and we can certainly make people’s life easier,
but users, I think, also have to kind of get a reality check that linked IRS
data with our DNA information is not going to be easily accessible to users,
that we are going to make you jump through lots of hoops before we give you
that, and I think that is our responsibility.

DR. BREEN: I think, though, that – you mentioned we had been talking on a
number of parallel tracks, and I think one is providing data access to users
outside the federal government, but the other is that even – we can’t even
provide access within the federal government in a timely manner.

So I think that we need to think about both of those things, and, then, I
think this notion of the portfolio that Julia had yesterday is a really, really
good idea, and a lot of people have mentioned that, that there can be various
hoops that you hop through depending on how much detail you want, and for
exploratory analysis, and some of the original stuff, maybe a public-use data
set or some very basic information is just fine, and it is only subsequent to
that, when you are – you have found you’ve got enough information to test your
hypotheses and you can write your grant with that, that then you can move
forward with the rest, and maybe we need to relax some of our standards about
how much information you need to provide in that grant, too.

And one other thing I wanted to mention was I think it was Gene who said,
Well, can’t people come in to the federal government and kind of take a look
around and maybe examine the culture and make some suggestions, and, certainly,
they can, through IPAs, and I know at NCI and at NIH, generally, people come in
and do program evaluations. They come in a team of people almost like they
would in an – Well, like they would in an academic department for accredation
purposes to evaluate what is going on and make suggestions on what are the
strengths and weaknesses and where you might want to be in five years or 10
years or something like that.

Plus, on a smaller scale, we have had people come in under IPAs –
anthropologists, management specialists – to evaluate what is going on and to
make suggestions.

So all of these things are possible, and I think there are a lot of
creative ways that we can think about using and building on what we already
have as we are trying to get legislation changed, because legislation change is
a long process, and so I hate to put all our eggs in that basket.

MS. GENZER: I wanted to go back to the data-center issue.

I think it is very important if we are moving more to data centers that the
data centers be staffed adequately so that once a person has access to it that
they can then get the data that they need.

For instance, if you need to – if you are granted access to certain
variables, that the agency that is providing the data at the data center has
the staff to be able to get that certain data to you that you have been
authorized to use, so that you are not cooling your heels for an unspecified
period of time, and that is especially important if you are a federal agency
working with a contract and your contractor has a specific schedule and you
don’t want that schedule to be way off whack so that the staff on the
contractor that you have available for your contract is now busy doing
competing contacts, because they were expected to be finished.

So wanted to mention that.

MS. TUREK: I was thinking about what Heather said this morning – A lot of
things we are doing, two months is too long. The issue – we need the answer in
a week.

So if we have to go to data centers, there needs to be some kind of
agreement that we can get much faster turnaround than two months, which means
we would have to negotiate a more general agreement that would allow us to do
certain classes of studies rather than a particular project.

I mean, it is in the nature of the policy arena that – two months it is
like the long run. We are all dead. I mean, frequently, the issue – and this is
something the Secretary is interested in, because he is the one who is being
given the information – or the White House or the Hill.

So we go up and say to him, Because we have to go to data centers, we can
no longer do this kind of analysis, it wouldn’t sit too well.

MS. MADANS: So global access.

MS. TUREK: I mean, I think you can – for certain kinds of studies, rather
than each study very specifically deciding that you are allowed to do certain
classes of studies.

MS. MADANS: We had a call from CBO last week, was exactly this, and that is
what we are doing. They are going to have a generic kind of topic-specific, and
when they need to run something, they just have to make sure somebody is there.
Unfortunately, our data center is not busy every day, so there is not someone
always there to welcome you, but as long as we are there, they can come and do
the analysis. So I think that is the kind of thing you have to kind of build
into it.

I don’t know – We have different authorizing legislation. I don’t know if
Census can do that, but ours does allow us to be more generic in terms of what
the project is.

MS. TUREK: If I understand with Census, it has to serve a Census purpose.
So you have to find some way that the results can be used that are in line with
your mission, not so – I mean, if Census wanted something and the best place to
get the data was the Census, would it be covered by Title 13 or would they be
told, You wrote this law. Too bad.

MR. PREVOST: Well, I am certainly not the one to speak for every word of
our mission, but, I mean, certainly, what our job is is to disseminate data. I
mean, it does no good to go out and be the collector of data if you cannot
provide it to the people who you are supposed to be delivering it to, and that
is why we have the Research Data Center Network.

And in doing this, and as we have said around the table here, that in using
the data, you can find out where the warts are and you can tell the folks what
they can do to improve the information; that is, if you are working with our
data at a research data center, is a Title 13 purpose, to research and to
suggest improvements to the data that we have at hand.

MR. RILEY: We have talked about some of the administrative data sets and
the fact that, I guess, the lack of documentation on some of them is a barrier
to people using linked data sets, particularly if they are used to, say,
analyzing survey data and they find the prospect of adding administrative data
sort of daunting, and CMS has started the Resdac Project to help new users of
Medicare data. Resdac is a contractor to CMS that helps people, not only use
just Medicare data, but they help SEER Medicare data users and so forth, and
they have taken that on as part of their responsibility.

So that might serve as a model for other agencies that have complex
administrative data sets, and it might help increase the demand for data linked
to those data sets and might be something to be considered.

DR. STEINWACHS: Is there something up on the Web that talks about what they
do? I was just looking for a document.

MR. RILEY: Resdac has its own Website. It is www.resdac. – it is at the
University of Minnesota.

DR. STEINWACHS: So it is probably –

DR. DAVERN: I think it just changed, because all the Websites changed, but
I you can just get there – resdac.org, I think, gets you there.

MR. RILEY: I think it ends in edu, but I think there might be something
about Minnesota in between resdac and –

PARTICIPANT: Google it.

PARTICIPANT: Google it. That’s –

MR. RILEY: Yes, Google it. Google resdac, you’ll get it.

DR. STEINWACHS: Yes, we’ve all got consensus, Google is the way to find it.

I thought I would start bringing us to close, because I know we have lost
some people, and I really appreciate all of you who have stayed throughout
this, because this is really what we had pictured as a real chance to have a
dialogue that was among the committee, the agencies, the users and the
producers and providers of very critical information that supports both the
nation’s mission and, from our point of view, is critical information to
understanding health outcomes and ways to intervene to improve that.

Just so you know what we are doing, we will be looking at what we have
heard and what we have gotten from you, in terms of our capacity to use that to
make specific recommendations to the Secretary and, as that moves ahead, we’ll
be happy to come back and share those products once they are actually approved
by the committee that would be going to the Secretary. They get posted at the
Website at the time that they are actually signed and sent to the Secretary.

There has also been a suggestion here, which we are going to take back,
that maybe there ought to be subsequent meetings, so that we may be coming back
to you talking about what kinds of topics.

There has been a lot of discussion about state data and might have some of
the same people around the table here, but people from the states or others.
There might be other issues, so that we are considering are there next steps
that are important for us to take in learning and gathering information, and
those will probably be more focused.

I think this has been a very open and, to me, a very, very productive – I
sort of thought I knew what was going on, and this convinced me, after about
the first five minutes, I didn’t, and I was here to learn a lot and have
really, really enjoyed it.

I also want to thank Cynthia. Cynthia has disappeared, but Cynthia may hear
me someplace that, without her, we wouldn’t physically be here, and there was a
lot of rushing to get this altogether.

And very much, Joan, I appreciate that you now – I now know you know
everyone and – Well, everyone that is important anyway, and without you, this
wouldn’t have happened and –

MS. TUREK: And I want to say thank you to everybody for agreeing to be part
of it. I think it was the participants who really made it, and, although you
hear me complain a lot, I think all of you are really great, special people and
you do wonderful work.

DR. STEINWACHS: This has been recorded. So, now, you can play it back
anytime you want. Joan has certified. Joan has said –

I also, again, from the committee’s point of view, want to thank Gene and
Nancy, because it was really their leadership that got us going as a committee
on this, and, then, Jim Scanlon said, Hey, ASPI is very interested in this.
Joan is working on this. Brought us together as a team, and so the rest of us
have benefitted from all of that, and certainly benefitted from everything that
you have come here and shared with us. So –

Also, so you know, the audio tape – I think I mentioned this – will be
posted on the Website, once it is done. There will be a written copy of the
transcript of this, and that our plan is to post the slides on the Website,
too, so they are available, so that when people listen to the tape, they can
see the slides as well, as we were going to copy them all for the committee
members, because many – made many points in there, and we don’t want to forget
those, and we want to capture those.

I don’t know if there are any other comments by committee members, but, if
not, thank you all very, very much.

(Whereupon, the workshop adjourned at 3:25 p.m.)