No, The Brookings Study On Student Loan Debt Isn’t ‘Garbage’

It’s just research like all research: conducted by humans, with limitations and caveats, but also deserving of a respectful, accurate reading.

There’s way too much student loan debt in this country. (I hold some of it myself.) I’ve written about it at length, and many times, over the course of years. I’ve written about ways to reduce the cost of college, about structural changes to the student loan system, about the desperate need for state governments to reinvest in higher education, and about a potential system of no-frills, low-or-no tuition federal universities that could graduate thousands without loan debt. It’s a big economic drag to have so many young people pouring money into student loans instead of into houses and cars, and it’s a moral travesty for so many people to suffer because they can’t afford to pay for the education they already received.

However.

It’s also the case that people who write and argue about the student loan debt crisis frequently speak about the problem irresponsibly. People throw around fake figures all the time, propose all sorts of doomsday scenarios, and generally represent the problem in ways that are not supportable from data. I can’t tell you how often I’ve had people talk about “every graduate running around with six figure debt!,” when six figure debt is extremely rare. Or when people compare the size of the problem to that of the housing bubble, which is simply not correct. The housing bubble was orders of magnitude biggerthan the student loan crisis. Pointing these things out is not a matter of disagreeing that there’s a problem, and it’s not a matter of being callous to those struggling under student loan debt. It’s a matter of being responsible in talking about a problem, under the theory that this is the only way to solve such problems.

So I’m seeing a lot of breathless sharing of this piece by Choire Sicha, attacking the methodology of a Brookings Institution report showing that student loan debt concerns are somewhat overblown. Like all studies, there are issues with the methodology of this one. But Sicha does not do a great job of critiquing this study. I don’t blame him– methodological critiques are really hard and take a lot of work. I’ve taken something like 9 graduate courses in research methodology and statistics, and I still find confronting methods and methodology an imposing task. But there’s lots that’s off, here. And Sicha isn’t mincing words: he’s saying the study is garbage, and people are listening to him. That’s not supportable, at least not from the evidence that he provides, and it’s unclear if he’s angry about the methods or about the conclusion.

Of all the households in that study, only about 1711 have “household heads” that are younger than 40. That’s what they’re extrapolating from. (And, intriguingly, a small number of those have a head of household younger than 18.) This is not a big sample!”

This is something I’ve written about before – people dramatically overestimate the sample size needed to make responsible statistical conclusions. A sample size of almost 2,000 isn’t just big, it’s enormous. The standard error of a sample of this size will be very low. Absent systematic sampling bias (as opposed to error), the odds of the underlying population being significantly different from a sample of this size is tiny. Saying that it’s not a big sample just displays ignorance about the standards applied in statistical research.

One effect of this age spread sample is that it includes college graduates from up to almost 20 years ago. This is literally not at all a study of college graduates of the last five years, or even ten years. We’re talking about people up to the age of 40, well into Gen X.

This is a strange statement. As Sicha acknowledges, the people sampled in the study are from ages 20-40. That certainly includes many people who have graduated in the last five to ten years. It also includes people up the the age of 40, but so what? Why is that disqualifying? The study admits that up front– it’s right there in the methods. And if you’re making the case that aggregating 20 year old debt holders with 40 year old debt holders necessarily invalidates your sample, you have to say why. The notion that there’s some dramatic difference between those who graduated within the last five years and those who graduated fifteen years ago is an empirical point that has to be proved. And it’s not as if tuitions have suddenly exploded; they’ve been rising, at a totally indefensible rate, for decades. (They have actually slowed their growth, but that’s cold comfort.) People who assert that the student loan debt issue is bigger rather than smaller have a burden of proof, as well.

Worse still:

And finally… this survey is, essentially, of rich people. No, literally

We apply survey weights throughout the analysis so that the results are representative of the U.S. population of households. The use of survey weights is particularly important in the SCF because the sample design oversamples high-income households to properly measure the full distribution of wealth and assets in the United States. This high-income sample makes up approximately 25 percent of households in the SCF.

Literally what they are saying there is that the information on which they are basing a sweeping assessment of American student loan debt is based on a sample in which 25% of those surveyed were “high-income households.” This is insane.

This is the problem with reading methods sections without having an adequate basis in research methodology. As commenters other than me point out in the comments, this practice is absolutely common, and quite responsible. Given an adequate sample, it’s not at all hard to adjust this kind of imbalance quantitatively. Sicha calls a standard, simple practice “insane,” which demonstrates the degree to which he fails to moderate his claims. I’m afraid he doesn’t know what he’s talking about.

Sicha’s basic problem is a very common one, which it’s understandable but unfortunate: overestimating sampling standards necessary to make responsible generalizations with statistical data. I get it. I remember when my stats prof told me, last fall, that in certain situations an n of 30 is adequate to make broad generalizations. I was flabbergasted. But when you pay attention to the processes of generating confidence intervals and standard errors, and when you understand the law of large numbers and the central limit theorem, it makes more sense. I understand why Sicha is put out by the head-of-household distinction, and I agree that you could view that as a limitation. But people asserting that this necessarily invalidates the data are going too far, and they themselves face a burden of proof in establishing that the heads of households are necessarily different enough from the broader population of debt holders to invalidate the sample.

If you’d like a sober, data-supported argument about the size of the debt crisis, I recommend this infographic by Derek Thompson of The Atlantic. It includes some data from Brookings– but also from the NCES, the NBER, the BLS, Harvard, UChicago…. Maybe they’re all in a conspiracy to underestimate this problem! But I doubt it. I instead think that the student loan crisis is indeed a crisis, a moral and practical problem of considerable size. But it’s not the size that most people think it is. And more, it doesn’t change this fact: that despite the endless concern trolling of almost our entire media, the constant tendency for the (college educated) professional writers in our culture to say that “college isn’t worth it,” a college education remains a very good investment for the large majority of graduates. American college graduates are, by essentially any international standards and in comparison to Americans with only a high school degree, in a very economically secure class. There are tons of exceptions; that’s how averages and aggregates work. Those who are among the exceptions deserve not just sympathy but structural change to improve their lives. I’ll keep saying: there is no reason for the federal government not to engage in broad forgiveness of the vast amounts of loan debt it controls. In a bigger perspective, a universal basic income could solve an enormous number of our social problems. Most certainly including those of recent college graduates.

But I have feeling that this kind of support will not be sufficient. By criticizing the louder and less careful claims about this problem, I’m necessarily going to be placed in the ranks of the bad guys. But that’s OK. I just think being careful matters, and the truth matters, and that solving this kind of problem requires having higher evidentiary standards, rather than lower.

This post was originally published on fredrikdeboer.com.

Freddie deBoer is a doctoral candidate in rhetoric and composition at Purdue University.

Photo: Shutterstock/Burlingham