No, The Brookings Study On Student Loan Debt Isn’t ‘Garbage’

Start your day with TPM.
Sign up for the Morning Memo newsletter

It’s just research like all research: conducted by humans, with limitations and caveats, but also deserving of a respectful, accurate reading.

There’s way too much student loan debt in this country. (I hold some of it myself.) I’ve written about it at length, and many times, over the course of years. I’ve written about ways to reduce the cost of college, about structural changes to the student loan system, about the desperate need for state governments to reinvest in higher education, and about a potential system of no-frills, low-or-no tuition federal universities that could graduate thousands without loan debt. It’s a big economic drag to have so many young people pouring money into student loans instead of into houses and cars, and it’s a moral travesty for so many people to suffer because they can’t afford to pay for the education they already received.


It’s also the case that people who write and argue about the student loan debt crisis frequently speak about the problem irresponsibly. People throw around fake figures all the time, propose all sorts of doomsday scenarios, and generally represent the problem in ways that are not supportable from data. I can’t tell you how often I’ve had people talk about “every graduate running around with six figure debt!,” when six figure debt is extremely rare. Or when people compare the size of the problem to that of the housing bubble, which is simply not correct. The housing bubble was orders of magnitude biggerthan the student loan crisis. Pointing these things out is not a matter of disagreeing that there’s a problem, and it’s not a matter of being callous to those struggling under student loan debt. It’s a matter of being responsible in talking about a problem, under the theory that this is the only way to solve such problems.

So I’m seeing a lot of breathless sharing of this piece by Choire Sicha, attacking the methodology of a Brookings Institution report showing that student loan debt concerns are somewhat overblown. Like all studies, there are issues with the methodology of this one. But Sicha does not do a great job of critiquing this study. I don’t blame him– methodological critiques are really hard and take a lot of work. I’ve taken something like 9 graduate courses in research methodology and statistics, and I still find confronting methods and methodology an imposing task. But there’s lots that’s off, here. And Sicha isn’t mincing words: he’s saying the study is garbage, and people are listening to him. That’s not supportable, at least not from the evidence that he provides, and it’s unclear if he’s angry about the methods or about the conclusion.

Of all the households in that study, only about 1711 have “household heads” that are younger than 40. That’s what they’re extrapolating from. (And, intriguingly, a small number of those have a head of household younger than 18.) This is not a big sample!”

This is something I’ve written about before – people dramatically overestimate the sample size needed to make responsible statistical conclusions. A sample size of almost 2,000 isn’t just big, it’s enormous. The standard error of a sample of this size will be very low. Absent systematic sampling bias (as opposed to error), the odds of the underlying population being significantly different from a sample of this size is tiny. Saying that it’s not a big sample just displays ignorance about the standards applied in statistical research.

One effect of this age spread sample is that it includes college graduates from up to almost 20 years ago. This is literally not at all a study of college graduates of the last five years, or even ten years. We’re talking about people up to the age of 40, well into Gen X.

This is a strange statement. As Sicha acknowledges, the people sampled in the study are from ages 20-40. That certainly includes many people who have graduated in the last five to ten years. It also includes people up the the age of 40, but so what? Why is that disqualifying? The study admits that up front– it’s right there in the methods. And if you’re making the case that aggregating 20 year old debt holders with 40 year old debt holders necessarily invalidates your sample, you have to say why. The notion that there’s some dramatic difference between those who graduated within the last five years and those who graduated fifteen years ago is an empirical point that has to be proved. And it’s not as if tuitions have suddenly exploded; they’ve been rising, at a totally indefensible rate, for decades. (They have actually slowed their growth, but that’s cold comfort.) People who assert that the student loan debt issue is bigger rather than smaller have a burden of proof, as well.

Worse still:

And finally… this survey is, essentially, of rich people. No, literally

We apply survey weights throughout the analysis so that the results are representative of the U.S. population of households. The use of survey weights is particularly important in the SCF because the sample design oversamples high-income households to properly measure the full distribution of wealth and assets in the United States. This high-income sample makes up approximately 25 percent of households in the SCF.

Literally what they are saying there is that the information on which they are basing a sweeping assessment of American student loan debt is based on a sample in which 25% of those surveyed were “high-income households.” This is insane.

This is the problem with reading methods sections without having an adequate basis in research methodology. As commenters other than me point out in the comments, this practice is absolutely common, and quite responsible. Given an adequate sample, it’s not at all hard to adjust this kind of imbalance quantitatively. Sicha calls a standard, simple practice “insane,” which demonstrates the degree to which he fails to moderate his claims. I’m afraid he doesn’t know what he’s talking about.

Sicha’s basic problem is a very common one, which it’s understandable but unfortunate: overestimating sampling standards necessary to make responsible generalizations with statistical data. I get it. I remember when my stats prof told me, last fall, that in certain situations an n of 30 is adequate to make broad generalizations. I was flabbergasted. But when you pay attention to the processes of generating confidence intervals and standard errors, and when you understand the law of large numbers and the central limit theorem, it makes more sense. I understand why Sicha is put out by the head-of-household distinction, and I agree that you could view that as a limitation. But people asserting that this necessarily invalidates the data are going too far, and they themselves face a burden of proof in establishing that the heads of households are necessarily different enough from the broader population of debt holders to invalidate the sample.

If you’d like a sober, data-supported argument about the size of the debt crisis, I recommend this infographic by Derek Thompson of The Atlantic. It includes some data from Brookings– but also from the NCES, the NBER, the BLS, Harvard, UChicago…. Maybe they’re all in a conspiracy to underestimate this problem! But I doubt it. I instead think that the student loan crisis is indeed a crisis, a moral and practical problem of considerable size. But it’s not the size that most people think it is. And more, it doesn’t change this fact: that despite the endless concern trolling of almost our entire media, the constant tendency for the (college educated) professional writers in our culture to say that “college isn’t worth it,” a college education remains a very good investment for the large majority of graduates. American college graduates are, by essentially any international standards and in comparison to Americans with only a high school degree, in a very economically secure class. There are tons of exceptions; that’s how averages and aggregates work. Those who are among the exceptions deserve not just sympathy but structural change to improve their lives. I’ll keep saying: there is no reason for the federal government not to engage in broad forgiveness of the vast amounts of loan debt it controls. In a bigger perspective, a universal basic income could solve an enormous number of our social problems. Most certainly including those of recent college graduates.

But I have feeling that this kind of support will not be sufficient. By criticizing the louder and less careful claims about this problem, I’m necessarily going to be placed in the ranks of the bad guys. But that’s OK. I just think being careful matters, and the truth matters, and that solving this kind of problem requires having higher evidentiary standards, rather than lower.

This post was originally published on

Freddie deBoer is a doctoral candidate in rhetoric and composition at Purdue University.

Photo: Shutterstock/Burlingham

Latest Cafe

Notable Replies

  1. Brookings left out the parents who took out parent-student loans to pay for their children’s college tuition! Why was that segment omitted from the study on student loan debt?

  2. Very odd stock photo choice on this article.

  3. You mischaracterized the biggest problem Sicha has with the Brookings report: He wasn’t objecting to the sampling being people from ages 20-40.

    His problem was the sample only included households in which the head of household is aged 20-40. … so this pretty much excludes all households in which the post-college kids live at their parents’ home… which I’m guessing is a huge chunk of those with $50,000+ in student debt.

    The study also excluded all debtors who’ve fallen behind on their payments…which is something like 1 in 3 with student-load debt.

  4. Avatar for paulw paulw says:

    Gotta say, that Atlantic chart seems kinda crappy to me. It doesn’t say, for example, what fraction of people with which debt levels actually graduated (which would be crucial to assessing the value of their degree), nor does it (apparently) associate any value with the time of young people deciding to get a degree. Finally, by assessing weekly earnings for everybody 25 and older by educational level, it may do a good job on saying whether college was a on average a good investment between 1969 and 2009, but it doesn’t say bupkus about whether it’s a good investment today.

  5. Let’s say you have 700 people, and you ask each “what’s the size of your student loan?” And the average is $30,000. So, you ask one more person: how much could their answer change the total?

    Even if they had $150,000 in loans (5 times the average), they would move the total by only 1/2 of 1% to $30,170.

    That’s why it works. At a certain point, another answer can’t move the total except for a wee bit. Factors that are limited (say, years of education) don’t go even into the dozens so no one can skew an average by somehow having 85 years when the other 30 have 14.

Continue the discussion at

3 more replies


Avatar for system1 Avatar for paulw Avatar for jootjoint Avatar for leofromchicago Avatar for harry_r_sohl Avatar for swift2001 Avatar for robcat2075 Avatar for salvomania Avatar for MinNJ2014

Continue Discussion
Masthead Masthead
Founder & Editor-in-Chief:
Executive Editor:
Managing Editor:
Associate Editor:
Editor at Large:
General Counsel:
Head of Product:
Director of Technology:
Associate Publisher:
Front End Developer:
Senior Designer: