[2006alum]

Here's a version of this story that's not behind a paywall.

There's one aspect of this study's methodology I found surprising, and which I think could be potentially problematic:

At each college, the roughly 90-minute test is given to one group of freshmen in the fall of their freshman year and to a separate group of seniors in the spring of the same academic year. A statistical analysis of the difference between the average scores of the two groups is considered a valid way to reflect the value added during four years of college.

The Journal filed public-records requests with more than 100 public institutions where students took the CLA+ between 2013 and 2016.

What's the problem? College admissions has gotten notoriously more competitive year over year in recent years, as we all know well from Stanford's falling admissions rate, particularly at the nation's most selective colleges. This means that if the number of spots at Stanford remains constant, but the number of qualified applicants increases, the average Stanford admit in in the class of x+3 is likely to be more qualified than the average admit in the class of x, all other things equal. If so, then the average class of x+3 freshman is likely to perform better on this test than the average class of x freshman. It also means that if colleges improve critical reasoning ability at a rate that is roughly comparable to the year-over-year difference in improvement to the applicant pool, you will see absolutely no difference in testing outcomes of class of x seniors and class of x+3 freshmen despite the college actually providing instruction that improves the critical reasoning of students during their time in college.

That isn't to say that colleges can't do a lot more to enhance their students' critical thinking skills. It does, however, suggest that studies that don't circle back to the same students or control for exogenous effects in the pool of students tested over time should be taken with a grain of salt. If applicant pools and admissions rates were staying constant, I would be less skeptical, but the last few years have seen large year-over-year growth in number of applicants, and so I am skeptical that this study is making true apples-to-apples comparisons.

Thus, it also doesn't surprise me that a conclusion drawn from this study was:

Some of the biggest gains occur at smaller colleges where students are less accomplished at arrival but soak up a rigorous, interdisciplinary curriculum.

The less accomplished colleges probably see negligible year-over-year improvement in applicant and matriculant populations. If so, then their sample populations at year x and year x+3 are closer to apples-to-apples.

Sorry to harp. Once a social scientist, always a social scientist...

[2006alum]

Here's a version of this story that's not behind a paywall.

There's one aspect of this study's methodology I found surprising, and which I think could be potentially problematic:

At each college, the roughly 90-minute test is given to one group of freshmen in the fall of their freshman year and to a separate group of seniors in the spring of the same academic year. A statistical analysis of the difference between the average scores of the two groups is considered a valid way to reflect the value added during four years of college.

The Journal filed public-records requests with more than 100 public institutions where students took the CLA+ between 2013 and 2016.

What's the problem? College admissions has gotten notoriously more competitive year over year in recent years, as we all know well from Stanford's falling admissions rate, particularly at the nation's most selective colleges. This means that if the number of spots at Stanford remains constant, but the number of qualified applicants increases, the average Stanford admit in in the class of x+3 is likely to be more qualified than the average admit in the class of x, all other things equal. If so, then the average class of x+3 freshman is likely to perform better on this test than the average class of x freshman. It also means that if colleges improve critical reasoning ability at a rate that is roughly comparable to the year-over-year difference in improvement to the applicant pool, you will see absolutely no difference in testing outcomes of class of x seniors and class of x+3 freshmen despite the college actually providing instruction that improves the critical reasoning of students during their time in college.

That isn't to say that colleges can't do a lot more to enhance their students' critical thinking skills. It does, however, suggest that studies that don't circle back to the same students or control for exogenous effects in the pool of students tested over time should be taken with a grain of salt. If applicant pools and admissions rates were staying constant, I would be less skeptical, but the last few years have seen large year-over-year growth in number of applicants, and so I am skeptical that this study is making true apples-to-apples comparisons.

Thus, it also doesn't surprise me that a conclusion drawn from this study was:

Some of the biggest gains occur at smaller colleges where students are less accomplished at arrival but soak up a rigorous, interdisciplinary curriculum.

The less accomplished colleges probably see negligible year-over-year improvement in applicant and matriculant populations. If so, then their sample populations at year x and year x+3 are closer to apples-to-apples.

Sorry to harp. Once a social scientist, always a social scientist...