[M T]

I take it that concerns were that it could have been much worse, but still, it is clear that the pandemic is affecting learning.

It isn't clear to me how much of any change in scores is due to just a change in the situation.  For instance, for all I know, if you added a second teacher to all classes, or you changed the schedule (longer days, for instance), the same sort of effect might be seen.

And, speaking of math issues, did you notice what appears to be a math error?

In an effort to be sure their 4.4 million-student sample, albeit large, was also representative of America's classrooms, NWEA researchers dug into the demographics of this new data set and compared it with the earlier fall 2019 test data — a sample of nearly 5.2 million children.

What they found, Tarasawa says, is that roughly a quarter of students were missing — meaning they didn't take the MAP test this fall

(5.2M - 4.4M)/ 5.2M = 15.4%, not 25%

I wonder whether Tarasawa was working from home....  Something doesn't add up.

Some students (last year's 8th graders) have aged out of this set of tests as others (this year's 3rd graders) have aged into them, but I'd expect those classes to be about the same size.  If you assume all 6 classes (3rd - 8th grade) are the same size, about 5.2M / 6 = 866,667 were in 8th grade in 2019 and wouldn't be taking the 2020 test.  So, there were 5.2M - 0.87M = 4.33M test takers in 2019 that would be expected to take the test again.  For 1/4 of  the 4.33M to be missing, that would mean that 4.33M/4 = 1.08M are missing. So this year, there were only 4.33M-1.08M = 3.25M test takers that took the test last year.   That would mean that 4.4M - 3.25M = 1.15M new test takers (2020 3rd graders).  That is 33% more than expected (0.867M).  Something doesn't add up.

Whatever the number missing, is this just a measure of how many students are being homeschooled rather than enrolled in a school and doing remote learning?  Or is it a measure of truancy?   I'd have thought the authors would have tried to determine that.

If I were those researchers, I would have extrapolated to a hypothesized 2020 score by adding in scores for those missing students based on their scores in 2019 but a year more advanced.   If, as they seem to suggest, the missing students are predominantly disadvantaged students, one might wonder whether they are also less proficient in (say) reading (esp. if they are from households that speak a different language that was tested).   If you remove more students with lower scores than you remove with higher scores, you will raise the average.  This might explain why the 2020 scores were not as bad as feared.  (Or, maybe they did think of this, tried it, and found no real reason to present it)

On the other hand, maybe more parents or even grandparents spent more time with the students during the pandemic.  Maybe that could have resulted in better learning by many students, thus raising scores.  Or, maybe despite the changes in circumstances, remote learning was more effective.  Maybe parents know their kids better than their teachers do, or pushed for homework to be done because they were more involved in the education process.   Or, maybe bored kids spent more time reading.  Or, maybe time previously spent in (or going to/from) school or other activities that didn't happen (for example, playing baseball, or watching sports) was better applied to their studies at home.

There are lots of possibilities.  Probably all of them contributed to the results in varying degrees.   We could argue left and right about them but quite possibly with no real result.

[M T]

I take it that concerns were that it could have been much worse, but still, it is clear that the pandemic is affecting learning.

It isn't clear to me how much of any change in scores is due to just a change in the situation.  For instance, for all I know, if you added a second teacher to all classes, or you changed the schedule (longer days, for instance), the same sort of effect might be seen.

And, speaking of math issues, did you notice what appears to be a math error?

In an effort to be sure their 4.4 million-student sample, albeit large, was also representative of America's classrooms, NWEA researchers dug into the demographics of this new data set and compared it with the earlier fall 2019 test data — a sample of nearly 5.2 million children.

What they found, Tarasawa says, is that roughly a quarter of students were missing — meaning they didn't take the MAP test this fall

(5.2M - 4.4M)/ 5.2M = 15.4%, not 25%

I wonder whether Tarasawa was working from home....  Something doesn't add up.

Some students (last year's 8th graders) have aged out of this set of tests as others (this year's 3rd graders) have aged into them, but I'd expect those classes to be about the same size.  If you assume all 6 classes (3rd - 8th grade) are the same size, about 5.2M / 6 = 866,667 were in 8th grade in 2019 and wouldn't be taking the 2020 test.  So, there were 5.2M - 0.87M = 4.33M test takers in 2019 that would be expected to take the test again.  For 1/4 of  the 4.33M to be missing, that would mean that 4.33M/4 = 1.08M are missing. So this year, there were only 4.33M-1.08M = 3.25M test takers that took the test last year.   That would mean that 4.4M - 3.25M = 1.15M new test takers (2020 3rd graders).  That is 33% more than expected (0.867M).  Something doesn't add up.

Whatever the number missing, is this just a measure of how many students are being homeschooled rather than enrolled in a school and doing remote learning?  Or is it a measure of truancy?   I'd have thought the authors would have tried to determine that.

If I were those researchers, I would have extrapolated to a hypothesized 2020 score by adding in scores for those missing students based on their scores in 2019 but a year more advanced.   If, as they seem to suggest, the missing students are predominantly disadvantaged students, one might wonder whether they are also less proficient in (say) reading (esp. if they are from households that speak a different language that was tested).   If you remove more students with lower scores than you remove with higher scores, you will raise the average.  This might explain why the 2020 scores were not as bad as feared.  (Or, maybe they did think of this, tried it, and found no real reason to present it)

On the other hand, maybe more parents or even grandparents spent more time with the students during the pandemic.  Maybe that could have resulted in better learning by many students, thus raising scores.  Or, maybe despite the changes in circumstances, remote learning was more effective.  Maybe parents know their kids better than their teachers do, or pushed for homework to be done because they were more involved in the education process.   Or, maybe bored kids spent more time reading.  Or, maybe time previously spent in (or going to/from) school or other activities that didn't happen (for example, playing baseball, or watching sports) was better applied to their studies at home.

There are lots of possibilities.  Probably all of them contributed to the results in varying degrees.   We could argue left and right about them but quite possibly with no real result.