Excuses, Excuses . . .

WaPo has a piece this morning on the Establishment’s reaction to Brian Davison’s suit, Davison v. Virginia Ed. Dep’t, No. CL14004321-00 (Richmond Cir. Ct., Petition for Mandamus, October 2, 2014), to require VDOE to release SGP data.

Two paragraphs in the WaPo piece capture most of the anti arguments:

Growth percentiles cannot accurately measure growth among the highest- and lowest-performing children, officials say, and they warn that in some cases student scores might be erroneously assigned to teachers who never actually taught them. In addition, they rely on consecutive years of test data that might not be widely available in schools serving transient populations.

And unlike value-added models used by other states, Virginia’s model does not attempt to control for the effects of poverty or other demographic characteristics. Critics of the growth percentiles say that disadvantages teachers who work with the neediest children.

Let’s take those one at a time:

Highest- and Lowest-Performers

Students who score “passing, advanced” two years in a row are not included in the SGP calculation.  That’s because they are doing just fine, thank you, and have precious little room to do better.  The SGP measures improvement.

As well, among the kids who are tested, the statistics have a problem at the extremes.  For example, on the 2014 8th grade reading results, we see the typical spikes at the 1% and 99% levels:

image

No teacher can complain about having a few extra 99s in the SGP average.  And, just maybe, that teacher is doing an outstanding job. 

Any teacher with a relatively large number of 1s is dealing with a tough group (so long as the rest of that teacher’s distribution is reasonable).  So the data will tell us to discount the average and look directly at the data (as we should be doing anyhow). 

What is the problem?

Students Counted for Wrong Teacher

The anonymous “officials” complain that the SGP might be inaccurate because students scores could be erroneously assigned to teachers who never taught those students.

This is not a criticism of the SGP.  This is a criticism of the local school officials who keep lousy records, and of VDOE, which lets them get away with it.

Transient Populations

The same anonymous “officials” complain that no SGPs are available for transient populations.  This hardly is a reason to suppress the SGP as a measure of progress of the non-transient students.

Aside: National “Teacher Quality” Group

WaPo also quotes the National Council on Teacher Quality for the propositions that the SGP is “not 100% accurate” and is a “real invasion of privacy.”

What examination, pray, is “100% accurate?”  None.  This is just another attempt to turn the unachievable perfect into the enemy of the good.

As to privacy, it’s easy to understand how ineffective teachers, and their masters, would prefer to keep evidence of their performances secret.  But these are public employees, paid with tax money to educate our society’s children.  How can their preference for privacy outweigh the public’s interest in knowing the effectiveness of our teachers, schools, and school divisions?

2d Aside: Data Could be Misinterpreted

We also have a Loudoun School Board member complaining that the SGP data “could be misinterpreted” if released.  What that official is not saying is that the public is more easily fooled if the School Board keeps that public uninformed.  And she is admitting that her School Board and the State Board of Education are unwilling or unable to educate the public in the proper use of this important tool.

Effect of Poverty

Aside from privacy, the arguments above are merely misleading attempts to say that the SGP is not perfect, so it must be abandoned entirely.  The “poor kids do worse” argument is a lie.

The SOL scores correlate fairly well with poverty.  VDOE uses the term “economic disadvantage” and their data show a clear relationship (here on the reading SOLs for 2014):

image

BTW: Richmond is the gold square on that graph: We have an unusually large percentage of economically disadvantaged students and even more unusually low reading scores.  The red diamonds are Hampton, Norfolk, and Newport News; the green is Charles City.

The SGP process, in contrast, compares progress among similarly situated students.  It produces scores that are largely independent of economic status.  VDOE itself quotes the Colorado data:

image

Indeed, VDOE has contradicted this lie:

A student growth percentile complements a student’s SOL scaled score and gives his or her teacher, parents and principal a more complete picture of achievement and progress. A high growth percentile is an indicator of effective instruction, regardless of a student’s scaled score.

In short, it is SOLs that provide a biased standard for comparison of teachers, schools, and divisions.  They compare scores of needy children to their affluent peers. The SGP removes that bias and gives a measure of how much a teacher, school, or division is teaching its students, be they rich or poor.

Yet, VDOE, the Teachers Ass’n., and the Loudoun School Board all want to keep these important data secret.  What do you suppose they are trying to hide?

SGP V – Bad Data?

Turning back to the question of the appalling performance of Richmond’s middle schools, especially in the sixth grade, let’s take a look at the SGP average by teacher by year.  Here, for a start, are the fifth grade data, first for reading and then for math:

image

image

These graphs make a couple of points:

  • In 41% of the cases, a teacher taught the subject for only one year year of the three.  For the most part, however, the teachers in both subjects taught the same grade and subject for either two years (21% and 22% for reading and math, respectively) or all three years (38% and 37%).
  • The dip in the reading SGPs in 2013 corresponds to the new reading tests that year and Richmond’s failure to align its curriculum.  Otherwise, the year-to-year changes give a quick idea of the variability per teacher.  To put a number on that, the average standard deviation of the annual SGPs is 11 for both sets of data; doubtless that number would be smaller if Richmond had not failed to prepare for the new reading test in 2013.

The sixth grade data show an entirely different pattern.  Again, first for reading and then math:

image

image

First, of course, the SGPs are much lower than the fifth grade numbers.

Remarkably, only one teacher (#66291) taught one subject (math) at the 6th grade level for two years in a row.  At least that’s what this database says.  Thirty-seven teachers taught reading only one of the three years while five taught two of the three years.  For math, those numbers are twenty-four and six.  As the graphs show, none taught for all three.

The small number of teachers reported in 2013 looks strange.  It is strange.  See below.

The seventh grade patterns are similar, albeit the scores are a bit higher and we see a few more teachers reported to be repeating the same subject.  Again, reading and then math:

image

image

Finally, the eighth grade scores, again reading first:

image

image

Aside from the remarkable teacher turnover implied by these data, the number of Grade 6 to 8 reading and math teachers is unbelievably low in 2013 except, perhaps, for the 8th grade reading.  As well, the counts of SOL and SGP results in the VDOE SGP database for grades 6-8 in 2013 look to be anomalous:

image

image

We have nowhere to go to test the number of SGP reports.  In contrast, the (very nice) VDOE SOL database gives the participation counts for the same years and grades.  Here, then are the Richmond numbers of SOL results in the SGP database, expressed as percentages of the participation numbers reported for the same tests.  First, reading:

image

Then math:

image

Let’s first deal with the ridiculous datum: The “#DIV/0!” for 2014 6th grade math is there because VDOE reported zero participation in that test:

image

Looking beyond that obvious error: VDOE attached a sheet to the earlier download, admitting that the SGP database suppressed data for classes fewer than ten students and for students who transferred from one Virginia school to another during a school year.  They provided counts for the suppressed data by test, ranging from 1.6% to 7.3%, with an average of 4%.

Except for the 2012 Grade 6 math datum, where the SGP database reports SOL scores for 106% of the students who took the test, the numbers here are much lower than even a 7% suppression (low in elementary school, very low in middle school).  They are ridiculously low for grades 6-8 in 2013.

Doubtless that explains the remarkable teacher turnover reported for the SGP in grades 6-8 for 2013: Leave out the test data and the reports by teacher also will be left out.

We’ll have to try to get a correct SGP database and see how that affects the SGP analysis here and earlier.

Sigh.

SGP IV – Rating the Teachers: Math

We have seen that the new SGP dataset shows a precipitous drop in the quality of instruction in math in the sixth grade in the Richmond public schools.

http://calaf.org/wp-content/uploads/2015/03/image14.png

The by-teacher averages add some context to that situation.  Let’s start with the fifth grade math distribution for the state:

image

And for the City:

image

The Richmond average is slightly below the state average.  In both cases, the fairly large standard deviation reflects the relatively large numbers of very good and very poor performers.

Next the sixth grade math data, starting with the state distribution:

image

Then Richmond:

image

(Note that, with this relatively small dataset, the average SGP by student, 24.2, is not the same, and is not necessarily the same, as the average by teacher, 20.9)

The Richmond average is 1.75 standard deviations below the state average.  Four of eighteen Richmond math teachers are more than two standard deviations below the state average.  Only one is above the state average.

The 7th grade math results are less awful but still unacceptable: 

image

The eighth grade math results are below average, but less horrible than the data for the previous two grades:

image

Of course, we’ve long known that the SOL scores of Richmond’s middle schools were a disaster. 

image_thumb9

The SOL scores (data again from 2014) decrease with increasing economic disadvantage. 

image

(Richmond is the gold square.)

In contrast, the SGP scores measure improvement compared to similarly situated students and are largely independent of family wealth.  The Colorado data are widely quoted on this point:

image_thumb4 image_thumb5

The SGP data confound the traditional excuse that too many of Richmond’s students are from economically disadvantaged homes.  Thus, Richmond no longer can blame the students for the miserable performance of its schools, especially the middle schools.  The problem is (some of) the teachers, (most of) the schools, and the administration, not the students. 

Rant Begins:

Offsetting that important progress in evaluating the schools, VDOE is refusing to release the identities of the under- (AND over-) performing teachers.  They claim that those records of academic improvement (or lack thereof) in particular classrooms in public schools, financed by public money, are “personnel records,” never mind that the personnel in question are not VDOE employees. 

Whether these data as to particular teachers are “personnel records” under the Freedom of Information Act is a question for the Judge.  In any case, VDOE has the discretion to disclose the records. 

At present, parents take their kids’ teachers willy nilly.  VDOE now has data in some cases to tell those parents whether the teachers are effective.  Yet VDOE thinks the “privacy” of those public employees is more important than informing the public about those employees’ performance.  VDOE’s refusal to share those important data that have been bought with taxpayer dollars is an abiding and outrageous insult to Virginia’s taxpayers.

Until someone comes up with a catchy acronym, I will call the gang of secretive bureaucrats at VDOE the State Department of Data Suppression.  The name may not be catchy but it surely is accurate. 

(And the abbreviation, SDDS, is a palindrome!)

Your tax dollars at “work.”

SGP III – Rating the Teachers: Reading

The most recent VDOE database of Student Growth Percentiles contains (anonymous) teacher IDs.  This gives us a first peek at how well, and how badly, some of Richmond’s teachers are performing.

With all the earlier caveats, let’s start with the statewide distributions of teachers’ average SGP scores in reading and math.

image

Brian Davison points out that both distributions are reasonably symmetrical, suggesting that we do not have an unusually large number of teachers doing particularly well or poorly.  That said, no parent will want a child to be subjected to the reading teacher in the first percentile, the other teacher in the second, or the three in the eighth.

The math scores are more widely distributed, showing a larger number of excellent and a larger number of awful teachers. 

image

Aside from targeting the lowest performers in both subjects, these data suggest that we need math retraining more than reading.

Turning to the data by grade, here is the distribution of fifth grade reading averages.

image

The orange curve is a normal distribution, showing the least squares fit.  The average and standard deviation of the fitted curve are shown at the base of the graph.

The distribution of sixth grade reading teachers is close to the same.

image

We already have seen that the Richmond average reading SGP plunges from fifth to sixth grades.

image

The Richmond distributions conform to that pattern.  First, grade 5:

image

(Note that this average by teacher is slightly less than the average by student, above.)

As you see, this distribution is a bit wider than the statewide distribution.  That is, Richmond has relatively more excellent fifth grade reading teachers than the statewide average, and also relatively more who are not performing.  Five (of sixty-seven) Richmond teachers are more than two standard deviations above the state average; three are more than two standard deviations below.

Those teachers at the low end need some work but, for the most part, Richmond’s fifth graders are in pretty good hands as to reading.

Then we have grade 6 reading results:

image

Only one of Richmond’s twenty-one sixth grade reading teachers produced an average student improvement better than the state average; none was more than two standard deviations above the statewide average.  Six (or seven, depending on the rounding) were more than two standard deviations below the state average and four were more than three standard deviations below.  The Richmond average is 1.5 standard deviations below the state average.

These data tell us that Richmond’s sixth grade reading teachers are not doing a bad job.  They are doing an appalling job.

Upon some reflection, the data also tell us two even more important things:

  • The principals (and the Superintendent) now have a quantitative measure of teacher performance (at least as to reading and math).  If they don’t do something (soon!) about rewarding the excellent performers and retraining or firing the poor ones, we’ll know they need to be replaced themselves.
  • VDOE is hiding the identities of these high- and low-performing teachers from the parents who pay them and the teachers and whose kids are directly affected by teacher performance.  Apparently those bureaucrats think it would be intrusive for the parents of Virginia’s schoolchildren to know whether their kids are in the hands or excellent, average, or lousy teachers.  I think the term for that kind of inexcusable bureaucratic arrogance is “malfeasance.”

Tomorrow, the Math situation.  (Hint: It’s even worse.)

New SGP Data II – Math by Division

Continuing the analysis of the third set of SGP data from VDOE (with the same caveats mentioned earlier), here are the 2014 division average math and Algebra I scores.

As before, Richmond is the yellow bar, Petersburg is red, Hampton is green, and Norfolk is blue.

Grade 4 math:

image

Grade 5 math:

image

As with the reading scores, Richmond is doing fairly well with math in grades 4 and 5.  Then then comes Grade 6 math where Richmond drops to next-to-last place:

image

Grade 7 math:

image

Grade 8 math:

image

Algebra I:

image

As with the reading scores, the Richmond math scores plummet when the students enter middle school.  Yet the state averages remain nearly flat (as they nearly should; if VDOE were not manipulating the data, the state average should be entirely flat at 50 on every test).

image

image

Something uniquely ugly happens in the sixth grade in Richmond.

I asked the formidable Carol Wolf:

What’s going on with the 6th grade?  Richmond’s [SGP] scores in both reading and math fall into a pit from fifth to 6th grade.  I’m looking at data that show that NO Richmond teacher for whom we have SGP data taught 6th grade reading two years in a row; ONE Richmond teacher for whom we have SGP data taught 6th grade math two years in a row.  Nobody taught either subject three years in a row.

Any ideas?

She replied:

I have asked several teachers what and why they think it is that Richmond’s 6th and 7th graders go from a fully accredited elementary school to being dumber than a sack of rocks when they hit middle school.  

Their collective answer:  The elementary schools are cheating.

Could be.  The 8th Grade SGPs (which are below but approaching state average values) are based entirely on the change from previous years’ middle school SOL scores, while the 7th Grade SGP scores can reach one year into elementary school and the 6th grade SGP scores are based entirely on the change from students’ SOL histories in elementary school.  If Richmond’s elementary SOL scores were artificially high and the middle school SOLs were low normal, the 6th graders and, to a lesser degree the 7th graders, would be starting at an artificially high SOL, so their SGP scores would show abnormally little improvement.  That is, the SGP scores would be abnormally low in the sixth and, to a lesser degree, seventh grades.

The 6th Grade teach-once-then-teach-something-else pattern would suggest that the new teachers get the sixth grade classes and that they get out as soon as they get any seniority.  That would be consistent with unusually low sixth grade SGP scores, whether the elementary SOLs were inflated or not.

Let’s label Carol’s suggestion as an hypothesis and try to think of an experiment to falsify it.

In any event, with the prophylactic effect of Richmond’s appalling dropout rate (and, probably, Richmond’s remarkable retest rate), the scores rebound for Algebra I.

image

 

Coming Attraction: SGP distributions by teacher.

New SGP Data–Reading by Division

VDOE has provided Brian Davison a third set of SGP data.  This set includes (anonymized) teacher IDs.

As a reminder, in the words of VDOE:

A student growth percentile expresses how much progress a student has made relative to the progress of students whose achievement was similar on previous assessments.

A student growth percentile complements a student’s SOL scaled score and gives his or her teacher, parents and principal a more complete picture of achievement and progress. A high growth percentile is an indicator of effective instruction, regardless of a student’s scaled score.

That last point is particularly important: The SOL is significantly affected by the student’s economic status; poorer students generally get lower scores.  The SGP, in contrast, measures improvement relative to similarly situated students; the SGP ranking is pretty much independent of economic status.  Thus, the SGP gives us a measure of quality that allows comparisons from division to division, school to school, and teacher to teacher.

CAVEATS:

  • VDOE has manipulated these data.  For example, they have suppressed the data in cases where the number of students (presumably in a class) is fewer than ten.  In contrast to earlier data sets, they have suppressed the no-retest scores for students showing a retest score.  They also have suppressed retest scores below passing (<400).   Those are the changes I know about or have found; there may well be others.   In any event, these data deviate from the expected mean and median of 50, probably because of the VDOE manipulations.  For example, the average of the 2014 sixth grade reading SGPs in this dataset is 47.7 and the distribution is not the expected flat line:

image

  • VDOE gave us anonymized teacher IDs, but not the names of the teachers involved.  More on this outrage in a later post.
  • The SGP is calculated from the difference between similarly sized SOL scores, which can lead to large relative errors in the SGP.  This suggests considerable caution in interpreting any individual student’s SGP score or the average (or distribution) for a teacher where the number of scores is small.

That said, here is the distribution of 2014 division average Reading SGPs.

image

On this graph, and those below, Richmond is the yellow bar, Petersburg is red, Norfolk is blue, and Hampton is green.

Excel is unable to readably list all the divisions on this graph; the list here and below excludes every 2d division.  Thus, on the graph below, the blue bar (Norfolk) appears between Surry and Essex Counties.  That’s just an artifact of the program, not evidence that Norfolk has disappeared.

The graph above averages the results of reading tests in the five grades, 4-8 (the SOL score in Grade 3 is the starting point so there’s no third grade SGP).  It turns out there is a considerable variation from grade to grade.  Here, then, are the data for each grade, starting with Grade 4 Reading:

image

Grade 5 Reading:

image

There you see Richmond doing quite well.  But wait.  Here are the data for the higher grades, starting with Grade 6 Reading:

image

Grade 7 Reading:

image

Grade 8 Reading:

image

These graphs show a distressing pattern that we will see repeated in the math graphs: Richmond does an average or better job in the elementary schools and a particularly awful job in the middle schools.  Note here the drop from a better-than-average 49 in the fifth grade to a third-from-the-bottom 33 in the sixth.

In summary, here are the 2014 Richmond reading averages by grade v. the state.

image

These data don’t tell us why our middle schools are doing such an awful job.  But they (in conjunction with the data by teacher – stay tuned for a later post) certainly tell us where the system is in dire need of repair.

 

Next post: Math SGPs by division.