Value Added Modeling: Administators v. Mathmaticians

Forums: Education, Unions, Mathmatics, Tenure, Administrators
Email this Topic • Print this Page

Two interesting articles came to my attention today...

One, titled "The Failure of American Schools" says:

Quote:

To comprehend the depth of the problem, consider one episode that still shocks me. Starting in 2006, under federal law, the State of New York was required to test students in grades three through eight annually in math and English. The results of those tests would enable us, for the first time, to analyze year-to-year student progress and tie it to individual teacher performance—a metric known in the field as “teacher value-added.” In essence, you hold constant other factors—where the students start from the prior year, demographics, class size, teacher length of service, and so on—and, based on test results, seek to isolate the individual teacher’s contribution to a student’s progress. Some teachers, for example, move their class forward on average a quarter-year more than expected; others, a quarter-year less. Value-added isn’t a perfect metric, but it’s surely worth considering as part of an overall teacher evaluation.

After we developed data from this metric, we decided to factor them into the granting of tenure, an award that is made after three years and that provides virtual lifetime job security. Under state law at the time, we were free to use these data. But after the New York City teachers union, the United Federation of Teachers, objected, I proposed that the City use value-added numbers only for the top and bottom 20 percent of teachers: the top 20 percent would get positive credit; the bottom would lose credit. And even then, principals would take value-added data into account only as part of a much larger, comprehensive tenure review. Even with these limitations, the UFT said “No way,” and headed to Albany to set up a legislative roadblock.

June 2011

http://www.theatlantic.com/magazine/archive/2011/06/the-failure-of-american-schools/8497/2/

The other, written by a group of mathematicians says:

Quote:

The latest instance of the phenomenon is
valued-added modeling (VAM), used to interpret
test data. Value-added modeling pops up everywhere
today, from newspapers to television to
political campaigns. VAM is heavily promoted with
unbridled and uncritical enthusiasm by the press,
by politicians, and even by (some) educational experts,
and it is touted as the modern, “scientific”
way to measure educational success in everything
from charter schools to individual teachers.
Yet most of those promoting value-added
modeling are ill-equipped to judge either its
effectiveness or its limitations. Some of those
who are equipped make extravagant claims without
much detail, reassuring us that someone
has checked into our concerns and we shouldn’t
worry. Value-added modeling is promoted because
it has the right pedigree—because it is based on
“sophisticated mathematics”. As a consequence,
mathematics that ought to be used to illuminate
ends up being used to intimidate. When that happens,
mathematicians have a responsibility to
speak out.

.........

Whether naïfs or experts, mathematicians need
to confront people who misuse their subject to intimidate
others into accepting conclusions simply
because they are based on some mathematics. Unlike
many policy makers, mathematicians are not
bamboozled by the theory behind VAM, and they
need to speak out forcefully. Mathematical models
have limitations. They do not by themselves convey
authority for their conclusions. They are tools, not
magic. And using the mathematics to intimidate—
to preempt debate about the goals of education
and measures of success—is harmful not only to
education but to mathematics itself.

Quote:

For a variety of reasons, analyses of
VAM results have led researchers to
doubt whether the methodology can
accurately identify more and less effective
teachers. VAM estimates have
proven to be unstable across statistical
models, years, and classes that teachers
teach. One study found that across five
large urban districts, among teachers
who were ranked in the top 20% of effectiveness
in the first year, fewer than
a third were in that top group the next
year, and another third moved all the
way down to the bottom 40%. Another
found that teachers’ effectiveness ratings
in one year could only predict
from 4% to 16% of the variation in such
ratings in the following year. Thus, a
teacher who appears to be very ineffective
in one year might have a dramatically
different result the following
year. The same dramatic fluctuations
were found for teachers ranked at the
bottom in the first year of analysis. This
runs counter to most people’s notions
that the true quality of a teacher is
likely to change very little over time and
raises questions about whether what is
measured is largely a “teacher effect”
or the effect of a wide variety of other
factors [Baker 2010, 1].

May 2011

http://www.ams.org/notices/201105/rtx110500667p.pdf

I think this is a very interesting juxtaposition of things to read on the same day.

Does reading one effect the way you read the other?

What do you make of this?

Topic Stats
Top Replies
Link to this Topic

Type: Question • Score: 1 • Views: 4,515 • Replies: 53

@boomerang,

boomerang's source wrote:

Whether naïfs or experts, mathematicians need
to confront people who misuse their subject to intimidate
others into accepting conclusions simply
because they are based on some mathematics....
...They are tools, not
magic. And using the mathematics to intimidate—
to preempt debate about the goals of education
and measures of success—is harmful not only to
education but to mathematics itself.

I agree with most of what the mathematicians are saying, except the part about the abuse of mathematical analyses harming mathematics itself.

After all, Math is God. You can't harm God ! Wink

I don't do all that much modelling, so I'll need a week or perhaps two to look into this, beginning with the Atlantic article.

(A professor at my local U. wants to teach me some stuff at times when I'm not at my job or tutoring, mainly because she's frustrated that no one else can understand what she talks about. So I'll be a bit busy...)

1 Reply

@Oylok,

But.... but.... but..... God is magic so how can math be god if math isn't magic?

Now I'm really confused.

If you do get time to delve into it, I'd love to know what your further thoughts are.

1 Reply

The testing and measuring of anything is subject to both errors and misinterpretation of the results. Despite this we have built a modern world of science, technology, and industry based on them.

No profession has a longer or better developed tradition of measuring the degree of proficiency and understanding of its subjects than education. From Aristotle until today the world of academia has relied on testing to select its leaders, admit its members, and choose whether to pass or fail its students.

It is merely a consequence of the frailties of human nature that our educators who themselves claim exclusive rights to run our schools based on their own professionsl education and tests; and who freely use thse devices to judge their students - then go on to express outrage that the same principles might be applied to them in the performance of their jobs.

Just who were the outraged mathematicians in this dispute was not made clear (did they work for the teachers union?). However, even there it is all too human for the appointed (and exam certified) experts to wring their hands at the possibility that lesser mortals might misuse their techniques.

1 Reply

@georgeob1,

Every teacher I've talked to says that they don't believe the current test measures anything worth measuring for students.

Every researcher agrees that the tests are worthless (except the ones that say they agree while their research shows they shouldn't).

If they don't think it's a valid measurement for children why in the world would they think it was a valid measurement for them?

1 Reply

@boomerang,

And yet the same teachers, unions and other elements of the education establishment have lobbied very hard to restrict teaching jobs exclusively to those who have taken and passed courses in the supposed techniques and "science" of education. (I have a Ph.D. in engineering, but I'm "not qualified" to teach a high school Algebra course in a California public school.) Indeed they value these credentials above real knowledge of the subject matter they may be teaching. If the course work and graded certification exams are so important for the qualification of teachers initially, why can't similar processes be used later on to measure the effectiveness of those same teachers in doing their jobs?

Why is it there are well-used measures of efficiency and effectiveness for just about every job under the sun, but only in public education is this deemed to be a meaningless and ineffective process?

The truth is the education establishment and its unions have established a monopolist hold on an important and fairly lucrative sector of public employment, and they will not admit anyone to hold them accountable for the attainment of any real goals or productivity in doing it. It's a very cushy deal and they won't give it up easily.

1 Reply

@georgeob1,

I don't want an engineer teaching my kid algebra, I want a teacher teaching my kid algebra. Teaching someone to do something and doing something are two different things.

I don't have a problem with them testing the teachers. But testing the students as a means of testing the teacher I do have a problem with. Kids vary, some might not have anything to eat for the last few days before the test, some might have had a parent walk out on them the night before the test, some might just be shitheads all of the time-- none of those things are the teachers fault yet we test the kids (no matter what) and use that to score the teachers. That's absurd. There are lots of ways to test the teacher without testing the students.

Yes, I'm sure it is fairly lucrative and that's why so many people want to privatize it. Siphon the money into testing companies or privately run charter schools (where the management company and the administrators make a fortune and the teacher turn over rate is 3 years) and out of the schools and then blame the schools for not being good. Nice work, "reformer"!

The truth is that these tests are meaningless for measuring achievement in children and even more meaningless for measuring the competence of a teacher. We're spending BILLIONS for information that any decent realtor would give us for free.

1 Reply

@boomerang,

boomerang wrote:

I don't want an engineer teaching my kid algebra, I want a teacher teaching my kid algebra. Teaching someone to do something and doing something are two different things.

An old saying comes to mind here,...."Then as can, do: them as can't, teach." Very hard to believe that one can teach well without a high degree of understanding of the subject matter.

It also appears here that you do believe there are differences in the skill levels and effectiveness of different teachers, If so, why are you so opposed to measuring their effectiveness - something that is done almost universally in every other profession?

boomerang wrote:

I don't have a problem with them testing the teachers. But testing the students as a means of testing the teacher I do have a problem with. Kids vary, some might not have anything to eat for the last few days before the test, some might have had a parent walk out on them the night before the test, some might just be shitheads all of the time-- none of those things are the teachers fault yet we test the kids (no matter what) and use that to score the teachers. That's absurd. There are lots of ways to test the teacher without testing the students.

Funny, that all these distorting possibilities also applied in the educational process used to qualify the teachers in the first place. Is that process also meaningless?

Please elaborate on the other ways of measuring teacher effectiveness that you might accept.

boomerang wrote:

The truth is that these tests are meaningless for measuring achievement in children and even more meaningless for measuring the competence of a teacher. We're spending BILLIONS for information that any decent realtor would give us for free.

If tests of knowledge are meaningless for children then do you also propose that we simply dispense with all of them? How will teachers make pass/fail decisions? How will we motivate students to work hard to learn and understand? Again, if these tests are meaningless then so are the exams and processes the teachers and education establishment themselves use to secure monopolistic control of the public schools. You can reject one without alsoi rejecting the other.

3 Replies

@georgeob1,

Quote:

If so, why are you so opposed to measuring their effectiveness

The problem is the measure. Most professions have much more clear objectives, which make for much easier measurement.

If you are supposed to make 20 bridges in a year, it can be measured that you have built 20 bridges in a year.

If the bridges are supposed to come in under budget while meeting specific safety requirements, that also can be measured.

But how exactly do you quantify success in a teacher? That's the big problem.

As boomer says, way too much is out of their control. A mediocre teacher with a classroom of kids with full bellies, involved parents, and lots of sleep is going to get kids who do well on test scores; while even a great teacher won't be able to significantly boost the test scores of kids with emotional upheaval at home, or empty bellies, or any number of other things outside his or her control.

I'm a big believer in rewarding good teachers, so the fact that it's such a problem to quantify whether they are in fact good has been a frustration to me.

I'm reminded of something I saw a while back:

Quote:

If doctors were treated like teachers:

1. "Charter hospitals" could certify "smart people" as qualified to begin practicing medicine without any prior experience in the field if they had had "some business background."
2. Since a "doctor" can "doctor" anything, a cardiologist would be on staff at a hospital in place of a urologist when there was a shortage of urologists. The cardiologist could "learn on the job." Of course, a general practitioner could be used in the place of any specialist since such a doctor would have "general knowledge" of anything involving medicine.
3. Whenever a doctor gave a patient a prescription, the patient's parents could come to the doctor's office demanding he or she change the prescription since the parents "knew better."
4. Because of a shortage of doctors, Mayor Bloomberg would institute a summer "crash course" in medicine for people who had no background in the field but "liked playing doctor" when they were little. Those who got through the six-week course would then be considered qualified to care for the most severely ill patients since no other doctors would want to do the job.
5. Doctors would qualify for "permanent license" if they showed by their rates of patient survival that they were "improving their scores." In order to do so, doctors would only treat the healthiest patients and refuse to treat the sicker ones to keep their rates of successful treatment high.
6. Many "Charter hospitals" would be established in which unlicensed doctors could practice the latest techniques on their patients, using the funds of public hospitals to subsidize them. Of course, only the healthiest patients, whose relatives cared enough about their condition to place them in a charter hospital would be admitted. Any patient exhibiting signs of serious illness would be immediately discharged and placed in a public hospital.
7. The average longevity of a doctor's career would be considered "normal" if he or she practiced for no more than five years.
8. If a hospital proved to have a poor "patient survival record," it would be closed down and three new hospitals would be created in the same building with nothing to do with each other but with three times as many bureaucrats running them.
9. Any patient who entered a doctor's care when already terminally ill would be expected to make a full recovery -- or the doctor would be considered incompetent.
10. A special program -- "Heal for America" -- would recruit students who graduated from the top colleges in the country but with no background in pre-medicine to "try to make a difference" by being placed in the most severely crowded and understaffed clinics and hospitals so they could know "what it feels like" to be a doctor, if only for a few years.
11. The American Medical Association would be condemned by politicians and health "experts" for "protecting incompetent doctors" on the basis of mortality rates in high-risk neighborhoods and the organization would be disbanded as a "menace to public health."]/quote]

http://www.indypendent.org/2010/11/04/if-doctors-were-treated-like-teachers/

2 Replies

@georgeob1,

Quote:

If tests of knowledge are meaningless for children then do you also propose that we simply dispense with all of them?

Only the meaningless ones. ALL tests are not equally meaningless, but standardized tests - such as the ones we use in this country - most certainly are.

Cycloptichorn

1 Reply

@georgeob1,

Quote:

Very hard to believe that one can teach well without a high degree of understanding of the subject matter.

Fine. Give the teacher an algebra test. Make it a really hard one. If they pass, they're qualified to teach it. Provided they have undergone all the other requirements needed to become a teacher -- like studying child development.

Quote:

It also appears here that you do believe there are differences in the skill levels and effectiveness of different teachers, If so, why are you so opposed to measuring their effectiveness - something that is done almost universally in every other profession?

Of course there are differences and I'm not against trying to find a way to measure it. Just don't make my kid (or anybody else's kid) your damn yardstick.

Quote:

Please elaborate on the other ways of measuring teacher effectiveness that you might accept.

Perhaps they could do it the way they did it for the 100 years before standardized testing.

Quote:

If tests of knowledge are meaningless for children then do you also propose that we simply dispense with all of them? How will teachers make pass/fail decisions?

Tests of knowledge are fine. Tests of regurgitated factoids have nothing to do with knowledge. They can make pass/fail decisions based on the quality of work the student has turned in over the course of the year, how involved they were in class, how well did they understand the lessons.

Quote:

How will we motivate students to work hard to learn and understand?

Those tests don't motivate students to do anything other than hate school and dread going there.

Quote:

Again, if these tests are meaningless then so are the exams and processes the teachers and education establishment themselves use to secure monopolistic control of the public schools.

I don't even get what you're going on about here with your references to "monopolistic control".

1 Reply

@sozobe,

I love that doctors - teachers thing. Thanks for posting that!

0 Replies

@Cycloptichorn,

Darn tootin'.

0 Replies

@boomerang,

Quote:

I don't even get what you're going on about here with your references to "monopolistic control".

Every evil in our education system, to George, is directly the fault of teachers unions.

Cycloptichorn

1 Reply

@Cycloptichorn,

I'm totally willing to bet he heard that on FOX news.

1 Reply

@boomerang,

boomerang wrote:

I'm totally willing to bet he heard that on FOX news.

I wouldn't say that about George. He's likely held similar opinions long before Fox News even existed.

It's likely not personal - he disdains all unions and unionism as some sort of cancer upon our society.

Cycloptichorn

1 Reply

@Cycloptichorn,

Oh... okay. Thanks.

1 Reply

@boomerang,

boomerang wrote:

Oh... okay. Thanks.

Yeah. I don't agree with his position on this issue, but I think it's unfair to lump him in with people who form their opinions based on Fox News broadcasts.

Cycloptichorn

1 Reply

@Cycloptichorn,

Thanks again.

I don't believe I've had much interaction with this particular poster.

I've found that most people who heartily endorse standardized testing as a way to rate teachers typically get their "ideas" from the FOX news talking heads. There is so much information out there that that exposes these tests for what they really are that I can't imagine that anyone who is really well informed or curious about the matter still puts any faith into them.

1 Reply

@boomerang,

You omit several relevant facts. IQ tests, SATs, LSATs and other like specialized formats are all standardized tests, subject to all the uncertainties and potential distortions to which you identified with respect to tests of public school students. Despite these distorting effects these standardized tests have proven themselves to correlate very strongly (not perfectly, but very strongly) with subsequent academic performance and success in various professional education programs and the professions themselves. The results of these tests are very widely and beneficailly used in a host of decisions about people, ranging from college admissions to employment and other like things. Would you propose that we do away with all such things? If so, how would you replace their apparently long term beneficial role in such decisions. Any such system is subject to imperfections, but the alternative to them is likely far worse with decisions being made by uncontrollable subjective factors inviting many forms of unfair abuse.

The unions that oppose the use of these tests or any other method of ranking the objective performance of teachers, have also resisted the termination of public school teachers for bad performance, the fact that no principal in the school system would accept them or even criminal coinvictions. A recent Op ed in the WSJ by the former head of the New York city schools outlines a very sad litany of this stuff. I believe it stretches credulity to the breaking point to suggest that the unions really have the public interest at heart on this matter.

3 Replies

Value Added Modeling: Administators v. Mathmaticians

Related Topics

Quick Links

My Account

able2know