My Photo
Blog powered by Typepad

« Bad news for Bush: A summation | Main | Bad news for Bush UPDATE »

February 13, 2006


Hey Terry:

A friend told me about your blog, and that you had asked a question of me. I'll make sure to check in regularly.

First, thanks for reading BrainstormNW. Feel free also to check in and participate in my blog, linked above.

To answer your question....

No, I do not think that the fact testing is a reliable measure of only a few well-defined academic skills and knowledge AT ALL repudiates the value of test based accountability.

You describe reading and math as a mere "sliver" of what schools teach. I take issue with that. It is like describing the concrete foundation of a high rise building a "sliver" of what the construction crew does.

Would you not test that foundation's strength before building the high rise?

It is a happy coincidence that the narrowly defined academic skills of reading and math achievement are not only essential to academic success, but happen also to easily lend themselves to standardized measures that have high validity and reliability.

That is, we can actually answer the question with a high degree of confidence "How well can Johhny read?" Further, the question of how well every Johnny and Jane in a school or classroom reads is an essential question to answer if we want to know whether a school or teacher is performing well.

Therefore, testing for reading and math achievement is a terrific way to hold schools accountable.

Does it hold schools accountable for everything we hope the kids learn? Nope. If there was a valid and reliable way to do that that did not get in the way of learning, I might support it. But there isn't.

Oregon has wasted 15 years and billions of dollars proving me right about this.

Let's test reading and math, using tests we know are reliable and that don't consume huge amounts of instructional time. Then we will know which teachers are effective at making sure all kids can read and do math, and we can evaluate curricula to see which are more effective.

And when kids master these basics they will be in a good position to extend and enrich their academic life with other higher order skills and knowledge that the schools also teach, such as expositiory and creative writing, history, civics, art, drama, speech, foreign language, current events, etc.

Why someone would be against testing for the acquisition of basic skills, given that we know we can do it reliably, is beyond me.

The only reasons I can imagine that someone could object to testing reading and math are: 1)the opinion that testing is not a reliable measure of reading or math achievement, or 2)the opinion that reading and math are not all that important for students to master, or 3) it isn’t the teacher’s and school’s fault if the student doesn’t learn to read and do math (In other words, you don’t WANT to be held accountable.)

Which is your reason?

Thanks again for reading BrainstormNW. For those interested, you can subscribe by calling (503) 675-7366.

Rob Kremer

Thanks for the response, Rob.

As a former reading teacher, I atually believe in testing for basic skills like reading and math. I just don't believe that you can hold either teachers or schools accountable for STUDENT performance on standardized tests, regardless of how reliable or valid they are (something of a red herring in the whole testing-based standards debate.)

Standardized tests are good diagnostic and placement tools for use by teachers. They should NEVER be used to label schools, nor should they be used to judge the performance of teachers.

When I ran for school board in 2003, I responded to a questionnaire from the African American Alliance (Tony Hopson, et al) with the crackpot suggestion that teachers with good records in the test score race should be promoted to administrative positions. What I wrote in response should also suffice as a critique of your "using the tests to hold teachers accountable" scheme. I wrote, in part:

"If you can convince me that student performance on two standardized achievement tests truly reflects the instructional prowess of individual teachers, I would happily accept your 85% benchmark criterion. That may be partially true at the elementary level, but it certainly isn't true at the middle or high school levels, where students typically have SEVERAL teachers durng the school day. Even at the elementary level, third graders are the products of several years of learning BEFORE they encounter their classroom teacher. I believe strongly that good teachers and good programs can make a difference in the learning of stedents, but its virtually impossible to judge good teaching on the basis of test scores."

I would add that elementary schools almost universally do a good job in reaching reading and math benchmarks, so I would assume that your concern is mainly with student test performance at the upper grade levels.

How do you propose to evaluate teachers in grades 6 - 12 using ONLY standardized tests?

I look forward to your response.

You ask:

"How do you propose to evaluate teachers in grades 6 - 12 using ONLY standardized tests?"

I don't. I never took that position. Why do you try to set up straw men?

If I have misconstrued your position on assessing the performance of teachers and schools, I apologize. But here's one of your recommendations for PPS:

"3) Launch a ten year effort like Chicago Public Schools is doing to contract out low performing schools to be operated by outside school management companies.

What other criteria beyond test scores determine which schools are "low performing"? Please set me straight, and I'll avoid the "straw man" critiique.


Now you ask an entirely different question. First, you ask me how I would evaluate teacher using ONLY test scores. I never took the position that test scores were the only factor in teacher evaluation.

Then you ask me to tell you how I would evaluate SCHOOLS, and whether there are criteria beyond test scores that I would use.

You may know that I am co-founder of the Arthur Academy charter schools, of which there are four around the metro area. We will open more in the coming years. Each of these schools has a contractual accountability system that includes test score gain targets, financial indicators, parent satisfaction measures and other more compliance oriented requirements.

A failing Arthur Academy would be one that does not meet these criteria and fails to correct the problem given adequate chance, which, depending on the issue, is 90 days to a year.

Test scores are an essential element of this equation, whether what we are analyzing is a teacher or a full school. If a teacher systematically fails to deliver test score gains over time, there is a problem. We look into the problem - it could be a lot of things that are not the teacher's doing. For instance what if several of the kids were absent for 30 days out of 45 - that would certainly show up on test score averages, but you can't blame the teacher for it.

However, assuming a normal distribution of student attributes, low test score gains show there is a problem with the teacher. If the teacher is unable or unwilling to correct this by getting better at using effective teaching methods, with the assistance of the instructional leader, then see ya later.

We just released a first grade teacher in one of our schools. Middle of the year. During the first few months we realized the teacher was not faithfully implementing our program. Assistance and counsel was provided. Enhanced oversight, regular communication. She would not do what we asked. Bye bye.

We were not willing to sacrifice the entire year's instruction for these kids. It is too important.

Same goes for an entire school. Guess what? Tough decisions like the one we just made with this teacher NEVER happen in the district operated public schools because the union won't let it. Ineffective teaching practices are allowed to survive over long periods of time.

Combine a low income/high minority/high percentage ESL student population with ineffective instruction and management that is not allowed to make changes, and I guarantee you the test scores will be substandard.

Close the school and put in place a management team that doesn't have its hands tied.

So, there you have it. I imagine you are a big union guy, so probably disagree with everything I have said. Test scores are an essential element in both teacher and school evaluation. Not the only element, but essential.

Arthur Academies are thriving, in part because the teachers and management know precisely how every kid is progressing and they have the management authority to make whatever adjustments are necessary to make sure no kid falls behind.

Many schools underperform because they are allowed to over long periods of time - allowed to because the governance model won't let the changes be made that might improve things.

So, let's change governance models. Close those schools that are structurally disabled, and replace them with a more effective governance model.


With your "... extensive background in school reform." perhaps you could address with a few more specifics the issues folks like Rob Kremer ACTUALLY advance.

Your profile mentions your concern with "... adequately funded public education" among other public policy issues. A lovely phrase and, tellingly, always the first thing union types mention. How much, by the way, is "adequate"?

Let me guess. More?

Your post primarily serves to demonstrates your pique that anyone should presume to consider measuring teachers' or schools' performance - using ANY yardstick.

Your self-serving mischaracterizations of Kremer's statements and positions would be laughable if they didn't reflect the selfish, closed-minded groupthink of many, if not most, of your OEA brethren.

As " idealist driven by a sense of moral indignation.", your " pragmat(ism)..." is turned on its head by your own inablity or unwillingness to see the reality of the FAILURE of public education as it's now constituted.

"Besides education, Terry is concerned with the environment, with human rights, and with democratic institutions"

What Terry is really concerned about is protecting the status quo of union-dominated, mediocre (at best) public education.

Unions First!

Children somewhere down the line.

First of all, Rob, it's TERRY, not Joe. Joe's my dog, and a smart one at that.

I didn't ask you about charter schools. I asked you about tradtional public schools that you want to close down because they "underperform". The ONLY standards in place now for such judgments are standardized achievement tests.

Since you bring up the issue, we disagree entirely on charter schools, or more specifically, the Arthur Academies, which are nothing more than vehicles for Direct Instruction. "Schools" like those, with tightly scripted lessons, don't need teachers at all. They need technicians. They may indeed be effective in raising math and reading scores for elementary students, but only because that's what the lessons are directly designed to do - raise test scores. Forget about anything else, like higher thinking skills, the arts, music, or, god forbid, an actual love of READING.

My concern about effective education has nothing to do with unions. Nor does school "governance", over which unions have little control. But there we obviously disagree, too. Unions assure workers of a living wage which I think is a good thing. That doesn't seem to be one of your concerns.

Wow, Terry with a dog named Joe:

You obviously know zip about either Arthur Academies or Direct Instruction.

Which is pretty typical among certified teachers, actually. You really ought to look into the "Project Follow Through" research, which is the single largest comparative education research ever conducted.

Direct Instruction obliterated all the other curricula - several of which were in the "critical thinking" camp. It dominated not only in measures of basic skills acquisition, but also in conceptual understanding AND self esteem!

Look for yourself - a simple web search will suffice, if your mind is not as closed as your incredibly ignorant statement above indicates.

""""Standardized tests are good diagnostic and placement tools for use by teachers. They should NEVER be used to label schools, nor should they be used to judge the performance of teachers."""""""

I wish the ODE had that mindset when they lie every year about Oregon being "tops in SAT scores"

A lie the OSBA and OEA and COSA help tell.

Am I the only one that noticed a very positive and potentially productive discussion about the role of testing in education started to turn nasty during the fifth comment and degenerated from there?

It turned sour, not when Rob mistakenly referred to Terry as Joe, but with Rob's comment, "I imagine you are a big union guy, so probably disagree with everything I have said."

Now I happen to respect Rob and believe he has a lot to contribute to the education debate (as he did with his earlier posts in this thread) but somehow we need to get past the knee-jerk tendency to make every education discussion about teachers unions and the false assumption that teachers who belong to a union cannot have any interest in educating kids beyond their personal self-interest.

I for one would love to see a continuation of the debate, picking up where the first four comments left off.

Thanks for the observation, Jack. Blog comments typically turn nasty (or snide) the longer they go on.

I appreciate your comment about union teachers. The real issue here is the role of testing in school reform, not about teacher accountability. I strongly believe that standardized achievement tests should not be used to evaluate the so-called performance of schools.

I don't think I ever got a straight answer from Rob about how tests can be used to determine which schools are up to snuff and which aren't. That's the discussion I would like to have.


Achievement tests that are anaylyzed on a "value added" basis (which Oregon's, at this time, are not) are how I would judge teachers and schools to see which are underperforming.

Tennessee has the most well established model, however other states including Oregon are moving down this path and the technical elements of the model have evolved somewhat from Tennessee's pioneering system.

The analysis looks at student academic growth each year. It matters not where the kid starts - he could be three grade levels below standard.

When the data tell us how many students in a teacher's class grew by at least one academic year in achievement for a year spent in the classroom, then we have a decent measure of teacher effectiveness.

The data from Tennessee are astounding. They know which teachers are effective and which are not, especially for the elementary levels. Their data shows that a student placed for two consecutive years in an ineffective teacher's classroom has something like a 5% chance of ever getting back to standard.

The benefit of value added analysis is that it removes demographics from the equation. All the attributes that we know are related to achievement - income level of parents, education level of parents, single parent families, language other than English spoken at home - the teacher and the school don't get "marked down" because of them.

Conversely, schools in affluent areas don't get to take bows because of the natural advantages their kids have.

In our current testing system, a student in NE Portland could enter fifth grade reading at a second grade level, and the extraodinary teacher could move her up the the fourth grade level in one year. Then she takes the test, and fails to meet fifth grade standard. Teacher/school gets dinged, completely unfairly.

A student at Riverdale could enter fifth grade reading at a 7th grade level, and the worthless teacher (which Riverdale has many of) might not move the kid at all - she ends up reading at fifth grade level. Takes the test and meets standard. Teacher/school takes a bow, undeservedly.

So, our current system is rife with these false negatives and false positives (Type I and Type II errors, in statistical language.)

A value added system avoids that problem.

The model Tennessee uses is also predictive. I had it demonstrated to me by the guy who developed it, named Bill Sanders. Using an individual student's test score data over his school career, they can ask a question such as: "if this kid wants to score 1200 on SAT, what will his trajectory of 8th, 9th, and 10th grade scores need to look like?"

So, you ask how I would use test data to judge teachers and schools? With a value added assessment system.

I fully admit that the picture gets murkier the higher the grade level, and that it is not worthwhile trying to test for history knowledge or writing ability on a value added basis. At least I have never seen how it is done; maybe somebody has figured it out.

I still cannot imagine a teacher taking the position that teachers should never be accountable based on test scores.

To go back to my analogy of a construction crew building a high rise, I asked "wouldn't you test the foundation of the building before building the rest?"

You essentially said yes, you do test the foundation, but if it is found lacking that we shouldn't hold the construction crew accountable.

So they get to go merrily on their way and build more foundations?

Glad to see you're back to a serious discussion of the issues, Rob.

As near as I can tell, what you refer to as "value added testing" is simply testing for growth, which I agree is a much more productive use of achievement test data. When I first started in Hillsboro, we always tested twice yearly in reading and math, and I looked carefully at the GROWTH of my students on standard scale scores, which are highly RELIABLE indicators of a student's progress over time (at least for what's tested, which involves the separate issue of VALIDITY.)

As I have written elsewhere, Oregon used to look at test score growth as part of its school report cards. Schools with low scores could actually get good marks if students demonstrated growth. As I recall, the state actually published aggregate scores by grade level, not merely the percentage of students meeting or exceeding benchmarks. For some reason (probably because of NCLB or lack of money) that was discontinued some years back.

That said, there are still issues with using tests to evaluate individual teachers, as you yourself admit. Tests MAY be legitimate indicators of teaching proficiency at grades three and five, but ONLY for basic skills, like reading and math. (Remember, I was a reading -AND math- teacher, so of course I think that those skills are vital.) At the upper grades, tying tests to teachers is more problematic, and as I said before, it's at the middle and high schools where reform is a more pressing issue.

But even in the elementary grades, tests look primarily at reading and math proficiency. I don't recall having any BAD teachers in my K-8 school, but I remember only one (in fourth grade) as an excellent reading teacher. The others were strong in other areas - art, music, science, the social sciences. I would be hesitant to evaluate any of them solely on the basis of what's measured on math and reading achievement tests.

There are other complicating factors as well, especially if tests aren't administered twice annually. The most significant is student mobility, which is a big problem at low income schools. To track student scores in such schools is indeed a difficult (and probably expensive) task.

Anyway, I do applaud your effort to introduce the value added (growth) component to our testing system. And I look forward to continuing the discussion in future blog posts.


I'm not sure where I ever got off the issues, but thanks. Do you agree that it is appropriate to use value-added analysis of test scores as an element of a teacher and school accountability system?

If you wish to be the best man, you must suffer the bitterest of the bitter.

I "like" you on Facebook. Would love these for my oldest boy!

The comments to this entry are closed.

Most Recent Photos

  • War_prez_prima_1
  • Bushvaca2nh
  • Dscn1145_2
  • Dscn1144_1
  • Dscn1144_4
  • Dscn1137_3
  • Dscn1137_4
  • Dscn1051
  • Dscn1046
  • Dscn0883_1
  • Dscn0881_1
  • 422d683505eb4821_1