Observation and Evaluation

Perhaps now on the cusp of a new year is not the time to reflect on the processes and goals of evaluation: student evaluation, professional evaluation, institutional evaluation.  We try very hard in education to allow each new year to begin with a clean slate for staff and kids.  It is a new year, not a continuance of previous years, though we all know adults who have had the same year over and over for decades.  And we know that whatever records, whatever evaluations, whatever assessments from previous years follow each adult and each child into this new year.  And, we know that perceptions of reality tend to be more powerful than reality, whatever that is.  How in the world do we determine who are our best employees, good employees, poor employees?  How in the world do we determine who are our most successful kids, least successful kids?  How in the world do we determine which are the best schools and school systems?  How do we observe, collect data and make a judgment about such complex human behaviors?  Are the goals and the processes of observation and evaluation so flawed as to be simultaneously pointless and impossible?

In my lifetime I have had one wife, one son and one daughter.  My wife is the best wife I have ever had.  My daughter the best daughter and my son the best son I have ever had.  Should I care how they compare to other wives and other sons and daughters?  Is this evaluation on my part biased?  Is it true?  Does it matter?  They are who they are as am I.  I can improve as can they.  But should my improvement goals be assigned to me by them, the state, or self-selected?  Who decides and how?  Who evaluates and how?

I like vanilla ice cream.  You may think that is a generalized attribute of my personality, and perhaps you are correct.  I have consumed a large variety of ice cream flavors, (I used to manage a Baskin Robbins store and used that opportunity to sample and sample.  It is from that experience that I commenced “Scoop from the Supe”.) and I know that to my taste vanilla is my favorite, especially real homemade vanilla and Blue Bell Homemade Vanilla.  That is my preference, those are my tastes.  Should I judge others for their tastes?  Should it be that chocolate eaters are somehow inferior to vanilla eaters because I am the boss and I like vanilla?  And what about those who do not have a favorite and love sampling a wide variety of flavors based on mood, availability and circumstance?  Are they somehow flawed?  Shall vanilla become the defined preference, the defined “best” because I say so?  Should the “Vanilla Factor” be applied to evaluation of kids, staff, schools and districts?

And frankly I laugh at my private sector friends who claim to have simple, objective measures of success and productivity.  When the market determines success we are simply saying that there is an intersection between peoples’ willingness and ability to pay and their tastes.  I like General Motors, you like Ford or Toyota or Honda.  If enough people fall in each set of likes then all four companies will be successful and argue that they are the best because some people choose them.  If the product of any given company was clearly the best using every variable available to us wouldn’t everyone buy it leaving us no market choice, and therefore eliminate the term market?  Private sector success is based on a much higher level of luck and taste than private sector folks are likely to admit, assuming that somehow they have the secrets to evaluation.  Why are there Blackberries, iPhones and Galaxies?  Why are there both Google and Bing?  The “market” determines this in a mysterious way not known until after the fact based on the intersection of peoples’ willingness and ability to pay and their tastes, not divine inspiration, sophisticated “objective” measures, and the shroud of superiority claimed by some.  If we all wake up tomorrow and decide that we just do not like iPhones and iPads then no amount of marketing will save Apple.

Add the Heisenberg Principal of Observation (a.k.a. the Uncertainty Principle) to the above conundrums and observation and evaluation really become suspect.  Heisenberg, a brilliant physicist, simply stated that the observation of any process will result in a change in the process by the simple fact it is being observed.  This is true of the behavior of atoms at the sub atomic level and true of the behavior of people.  Teachers know this to be true on the day the principal sits in the back of the room to observe.  I know this to be true comparing how I sing in the shower with how I sing a solo in church.  We are altered by the observer and influenced by the variables of judgment of the observer.  One of the most powerful arguments against standardized measures of the quality of schools and kids is that we all now teach to the tests and the other variables of measure.  Of course we do.  Tell us the measure, what you are going to observe about our behavior and we refocus on that measure.  No one does differently!  Tell a private sector company that profit is no longer the measure of success but employee salary and satisfaction is the variable of importance, then the behavior of the company will dramatically change.  They will focus on the measure of judgment. 

We have huge problems in schools around this notion of observation and evaluation.  If every child is happy and productive, and every parent is pleased with their child is the school successful?  What if that is true but the standardized measures indicate that the school is woefully under performing compared to other schools and that state imposed sanctions should ensue?  What if a teacher always has a high number of students who are successful on standardized measures, but parents and kids dislike the teacher?  Is that teacher a success or a failure?  Can we be all things to all people, lowering taxes and funding while increasing services and outcomes and popularity?  I think not.

It has been my observation that unless I am consciously thinking about it, I have the most affinity and affection for those people who think like I think and act like I act and value what I value.  In other words, “The Vanilla Factor”.  If I must judge a student, an employee, a school or a district no matter how hard I may try to make the observational data and the evaluation “objective” the bottom line is that my values are inherent in the measures and my preferences are inherent in the judgment.  Currently, the Texas Legislature is determining those measures and values and they are doing so without professional license.  They are supported by the notion that measures of observation can be objective and used for comparison, and that somehow comparing school districts is an important thing to do.  Our performance is not determined by the market, it is determined by the day-to-day observation and evaluation of every kid, every employee, every parent and every community.  The Legislature mandates the outcomes on STAAR and other variables over the local variables so that communities who believed their schools were good are now told they are not so good using standardized measures.  For instance, when I began my career, high school dropouts were viewed as a solution, not a problem.  Kids who could not perform and misbehaved were encouraged to quit, much like charter schools and private schools still do today.  Now for public schools if they drop out it is a bad thing and we are held accountable.  Amazing.  To see my thoughts on all standardized measures please view my previous post “Wrong on Two Counts.”

The very act of observation impacts performance.  The variables used to observe performance impacts performance.  If standardized outcomes mean more than anything else we will sacrifice kids and staff for the purpose of achieving what I believe are spurious outcomes at best.  The evaluation that follows the application of such observational data is the most important assessment to the state.  The evaluation locally is much more likely to be based on the Vanilla Factor:  If I like you and you think like me you are good.  But, if you have the wrong bumper sticker it is time to go.  If I perceive my child is somehow short changed or treated unfairly then as a parent I am mad.  If we do not value this program more than any other then I am mad.  It is fascinating to look back over Board Notes and identify what issues brought parents and patrons to the Board to complain.  None have ever showed up concerned about TAKS or STAAR outcomes.  It is always the Vanilla Factor.

It is that vice that is squeezing the life out of public education.  We all know we are held accountable for standardized measures and we understand that given the nature of standardized measures at least half of the districts in the state will be in the lowest two quartiles.  Every Friday night half the football teams in Texas lose.  Should we therefore conclude that 50% of the football programs in the state are failures?  We also know that there is a tendency to judge folks on variables more intimate than standardized measures:  Do we like this person?  Do they think like I think?  Are their values like my values?  If so, we shall keep them regardless of standardized measures.  If not we shall run them off regardless of standardized measures.  Therein lays the squeeze.

I am so very fond of and proud of the professional leadership in Edna.  As I look around the Cabinet table at Fred, Nancy, Madalyn, Jamie, Demetric, Paul and Deborah I could not feel better about the quality of our leadership.  I am equally personally proud of the diversity around that table.  None of us are exactly alike.  Some of us value this measure more than that measure, etc.  We think differently on a variety of topics.  We all have different leadership styles, talents, history and to some extent values.  But when we arrive at consensus I am convinced it is the best possible solution available.  Hence I value collaborative decision making, diversity, and I work hard to avoid the Vanilla Factor in my observation and evaluation of this team.  I want to work with folks who prefer chocolate and folks who prefer something different every time they eat ice cream.

For the past 14 years this is the best school system I have worked for.  For the past 14 years this has been the best Board I have worked for.  Our standardized measures are good, our intangible measures are good, our employees are the best group I have worked with and our kids are the best kids I have worked with.  I care about the standardized measures only because the state tells me I have to care and I find them interesting, but I believe the state is wrong.  If the state is going to judge us entirely on standardized outcomes then the state should at least have the courage to say all efforts toward improving those outcomes are supported, i.e., teaching to the test, and all efforts to achieve success on any other outcome is pointless.  Parents need to know that. Parents also need to understand how standardized outcomes work via the normal curve and that even if we improve every year we are not likely to ever improve our relative position compared to other districts unless those other districts have a serious decline.  Parents and patrons and teachers need to know that nothing matters more than those outcomes; not our football season, not the number of volleyball coaches, not the number of kids in the marching band, not home economics, and not class rank.  Nothing matters more than passing STAAR.  When we say we ensure a quality education for all we mean ensure all pass STAAR.  Ludicrous. 

Or, if an employee is going to be evaluated totally on the Vanilla Factor, he or she needs to know that as well.

All evaluations are based on observation of some variables.  All observations impact the performance of the observed.  All variables of observation are influenced by the preferences and values and beliefs of the observer.  If all observation of human behavior is by definition subjective, all evaluations, therefore, are subjective.  Should we stop evaluating?  No.  Should we be honest regarding our evaluations?  Yes.  If our evaluations are purely based on state performance measures let us say that and be prepared to take the lumps when the parents and community complain about our total shift in focus.  If our evaluations are purely based on the popularity, the alignment, and/or the local Vanilla Factor, let us say that and be prepared to take the lumps when the state announces we are worthless.  To come anywhere close to a fair evaluation, to have any sense of ethics at all, we absolutely must be honest up front regarding the variables we will use to measure success and we must announce that we have discovered a great truth:  we cannot be all things to all people. 

Why not?   Because not all people prefer vanilla.