Archive

Archive for the ‘special education’ Category

testing, testing, one, two, three

April 26, 2011 2 comments

Testing.  It’s become so much part of the life of a learner or a teacher, at any age.  And it’s a fascinating topic.

Okay, I’m one of those weird people who thinks of test-taking myself as a sort of competitive athletic event, one at which I’m really quite good even while thinking that the vast majority of tests I’ve ever taken were nearly completely pointless.  No, that’s not my impostor syndrome kicking in.  It has to do with a central concept in test design, which I’ll explain below.

What I love most about assessment is how useful it can be when done well.  One of my colleagues says that testing doesn’t bring out the best in people, it doesn’t bring out the worst in people, but it brings out the most in people.  We put you in a situation where your normal compensatory strategies for getting along in the world aren’t going to work.  As Peter Ossorio says, when you ask a person to do something they can’t do, they’ll do something they can do.  You’ll figure out something to do, the best you can, and what you do will be a reflection, in some way, of you.  It’s like science — each test is an experiment that you and I do together.  No one bit of data proves anything by itself, but when we put things together and look for themes, consistencies, divergences, a story begins to emerge, and it often does so surprisingly quickly.

But what bugs me is how little most folks understand about tests of all stripes — most importantly, how they’re built, how they work, what they’re good for… and what they aren’t.  So what I’d like to do is to kick off a random-access series of posts on various aspects of assessment, including ordinary classroom tests, high-stakes testing for No Child Left Behind Allowed Ahead (also known as No Teacher Left Standing) and other similar “accountability” movements, bubble tests like the dreaded SAT and its ilk, and, of course, my favorite, the one-on-one kinds of tests used for special education and other diagnostic work, the kind that seriously geeky people like me give.  Those include cognitive tests, neuropsychological tests, academic tests, psychological tests, behavioral questionnaires, and other fun stuff. I’ll start there because, well, because I like them and I think they’re really pretty interesting.  I’ll try to chew off manageable chunks to talk about, and over time, I hope people learn something.

The most serious and popular misconception I encounter is a fundamental misunderstanding of what tests can do.  They’re not magic, and neither are those of us who give them magicians.  We’re just very observant (or at least we’re supposed to be!), and we’re using them to make a series of structured observations.

Again, this is like science.  When I was training as a molecular biologist, one of the things I had thwacked into my head (through reading in the literature some of the truly impressively weird things that happened when people didn’t remember it) was that no experiment ever tells you anything about the real world.  It tells you what happened on that day when that person did that experiment in that way.  You might use that information to conjecture about the nature of the real world based on your data, and over time, as you build up more data, you can get a better and better sense of what the real world might be like.  But you might see a different experiment, claiming to answer the same question, where you get different results.  Uh, oh.  Where do you look, to figure out what was going on to find the difference that made the difference?  In the Materials and Methods, the specifics of how the experiment was designed and constructed.  Very often, that’s where the difference lies.  You cannot separate data from the experiment that generated it.

Same with assessment.   No test, no matter how beautifully it’s designed, how skillfully it’s administered, and how insightfully it’s interpreted, can possibly tell you anything incontrovertibly true about the real human being.  The test tells you what that person did on that day on that test with that tester in that environment.  It might reflect something probably true about the person, but you have to stay humble with your interpretation.

Since you will always value what you measure, it makes sense to think very carefully about how to measure what you actually value. In education, we talk about the idea of “alignment” — we’d say that this test is or is not well-aligned to the skills we want the student to be able to demonstrate.  That’s what I was talking about above, why I don’t respect the very bubble tests that I tend to be able to blow out of the water.  They typically test what is easy to measure, but not what a thoughtful professional would consider all that valuable.  At the conclusion of many thousands of hours of clinical training, psychologists in most states have to take a detailed fact-recall bubble test covering basically the entire field.  We to prove that we know which classic theorist suggested that you were running from the bear because you were afraid, versus which one suggested that you were afraid because you were running from the bear.  But we don’t have to demonstrate the capacity to actually manifest any clinical competencies with actual, oh, I dunno, human beings in distress.  In test design, we talk about the very-closely-related concept of “validity,” which comes in many flavors.  In this case, the construct validity of the test — how it defines what it is that it’s trying to measure — is awful.  Fact knowledge within a domain is a useful thing, and might be a good prerequisite to beginning clinical work.  But the public is not protected from incompetent psychologists by choosing only those who can remember the facts printed in their textbooks.

I think the best-aligned test I ever took was the qualifying exam for the Ph.D. I didn’t get in cancer biology.  I was required to dive in to fields I was unfamiliar with, learn about the prior research in those fields, and propose new lines of research that would answer important unanswered questions.  Minus the speed with which I had to do it (three of these, in completely different fields, within a single week!), this test was testing very much what I would need to do if I became a principal investigator running my own lab someday.   Of course, the alignment/construct validity of that test wasn’t perfect either.  What it didn’t explore was the personality traits which set me up to be a very sad and bored and frustrated person in the lab, the precise difference between thinking about science, which I love and am good at, and doing bench science on a day-to-day basis, which I don’t and am not.

What I find most concerning about the high-stakes testing (aka “accountability”) movement in education is that it tends to use tests with poor validity in a variety of domains (construct validity, content validity, and predictive validity being the most notable), and that it tends to ignore other underlying methodological differences between comparison groups (most notably, differences in the populations being served and the resources available to teachers and administrators to serve them, but also differences in how various jurisdictions define their goals and standards).  When science teachers teach kids about experimental controls, we start with the idea of a “fair game.”  But there’s no way on earth that these “games” are fair.  There’s nothing truly “standardized” about these experiments, and almost every interpretation that is made of them is a massive overinterpretation from inadequate data.  Gives serious testing a bad name.  Harrumph.

Okay, so my plans for this series of posts right now involve topics like the various types of validity and reliability (the twin pillars of assessment for people who actually want usable data!), and a sort of overview of each of the major types of clinical testing (e.g., cognitive, academic, neuropsychological, behavioral, projective) and what they are and aren’t good for.  I’ll do classroom and educational and high-stakes stuff later, but I’d rather start with what I do the most of.  If there are specific ideas or questions you’d like me to address, feel free to drop them in the comments area here.

Advertisements

Meetings: the final frontier

August 24, 2010 Leave a comment

In this article, a parent of a child with significant learning disabilities writes about how having her son in the meeting provided important information to the IEP team about the reality of what was going on in the classroom.  A few good grades were being taken out of context to indicate that he was doing extremely well, and the team was on the verge of exiting him from the special education system, until he pointed out that there were good reasons why he had done well on those assignments that had nothing to do with his disabilities having magically been cured.

http://www.ncld.org/at-school/your-childs-rights/advocacy-self-advocacy/a-parents-perspective-why-my-son-attended-his-own-iep-meetings

That’s an excellent point.   But I’d like to extend it.  I would say that in general, kids’ input should always be sought in the IEP process, as well as in any legal or other processes where grownups are talking with each other and making high-stakes decisions about them.

If a kid cannot behave in the meeting or tolerate having all of those grownups talking about them or about their situation, okay, fine.  Have a parent, therapist, guide, adult friend, or other safe adult who does not work for the school (or court, or whatever), someone who has no reason to pressure the child to give any specific answers, sit with the child ahead of time.  Explain the purpose of the meeting, explain the questions that the adults will be thinking through, and take the time to make sure the child understands as well as they are developmentally able to.  (Since I work primarily with gifted and multiply-exceptional kids, they tend to be able to understand this information at very young ages, and they tend to be very upset when they are aware of people talking about them without seeking their input.)  Invite the child to offer their own experiences, ideas, or opinions.  Ask things like, “What would you like the grownups to know?”  “What would a good solution look like for you?” “What are you most hoping (or most worrying) that they will say?”  And listen attentively to the answer.  If you can predict what the adults’ concerns about the child’s proposals might be, offer them for the child’s consideration.  You might be surprised how much depth of thought you will hear.

(Side note: those who are fans of Ross Greene and Stuart Ablon’s work around teaching kids to self-regulate will recognize this process, and those who are not familiar with it should read their book (professionals should read this one) or check out the website.  It’s not just for “explosive” kids.)

I also strongly believe that as soon as the kids are capable of tolerating the experience, they should be members of the team and participate in the meetings themselves.  I’d like to see them there by late middle school or early high school age, and absolutely by late high school age.  If they need to have an educational advocate, therapist, adult friend, or other non-parental helpful person sitting next to them to help them understand what is going on and to help them figure out how to express their own ideas appropriately, then that’s a great service to provide for them.

Think about it… when they’re 18 years old, legally, they chair the IEP meetings.  If they don’t want help, we can’t force them to accept it.  We can’t schedule a meeting if they don’t consent, and we can’t hold a meeting if they don’t show up.  Absent certain really seriously exacerbating circumstances, adults have the right to make their own decisions, no matter how foolish.  (I have personally see this play out, at times tragically, in a few situations, where a newly-minted legal adult made decisions that were mostly informed by their lack of experience.)

So part of the goal of the whole process is to train them to be good at the role of team member, and eventually to take on the role of team captain.  That’s good, because in real life, we are all captains of our own teams.  This is a great thing for them to learn to do, and a nice well-constructed venue for them to learn to do it in.  But they won’t learn how to do it without guided practice.

Are there kids for whom this kind of self-determination is not a realistic goal?  Certainly, there is a very tiny minority of children who will not be able to handle it, even with guidance and training and practice.  But if you feel that a kid shouldn’t be present in their own IEP meetings by the mid to late teenage years, chances are good that you also need to be thinking about legal guardianship or conservatorship in adulthood.  If you feel that a certain 16-year-old cannot even meaningfully participate in a meeting that may decide the course of their life, then I would be very, very concerned about that same person at 18 years old having the right to manage all aspects of their life on their own.

If your concern is that the child may be upset by the process, particularly if the child’s disability affects their emotional stability, let me say two things.  First, by high school age, even kids who are below average in intelligence tend to be aware that people are making decisions about them without their participation.  That’s really upsetting, too.  Second, one of my predoctoral internships was at the Arlington School.  It’s not in Arlington.  It’s a therapeutic high school on the campus of McLean Hospital, one of the world’s premier psychiatric hospitals.  The kids there are all there, at enormous school district expense, because they have debilitating major mental illnesses and really honestly cannot emotionally handle being in a regular high school.  If you were going to say that a kid couldn’t handle being in their own IEP meeting, those kids would be high on the list of kids you’d assume couldn’t cope.  Yet they do, and dialogue with them is an important part of the process both for the team and for them as they move into adulthood.  So I don’t think I’d be too quick to assume that a kid can’t learn to tolerate the experience.

Properly handled, having kids participate in their own meetings can contribute to their own maturational process.  Special education shouldn’t be something we do to kids, it should be something we do with them.