Home > assessment / testing, executive functioning, social-emotional, surviving school > bubble, bubble, toil and trouble… (multiple choice exams)

bubble, bubble, toil and trouble… (multiple choice exams)

I hear a lot, particularly around late fall / early winter, about students who have a particular difficulty with multiple-choice exams, like the SAT, ACT, GRE, and so on.  I personally think bubble tests are nearly always worthless in terms of telling us anything we actually want to know about a kid (it is possible to create a really good multiple-choice exam to explore real understanding of content and mastery of higher-order thinking skills… but it is extremely rarely actually done).  Unfortunately, they are a fact of life for students.

Sometimes, bright kids are like me — I cordially despise these tests, but I’m very good at them.   Always have been.  Even when I haven’t actually learned the content (in fact, especially so, since I have a terrible memory for the kind of isolated facts these tests so often rely upon).   I consider them something of a competitive sport.   But very often, “bubble tests” are a bright-to-gifted kid’s personal nemesis — “I understand it all, but I just can’t get things right on the stupid test!  I can’t remember the nitty little details and I can’t decide what answer they think I’m supposed to give and it’s all just awful!”

Neither the research literature nor the professional lore would support the idea that some people should be diagnosed with 315.9 Learning Disorder, Not Otherwise Specified, Cannot Take Multiple-Choice Exams.  Typically, if a kid (or adult) has persistent problems with bubble tests, one or more of several things is going on…

Test anxiety:  These tests tend to push an already-slightly-anxious person’s buttons.   The pace tends to be very rapid (in the realm of one minute or less per question), the stakes tend to be perceived as high (will you get into such-and-such program?), there are more wrong answers than right answers but they’re all pulling at you… eek!  According to the inverted-U hypothesis (aka the Yerkes-Dodson law), overly-high levels of arousal tend to decrease performance.  Moderate levels of arousal are good (see below under EF/ADHD), but if you get too buzzy, you end up crossing the line into freaked-out, and no one can concentrate well or do their best when they’re freaked out.

Lack of test savvy:  While I don’t think there’s any value to the kinds of apocryphal loree. kids like to rely on (“if you’re not sure, choose C”), it is worth recognizing that these tests are written by human beings.  Your goal is not actually to get the right answer.  Your goal is to think like the writer of the test questions, to see the underlying question they’re trying to ask, to spot the trick they’re trying to lead you into, and to choose the answer they want you to choose.  This is absolutely a learned and learnable skill.

Executive functioning problems (including ADHD):  Most often, the kids with ADHD are impulsively choosing the first “pretty-good” answer they see, or they drift off before reading the question thoroughly and carefully considering all possibilities.  Many of them have a hard time keeping their arousal level high enough to stay focused, keeping their focus on the task, and maintaining a working tempo that will let them finish in time.  Note also that kids who aren’t getting enough sleep will also typically struggle with these things.   Coffee can sometimes help… but it’s not as good as having the brain properly rested in the first place.

Issues with speed and pace: This is properly a subset of “executive functioning” above, but it’s something that a lot of folks have trouble with specifically on this kind of test, even when they don’t have trouble with it in real life.  Timed test-taking is itself a skill.  It is difficult to maintain the pace and rhythm needed to get the whole thing done.   People often get bogged down in a few hard questions and then can’t pick up the pace after they extricate themselves from the bog (too much mud on the boots, if you will).  Also, many kids have a hard time maintaining the required effort over the long period of time the tests take (the high-stakes tests such as the SAT are often several hours long).  Staying focused that long without reorienting cues is something we don’t practice that much these days.

Language comprehension problems (including Asperger’s):  Test questions are often quite finicky in terms of language — they’re highly specific in their meaning, and if you don’t read really carefully and focus on (1) exactly what they’re saying and (2) exactly what they meant (yes, I know those might sometimes seem like opposites… that’s part of the game!), you will trip up.  The wrong answer choices are almost always based upon the typical misreadings of the questions — these “attractor” answers are the reason that some kids actually do worse than chance when they guess.  Kids who have trouble in this domain often also have subtle weaknesses in the rest of the “real world” in terms of reading comprehension, analytical writing, and oral direction-following.

Overthinking:  I’ve often seen bright-to-gifted kids overthink these stupid multiple-choice questions, choose the second-best answer on a technicality because, “Well, it could be that,” etc.  Sometimes they’re getting all proud of themselves for coming up with a technicality, like, “Lookit me, I’m smarter than the test, ha, ha!”  But in school, teachers only rarely grant credit retroactively for coming up with a clever justification, and on those high-stakes tests, you’re almost never going to get credit this way.  The goal is not to get the right answer.  The goal is to get the answer the test writer wanted you to get.  Personally, when I was taking the tests for my high school teaching credential in science, while I pegged the upper reaches of the scores, I found it interesting that I had a relative weakness in the areas of science I knew the best (courtesy of ten years of training as a molecular biologist).  Why? Because I got stuck saying things like, “Okay, B is the actual right answer.  However, the overwhelming majority of the population thinks it’s C, and lots of textbooks say it’s C, too.  Did the person who wrote the test know about B, such that C is the attractor answer, or am I supposed to say C because that’s probably what the person who wrote the test thought the answer was?”

So, what to do?

What I generally recommend in terms of intervention, regardless of the cause, is to provide explicit instruction and guided practice in the specific skills involved.

Numero uno, the most likely area of weakness.  Make it a habit.  Always.  Read the  question carefully.  Read all of the choices.  Think through what each choice means and why it would be a good or bad choice.  Then (and only then) choose the best one.

Practice solving items by thinking out loud with a test-savvy tutor.   When mistakes are made, go over the explanations for the answers and use these as learning opportunities to understand better how test-writers think.

If there are specific high-stakes tests at issue then get the Big Thick Book of Real Practice Tests from the local bookstore and study the test itself. Learn to identify common question types.  In fact, it’s often worth it to practice rapidly identifying question type as a separate skill.   When you’re good at knowing the general kinds of questions, your study can then focus on strategies which fit each type — that makes your work a lot more efficient.   It’s better to practice a whole bunch of questions of the same type and master the skill, saving the mixed practice for when you’re reviewing skills you’ve already mastered.

As with any sport or musical instrument, regular practice, on items that are difficult enough to be challenging, is what you need to improve.  Massed practice (“cramming”) might feel like, “Ooh, I’m doing something heroic, this has gotta work.”  But it doesn’t work anywhere near as well as regular practice.  I know, I know.  You think it does.  Everyone thinks it does.  Sorry.  It doesn’t.   You’re not special.

Frankly, I generally don’t recommend the courses from test-prep companies unless you’re a kid who honestly won’t do the Big Thick Book technique reliably.  The courses tend to be basically just the same thing as the books, only there’s a grownup standing at the front of the room keeping you on task.  If that’s the only way you’ll reliably study, well, okay, fine.  But if you’re trying to take a high-stakes test that will get you into, oh, say, college, where, did anyone mention, no one reminds you to get out of bed or do your homework, perhaps this would be a good time to learn to get yourself to do the stuff you don’t like to do.

Particularly if anxiety or drifting-off is an issue, practice, practice, practice, under the most realistic conditions you can muster up.  Try out different techniques for reducing your anxiety or getting yourself woken up to the right level, and figure out what works best for you and is legal under test conditions (that is, if you do best with music, sorry, you will almost never be allowed to have an mp3 player on a standardized test, so you need to come up with something else).  If the unfamiliar location of a high-stakes test is a problem, try taking practice tests in different locations (public libraries are good).  Take them timed.  No food.   No potty.  No breaks.  No standing up.

For timing or pace issues, practice with a loop timer, gradually decreasing the time per item, to work on tempo.  Set a tempo that will get you finished in approximately 80% of the time allotted — that leaves time to work on the really hard questions that will take more thought.

Practice a strategy that will maximize the number of items answered probably-correctly.  A lot of people get stuck on hard items and won’t move on until they’ve figured out the answer.   It’s much more advantageous to look at each question, and if you know or can quickly figure out the right answer, do it, and if you don’t, circle the item number and move on.  That both ensures that you get to all the easy ones (which have the same point value as the hard ones!) and puts the content of the hard ones into your head where it can cook.  Once you’ve done that, go back and do all of the moderately-hard ones, the ones you can get with some serious thought.   Cross off the circles as you answer them, so that you can easily scan for the not-yet-done ones on later passes.  The very-hard ones should not get time wasted on them until you’ve done the moderately-hard ones.

Yes, I know, if you’re taking a computer-adaptive test (where it insists that you answer each question because it’s adjusting the difficulty level of the next question based on whether you got this one right), you may not be able to use this strategy, but if you can, it is a huge benefit.  Note, by the way, that on some computer-administered tests, you can skip forward and go back as you wish.  If so, then use the scratch paper to keep track of the item numbers you have skipped and cross them off as you get them dealt with.

Know the scoring rules of the test. If there is no penalty for guessing, you should make sure to answer every question even if all you’re doing is bubbling randomly in the last minute.  If there is a penalty for guessing (typically -1/(n-1) where n is the number of choices, such that a purely random guessing pattern would result in a score of zero), you need to get a bit more strategic.   Some people are good guessers — they guess above chance.  If you’re one of those people who guesses at or above chance, again, you should always guess on every item.

However, some people are not good guessers, and actually guess below chance, typically because they’re getting caught up by those attractor answers.  The usual advice to guess if you can eliminate even one answer as definitely correct is wrong, or at least oversimplified. If you can eliminate one or even two out of four choices as definitely wrong, but then choose the wrong answer of the remaining choices often enough that your guesses are below chance overall (that is, if you work your way down from four choices to two but then still pick the wrong answer more than 3/4 of the time), then you’re still guessing below chance.  “Almost right” or, “It was my second choice,” doesn’t count (this is, as the proverb says, neither horseshoes nor hand grenades).  You need to gather data on your own guessing patterns to know whether guessing is an advantageous strategy for you.  This is a great use for the Big Thick Book.

Furthermore, if you’re a bad guesser, or even if you’re a decent one, study what is tripping you up when you guess wrong.  What are the traps you’re getting caught in?  Can you create specific rules and checklists for yourself to make sure you don’t forget about them?  For example, when I’m doing quantitative comparisons, I always check to see what happens if the variables have values of 0, 1, -1, some other negative number, and a fraction between 0 and 1, trying to find a situation where the obvious answer is wrong.    When I do reading comprehension tests, I always read the questions first, and then read the passage with a pencil in hand so I can mark it up.   Stuff like that.

What are the specific skills or content areas that they seem to always throw at you and you always forget?  How can you make sure you get it into your head long enough to write it on the scratch paper as soon as the test starts?  I’m a hawk when it comes to cheating, but even I don’t think it’s cheating if you write a “cheatsheet” out of your head during the test.   I knew one dyslexic young man who could not for the life of him memorize the quadratic formula, but he was a great conceptual thinker and could remember easily how to derive it.  He got to the point where he could rederive that thing in thirty seconds flat on the scratch paper (and he went on to major in mathematics in college).

Anyone have any special tried-and-true techniques that work for them for studying for bubble tests?  Inspirational stories on how you destroyed a stupid test that had been making you miserable?  Post them below!

Advertisements
  1. Jeff
    December 23, 2010 at 10:00 am

    I’m a high school chemistry teacher. One of the strategies that has helped my AP students is, for questions they missed, to write a brief (bullet points, not complete sentences) description of:

    * what the question was trying to assess
    * what trap they fell into
    * what made the “correct” answer correct

    Actually having to distill the crux of the question and the trap down into a couple of bullet points and write them down helps reinforce the distinctions they need to internalize to be successful on the test. Areas that they have more trouble with end up getting more reinforcement.

    • December 25, 2010 at 2:07 am

      Thanks, Jeff! I like the idea of keeping the structure simple and forcing people to really focus on the essential bit of content and/or language they were missing. Great strategy.

  2. December 23, 2010 at 10:10 am

    I never “studied” for multiple-guess tests, nor did I develop any specific strategies. All such tests I’ve taken seem easy enough that if you know the material they are supposed to be testing, the tests themselves are pretty easy. (My son seems to have much the same response, scoring over 700 on both Math and Critical Reading on the SAT in 6th grade.)

    • December 25, 2010 at 2:06 am

      I agree that there is no substitute for knowing the material being tested. Jeff’s suggestion is particularly helpful in terms of getting kids to recognize what the material is that they think they know that they perhaps don’t quite know as well as they think they do.

      I think some people, like you and your son and me, rather easily and naturally, just intuit the unwritten rules. I remember when I was taking an IQ test at age 5, and I remember thinking even then on one item about “what the person who wrote the test probably wanted me to say.” But a lot of people who know the material quite well still don’t think that way without guidance and practice — hence the need for preparation.

  3. Deb Porter
    December 23, 2010 at 10:47 am

    Thanks for this. Spot on, and very helpful.

  4. Judy
    December 23, 2010 at 2:24 pm

    Hi Aimee,

    My 2E son wanted to get into the John Hopkins Study of Exceptional Talent (SET). This meant a 700 on the SAT Math or Reading before turning 13. I got him the Kaplan web based SAT prep course from Homeschool Buyer’s Coop for $49. He completed the course by working at it on and off over a 6 month period. He also did some tests out of the Official SAT Study Guide the week before the test. Six months in advance of the test date, I had the school counselor send in an official form requesting small group administration, extra breaks, and a large block answer sheet (no bubbles to fill in). These were all approved. I also requested that he be allowed to use a dry erase board as scratch paper. This was denied. He got a 740 on the reading and a 720 on the Math, so it all worked out well. I agree with you that test strategies and test specific practice are very helpful. Also, when asked, we say that his scores mean that he is very logical and good at taking multiple choice tests. We do not assume that this outcome is a measure of much beyond that.

    Judy

    • December 25, 2010 at 2:16 am

      Basically, yeah, these tests typically don’t measure a whole lot of anything that anyone actually cares about. The talent-search folks use them because they’re widely accessible and not very expensive (either for the talent search (because the test companies pay to have the tests developed and normed, and the students pay to have them administered and scored) or for the students (because compared to other options available to everyone around the country regardless of local law or policy, these are very inexpensive).

      It’s good that your son was able to get accommodations. Typically the College Board and other authorities are very very strict about these. I’m not sure if they’re more lenient with the talent-search kids. Anyone who is planning to ask for accommodations, it really helps to (1) already be getting the kind of accommodations you’re asking for in an IEP or 504 plan (2) have a recent (past few years) formal assessment (from the school or a private practitioner) in hand, showing the need for those (3) start early — it can take months to get things agreed to.

  5. Helen
    December 25, 2010 at 1:53 am

    I think high scores on tests such as the SAT are pretty good indicators of one particular type of cleverness. People who read very fast and sum up information very quickly are greatly favored; those skills do indeed come in very handy in most academic situations. But I would never say that low scores were definitive of anything, given the enormous number of reasons people might score lower than they should.

    I’m an anxiety-prone person in general, but SATs were one area where I never felt I had to worry or prepare at all — to me that was the whole point, that you could just show up and react. (I think one of my teachers did spend part of a class period on testing hints, right before the PSAT, but it was just the most basic stuff.)

    • December 25, 2010 at 2:29 am

      This gets at a central question about testing, that of validity. Basically, just because we test something in a certain way, that doesn’t necessarily mean that we defined what it was that we were interested in correctly, that we created a test that tested the thing we actually thought we were testing, that the test is in any way useful for predicting any future behavior, etc, etc. I am planning to put together a series of posts about basic concepts in assessment and test types and such… watch the blog for that…

      In this specific case, yes. Bubble tests consist of questions largely loading the lower ends of Bloom’s Taxonomy, even though the content itself may be quite advanced. You have a very short period of time to process the question and formulate a response, or, more properly, to choose a response from among the very small menu of options. The questions rarely have anything to do with each other, so you don’t have to hold much in your working memory — you do one and then you’re done, move along. There are, I suppose, some fields of study or professions in which that style is advantageous. But if you’re trying to test for the kinds of things kids need to be able to do well to succeed in college, it doesn’t seem to be all that well-aligned.

      • December 25, 2010 at 3:02 am

        I’m curious about whether there are any tests that align well with success in college, and whether they are affordable.

        I have seen some figures that suggest that the SAT II tests correlate better with first-year college success than the other tests commonly given for admission to colleges (which made me particularly irritated with the recent changes in University of California admissions practices, which moved away from the better data sources towards the less predictive ones, in the interest of increasing the number of first-generation college students).

      • December 27, 2010 at 11:58 am

        I’m not sure if much of anything will predict “success in college,” in no small part because the underlying populations are so heterogeneous, as are the definitions of “success” and “college.” That is, not that “some colleges aren’t real colleges,” but “successful in computer science in the highly competitive University of California system” and “successful at a small liberal arts seminar school” and “successful in a pre-professional associates degree program at a community college” are all quite different from each other, and that doesn’t even begin to touch the heterogeneity of the colleges.

        This is a situation where the admissions folk at any given school would do well to develop their own local norms — instead of trying to figure out which test predicts whatever the standardized-test-developers thought was the right way to define “success” for all kids at all colleges, they should think about what they think of as defining success for their own students, and then look retrospectively at the admissions data from the past however-many years to see which markers are the most empirically well-correlated with success in their own actual experience with those kids.

        I think that they should also disaggregate the data for the first-in-family kids, because there is such a huge amount of stuff about what you’re supposed to do in order to get into a school and what you’re supposed to do in order to stay in that school and do well. Stuff that was totally obvious to me, because, well, I learned it at my PhD daddy’s knee, was completely unknown to a lot of the adolescents / young adults I’ve had professional relationships with (teacher, therapist, etc). I would predict that for this specific group, (1) the best-predictor set of variables are going to be different from those which are the best predictors for the general population, and (2) even more than the before-they-get-here variables, they need to look at the variables around what kinds of supports they provide for those kids after they get on campus.

      • Jeff
        December 26, 2010 at 10:01 pm

        This is a bit of a tangent, but at last summer’s AP Annual Conference, the regional coordinator (I think that’s her title) for Massachusetts remarked that the single best predictor of first-year college success seems to be last semester senior year grades in high scohol. Evidently the student’s study habits that are in place during the transition from high school to college (“senior slide” versus “stay on target”) play a substantial role in shaping the student’s study habits as the student adjusts to the new set of academic and social opportunities and expectations.

        Getting back to the subject of testing, I’d say that with the recent emphasis on testing and test preparation, test scores are becoming less and less of a predictor of college success. An interesting study of ratings of college professors at the Air Force Academy showed that there was a positive correlation between the students’ ratings of a professor and the students’ performance on the final exam for that course (which was common to all sections), but a *negative* correlation between both of these and their performance in the next level course in the same subject. Evidently, the popular professors were preparing students for the final exam, whereas the unpopular professors were teaching the subject.

        As another example (even though I’ll be the first to point out that “anecdote” is not the singular of “data”) I have seen the same trend with my AP students–one of mine from last year earned a 2 on the AP Chemistry exam last year, and scored 140/188 on an incoming assessment of her depth of knowledge of the topics covered in college freshman chemistry. Her roommate, who had scored a 4 on the AP Chem exam scored 80/188 on the same incoming assessment.

      • December 27, 2010 at 12:02 pm

        Interesting, on all counts! (If you have the references to any of the studies from the first two paragraphs, let me know.)

      • Jeff
        December 27, 2010 at 12:37 pm

        The Massachusetts College Board Liaison’s name is Heather Ayres . The way she described the statistic made it sound like she should have data or a reference to back it up.

        I’ll have to ask my ex-stepbrother-in-law if he has specific information about the Air Force Academy study.

      • December 29, 2010 at 4:10 am

        Thanks!

  6. December 25, 2010 at 4:28 am

    gasstationwithoutpumps :
    I’m curious about whether there are any tests that align well with success in college, and whether they are affordable.

    For example, is there any data on CWRA http://www.cae.org/content/pro_collegework.htm ?

    • December 27, 2010 at 12:28 pm

      Hm… that looks quite interesting. Looks like they’re doing the research as they go… hope they publish it at some point. The test’s predecessor (for college kids, the Collegiate Learning Assessment), has some references, but mostly in non-research-journal sources — gets some positive feedback, some not-so-sure.

  7. HelenS
    December 25, 2010 at 9:57 am

    I can’t actually think of an academic field where reading quickly and being able to extract the most relevant information quickly would *not* be useful. I can think of fields where reading and processing information slowly is less of a handicap than it would be in, say, English or history, but that’s quite different.

    • December 27, 2010 at 12:00 pm

      Good point. It’s not that it’s never a useful thing, but that it misses so much of what we also care about.

  8. Allan
    December 29, 2010 at 3:06 am

    Although I liked what you wrote, I do have a few nits to pick on minor points:

    1) I heartily agree that poor performance on standardized tests is not in and of itself a “learning handicap.” However, I have known several people whose mild but significant learning handicap as assessed on the conventional tests first showed up as a problem with standardized tests. Some kids are smart enough to think their way around their handicap in some situations. So, a student who, for example is the top student in Honor’s Physics and AP Calculus in the school but scores below 300 on the SAT math (I had this kid last year) probably should strongly consider being evaluated for a learning handicap, especially after eliminating the possibilities you mentioned.

    2) Your audience on this blog is mostly the well educated. The parents among this group push their kids. That is not typical. I, being like you, used to give advice of “buy the big book.” But most, not just some, but most, kids won’t take that seriously. So, having the class in which the testing coach stands over the shoulder–although superfluous for you or me–is probably effective for most students.

    Thanks for the post–I look forward to the promised, more comprehensive one on testing.

    Allan

    • December 29, 2010 at 4:10 am

      Allan, absolutely. I often have to remind people that no one tells people with subtle learning disabilities that they have to do badly on things — they’re usually trying to compensate as well as they can, and often are not themselves aware of how much they’re not able to do reliably. So often the kids make it to middle school or high school (and sometimes even college) before anyone realizes they have problems. And yes, as you say, sometimes the bubble tests are the canary in the coal mine. Anytime an obviously bright kid is persistently struggling, it’s worth figuring out what’s going on.

      I agree that many kids will not study without the adult standing over them. If the goal is to get the kid’s scores on the test to be as high as possible, and the family has the means to provide the test-specific class, then I certainly wouldn’t say not to do it. It’s likely to be more effective in terms of raising the kid’s score on the test.

      But I am skeptical of the long-term value of such a strategy. If a kid needs long-term work with a coach in order to teach them how to regulate their own study habits, and the parents can afford to provide it, then I think that’s a better route. The kid who cannot get themselves to study for the high-stakes bubble test is also likely to be the kid who cannot get themselves to get to class and do homework reliably in college, where no one is looking over your shoulder. And I hate to see folks piling up money in tuition and student loan debt and such when the kid really is not ready for the big leagues. So the intervention shouldn’t be “get them to do well on this test,” but “get them better at self-regulating so that they can both do well on the test and do well in the program they’re trying to get into.”

      • December 31, 2010 at 1:04 am

        I’ve seen several students not get diagnosed until grad school, some not until they were re-entry students in their 50s. We’re not talking little problems here either, but things like bipolar disorder, serious ADHD, and suicidal depression.

        For some students, the PhD thesis is the first challenge big enough that they couldn’t compensate.

  9. Helen
    December 31, 2010 at 2:52 am

    Aimee Yermish :
    Anytime an obviously bright kid is persistently struggling, it’s worth figuring out what’s going on.

    One of the many, many problems with the way high schools are set up is that it has become NORMAL for high-ability students (and others) to have to struggle to some extent: either they’re struggling to get classes that are challenging enough, they’re struggling socially/emotionally (because anyone would be depressed in the environment they find themselves in), or they’re drowning in meaningless requirements (sometimes all three and more). In such an environment, struggles due to things like learning disabilities or ADD too often just get lost in the noise. Not to mention that people sometimes don’t see the difference between “struggle” and “work” (that post-Puritan ethic that says there has to be suffering involved or it doesn’t count).

  1. April 26, 2011 at 5:07 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: