Soon it'll be time for kids to start taking tests and for educators to start administering them -- all under a cloud of suspicion and mistrust about the tests' security. So I thought I'd pass along this Q and A with test security expert John Fremer, who talks about how big,
how complicated, how dangerous cheating prevention efforts can be
(people have been killed!), who cheats (everyone but state agencies,
apparently), and high-tech tactics that might be used to prevent
cheating on the "common core" assessments" (can you say "Linear On The
Fly Testing"?). What's your take on test security -- too much, too little, the wrong kind? Read below and share your thoughts.
Q: How big is the testing industry, in terms of tests or dollars per year? What about the test security industry?
JF: The testing industry is tens of billions of dollars annually
with many facets - test development, test administration, test scoring,
test program management, test preparation, etc. The test security
industry is much smaller and much of the expenditure is buried within
other contracts. Overall it is probably in the tens of millions of
Q: What would you consider a normal, average amount of cheating each year?
JF: My guess is that one to two percent of educators have some
involvement in cheating when they play a major role in high stakes
testing. This is a very low rate compared to many other fields, although
some fields have even lower levels of cheating.
Q: Education's not the worst? What fields have higher rates of cheating, and which have lower?
JF: The Information Technology area has very significant problems, in
part because a substantial number of their test takers take exams in
other countries around the world. The legal testing arena has
relatively low rates of testing misbehavior, in good measure because
they exercise very firm control of the testing process.
Q: What did you find when you looked into cheating allegations in Atlanta?
JF: We confirmed that there were some schools and some classes
within schools where the results strongly suggest testing misbehavior on
the part of educators. In Atlanta, looking at the school with the
greatest number of inexplicable results, the probability of those
results occurring by chance is one over ten to the 52nd power. I liken
this to flipping two coins and having the first land and stay on its
edge and the second landing and staying on the edge of the first.
Q: What if anything do you think you or the state or
district could have done to prevent the dispute over your findings
that's taking place with the Governor?
JF: I think things would have gone more smoothly if I had arranged
to meet directly with the Governor's Office of Student Achivement to be
absolutely certan that our data requests and the reasons for them
where spelled out in the clearest possible way. As to the Atlanta Public
Schools, I have recommended that they get permission to run their own
cheating detection analyses to give them more ability to focus on what
specific kinds of problems are occurring and where.
Q: How's your relationship with Governor's Office of Student Achievement ED Kathleen Mathers these days?
JF: People seem intrigued by the disagreement that I have with
Kathleen Mathers about Caveon not getting all the data we asked for and
with the outcome of Caveon and KPMG's analyses and with our conclusions.
Still, I truly applaud her for carrying out cheating analyses and then
calling for a thorough review of what accounted for the results that her
vendor came up with. Kathleen's activities are quite commendable and in
the forefront of how a state testing leader should address possible
Q: Is most cheating done by kids, teachers, school-level staff, or district or state staff?
JF: Cheating is found in all of educational testing, including
teacher-made tests which account for the majority of all testing in
education. In these situations cheating by students is the most common;
there's no reason for teachers to cheat.
Q: What about in school- district- or state-wide testing situations?
JF: Student cheating may still be the most prevalent, but attention
tends to focus on educator cheating, of which most occurs at the school
level, some at the district level, and almost none to my knowledge at
the state level.
Q: Is most cheating done on high-stakes tests that affect graduation or on lower stakes testing that affects school ratings?
JF: Cheating occurs on all high stakes tests, regardless of the
purpose. The degree of cheating is very much impacted by the way the
program is managed. Very careful monitoring, clear testing rules,
explicit consequences, systematic cheating analyses, etc. when done
carefully and thoroughly can keep cheating to a quite low level. It is
extremely difficult to completely avoid cheating and very expensive to
Q: Are there more cheating problems now or is it just that there's more testing and more coverage in the media under NCLB?
JF: I believe that there is more cheating now. There is also more
media coverage, but I think that is partly because there is more
cheating to cover. It is true that the public's appetite for cheating
stories seems very strong.
Q: What's the worst test security breach or cheating scenario you've ever seen or heard about?
JF: We track stories about test cheating from around the world and
proctors have been killed while trying to maintain test security.
Q: What's the worst breach over the past year or two?
JF: In a certification testing program outside of education, an
entire computer-delivered test item bank was stolen before even one
person had taken the test for real.
Q: What field did that happen in, and was it reported to the public?
JF: It was Information Technology. I don't think that there was a public report about it.
Q: Will the common core of standards and the new tests coming with them make things worse or better in terms of cheating?
JF: The incentive to compromise the tests will go up because of the
widespread use of common test items so the return to "test pirates" will
go up. Also the same test items will be given in many different
places. Unless the cooperating states prepare conscientiously and
follow through with great skill and focus, cheating will go up
substantially. Fortunately we have time to get this right.
Q: What will be required to keep the common tests secure?
JF: In some content areas, test items can be composed as they are
being administered, sometimes called "Linear on the Fly Testing." In
this approach there are no test items to steal in advance and the ones
administered will not be used again exactly as is. This is a powerful
anti-cheating approach where it can be applied.
A 35-year ETS veteran who's now Executive Vice President
of Caveon Testing Security, Fremer is an
increasingly familiar name in news coverage of the testing industry --
including most recently the hotly disputed instances of cheating in the
Atlanta Public School system, in which Governor Sonny Perdue and others
believe that cheating was much more widespread than Caveon and other
have found it to be (see AJC's Get Schooled for some background). Caveon works with 15 states and the DoD.
Cross-posted from TWIE