Iowa State University
College of Liberal Arts and Sciences
E-Mail & Phones |
  • Testing the testers

    English professors working to verify effectiveness of new TOEFL listening test.

  • Each year over 1 million international students take the TOEFL (Test of English as a Foreign Language) with the expressed desire to study at an institution of higher learning in the United States.

    For the past 10 years, Educational Testing Service (ETS), a Princeton, N.J.-based firm, has been working to modify the test for Internet usage. The new TOEFL will now not only include listening, reading and writing components but speaking as well.

    That's where a pair of Department of English professors at Iowa State come in.

    Dan Douglas, professor of English, and Volker Hegelheimer, assistant professor of English, have received a two-year, $124,000 grant from ETS to investigate the cognitive processes and use of language and content knowledge by test takers who use the new TOEFL listening test tasks.

    "They (ETS) want to know if the TOEFL does what they want it to do," said Douglas, who has consulted with ETS for several years. "(In our study), we have the students verbalize their thoughts while taking the listening test to see if they are using language skills the way the test developers thought they would."

    While developing the new TOEFL listening section, ETS tape-recorded actual classroom lectures, student conversations and student office visits with faculty to form the basis to write scripts. Professional actors then audio recorded the scripts that are used in the new TOEFL.

    Students taking the test listen to a 3-5 minute lecture or conversation (they can take notes) and then answer five or six questions. The process is repeated several times with different scripts.

    "Listening is a very important skill to have as a student," Douglas said. "Students spend a lot of time listening in lectures. You may not have to speak the language well in class as a student, but you sure have to be able to listen."

    In their study, Douglas and Hegelheimer are interacting with the new TOEFL listening test tasks using three methods:

    *Verbal protocol analysis of participants' test-taking processes;
    *Analysis of screen capture of their online behavior, and;
    *Content analysis of their handwritten notes.

    A pilot test was conducted last spring with 12 Iowa State students from three different language groups (Chinese, Korean and Spanish) in a variety of academic disciplines.

    "This test is much more realistic than the previous test," Douglas said. "It uses academic language that people actually would say on a college campus."

    Using sophisticated technology, Douglas and Hegelheimer are able to capture every keystroke on the computer. A video recording takes special note of the test taker's facial expression and note taking.

    "Tentative results from the pilot study indicate that taking notes makes a difference," Hegelheimer said. "So far, most of the students take notes but those who actually refer to their notes while answering the questions tend to do better on the test."

    "That's somewhat surprising since previous ETS studies didn't show that," Douglas said.

    Douglas and Hegelheimer presented their initial results from the pilot test over the summer at conferences in Canada and Madison, Wis. A full-scale test will be conducted this calendar year on campus.

Dan Douglas and Volker  Hegelheimer

Dan Douglas and Volker Hegelheimer

Around LAS

October 3-16, 2005