English professors working to verify effectiveness of new TOEFL
listening test.
Each year over 1 million international students take the TOEFL (Test
of English as a Foreign Language) with the expressed desire to study at
an institution of higher learning in the United States.
For the past 10 years, Educational Testing Service (ETS), a Princeton, N.J.-based
firm, has been working to modify the test for Internet usage. The new TOEFL
will now not only include listening, reading and writing components but
speaking as well.
That's where a pair of Department of English professors at Iowa State come
in.
Dan Douglas, professor of English, and Volker Hegelheimer, assistant professor
of English, have received a two-year, $124,000 grant from ETS to investigate
the cognitive processes and use of language and content knowledge by test
takers who use the new TOEFL listening test tasks.
"They (ETS) want to know if the TOEFL does what they want it to do,"
said Douglas, who has consulted with ETS for several years. "(In our
study), we have the students verbalize their thoughts while taking the listening
test to see if they are using language skills the way the test developers
thought they would."
While developing the new TOEFL listening section, ETS tape-recorded actual
classroom lectures, student conversations and student office visits with
faculty to form the basis to write scripts. Professional actors then audio
recorded the scripts that are used in the new TOEFL.
Students taking the test listen to a 3-5 minute lecture or conversation
(they can take notes) and then answer five or six questions. The process
is repeated several times with different scripts.
"Listening is a very important skill to have as a student," Douglas
said. "Students spend a lot of time listening in lectures. You may
not have to speak the language well in class as a student, but you sure
have to be able to listen."
In their study, Douglas and Hegelheimer are interacting with the new TOEFL
listening test tasks using three methods:
*Verbal protocol analysis of participants' test-taking
processes; *Analysis of screen capture of their online behavior,
and; *Content analysis of their handwritten notes.
A pilot test was conducted last spring with 12 Iowa State students from three
different language groups (Chinese, Korean and Spanish) in a variety of academic
disciplines.
"This test is much more realistic than the previous
test," Douglas said. "It uses academic language that people actually
would say on a college campus."
Using sophisticated technology, Douglas and Hegelheimer are able to capture
every keystroke on the computer. A video recording takes special note of the
test taker's facial expression and note taking.
"Tentative results
from the pilot study indicate that taking notes makes a difference,"
Hegelheimer said. "So far, most of the students take notes but those
who actually refer to their notes while answering the questions tend to do
better on the test."
"That's somewhat surprising since
previous ETS studies didn't show that," Douglas said.
Douglas and Hegelheimer presented their initial results from the pilot test
over the summer at conferences in Canada and Madison, Wis. A full-scale test
will be conducted this calendar year on campus.
