From Longwood to Kámospatak and Back: The Letter Writing Task of the 2001 English Entrance Test
Horváth József
University of Pécs, Dpt of English Applied Linguistics
A story
Wednesday, May 23, 2001. In the morning, the written test of the English entrance exam gets underway. Candidates receive their test papers, which include a component called Multiple Choice Test and another one, Composition. For the former, they have two hours, and its maximum score is 75 points. For the latter, they have one hour, and its maximum score is 25. Dictionaries may be used for the composition task.
When they are finished with the first task, candidates begin work on the letter writing part. The instructions say:
PART B – COMPOSITION
Read the letter from your imaginary English pen-friend carefully, then WRITE A REPLY TO IT (300-350 words).
WHAT YOU NEED TO KNOW FOR A SUCCESSFUL COMPLETION OF THE TASK:
1. The names in the letter have both male and female versions, you can choose either;
2. Your address is: 3176 Kámospatak, Hegyalja u. 11.
3. The three paragraphs in the letter deal with separate topics. You are expected to write about the topics in paragraphs 2 and 3 at greater length than about the one in paragraph 1.
Let’s consider what candidates may do now. First, they have received a letter from their imaginary pen-friend. Some may have such pals, other may not. Whatever the case, as the reply letter is expected to be written to an imaginary friend, the nature of their friendship is not made concrete by the task as this is not specified by the rubrics or, as we shall see, by the input letter.
Second, they have to write a letter of 300-350 words. How candidates will go about ensuring that their letters will not exceed or go below the range is hard to say – some will surely count the words in each paragraph as they draft their answers.
Third, they are told what they need for success. The air becomes electrified as candidates read this section. They get to know that there are three prerequisites for success – that is, passing the writing test. I will examine each of these requirements.
Names in the letter, candidates are told, can be either male or female names. As we will see later, there are seven names in the input letter: Elm, Castle, Longwood, England, Dani, Süni, and the English pen-friend, Les. Fine, some candidates may say. England and Süni, then, may be either male or female names. They do not know yet how this will be helpful, but they may consider it.
Another aspect of success is the address. If I know this I will be successful, some candidates may mumble to themselves, and they readily put the address in the top right corner of their answer sheets – the same place where the English pen-friend’s address appears.
And there is one more hint – about the topics of the input letter. Candidates have to say less about the first topic than about the second and third. This info, too, registers in many candidates’ mind, and now they can start the exploration of the input letter.
Testing writing
But before we take that path with them and discover what they are doing and how they are doing it, let me stop here to consider the framework of the writing task. As is clear to anyone who has prepared students for the entrance exam or those of us who have read candidates’ scripts, this task is a direct test of writing that aims, according to the specifications, to elicit language use for a communicative purpose. It is part of the written exam besides the language use and reading components (subsumed under the name Multiple Choice Test because these parts use the MC test technique) because universities need valid and reliable information on how candidates perform on a productive test.
The validity of the test has not been doubted, but when it comes to reliability, there are problems (Heltai, 1996, pp. 54-56; about entrance requirements, see Halápi & Király, 1999). As this is a subjectively marked test, using human raters, we cannot assume, without evidence to the contrary, that the dozens, maybe even scores of markers working at various universities will mark the same scripts the same way or that each of them would mark the same scripts the same way a second time. Strangely, no large-scale inter-rater and intra-rater reliability studies have been conducted on what many consider the highest-stakes exam of English in Hungary. But such an investigation will have to be carried out sooner rather than later – the entrance exam as we know it today will change in 2005 when the new school-leaving exam is introduced.
In this paper, I will not deal with intra- and inter-rater reliability. I will, instead, focus on the construct and content validity of the test in arguing that the rubrics and the input letter of the 2001 entrance exam had serious flaws (see also Horváth, 1997 for another aspect of the validity of an earlier entrance exam). I will also aim to assess the functionality of the assessment scale. My aim: to help contribute to a better test in 2002 and beyond. (Those interested in the underlying theory of testing writing will find Grabe & Kaplan, 1996 a useful resource.)
We can now return to the candidates and to our discovery of what they are doing.
Two letters, the worlds
The candidates now begin reading the input letter, to which they have to respond. It was sent by the pen-friend, Les, who had this to say:
Elm Cottage
25, Castle Road
Longwood
LA4 7JK
England
18 May 2001
Dear Dani,
I was sorry to hear from Süni (I hope I’ve spelt it right this time) that you’ve just come out of hospital. What exactly happened to you? Süni didn’t seem to know the details. I remember how uncomfortable I felt when I had to spend three days in hospital when I was having my tonsils out. I hope it was a less upsetting experience for you!
As you know, I write articles for the school magazine and my editor has just asked me to write one with the title ‘The New Man and the New Woman of the Third Millennium’. I am supposed to collect ideas on what young people think male and female ideals of the future will be like and also the reason for their preferences. I would appreciate it if you would share your ideas with me on this subject – I need to hear views from as many people as possible and I have very little contact with your part of the world, unfortunately.
You won’t believe it, but I’ve started to learn Spanish! I’ve been going to Spain on holiday with my parents for the past four years and now, at last, I’ve decided I’m fed up with just being the ‘strong silent type’ when it comes to communicating with the locals. The problem is I seem to be making slow progress, or is it that I’m just too impatient? As you are such an experienced foreign language learner yourself, could you give me a few tips on how I could become as proficient in Spanish as you are in English?
I’ve got to finish now. We’ll have a poetry-reading session at school at 5 p.m. Do write soon. I’m really looking forward to hearing from you.
Cheers,
Les
Having carefully read the letter, candidates begin to underline key words in various paragraphs so they can respond. Knowing what the three steps to success are, an imaginary candidate may write the following script. Note that this is a letter I have written, on the basis of what I have read in the rubrics, the input letter, and the assessment scale.
Kámospatak
Hegyalja u. 11.
3176
Hungary
23 May 2001
Dear Les,
I was sorry to hear that you felt uncomfortable when you had to spend three days in hospital when you were having your tonsils out. It was an upsetting experience for you! I felt uncomfortable, too, when I had to spend six days in hospital when I was having my appendix out. It was a lousy experience for me, too.
As you know, I also write features for the school journal, and my editor has just asked me to write one with the title ‘The Old Man and the Old Woman of the Second Millennium’. What coincidence! Maybe we should write an article together. I am supposed to collect ideas on what young people think male and female ideals of the past were like and also the reason for their ideas. I would appreciate it if you would share your ideas with me on this subject – I need to hear views from as many people as possible, and I have very little contact with your part of the world, unfortunately. I will give you a buzz and we can talk about it.
You won’t believe it, but I’ve started to study German! I’ve been going to Germany on holiday with my buds for the past five years and now, at last, I’ve decided I’m fed up with just being the ‘strong silent type’ when it comes to communicating with the locals. The problem is I seem to be making slow progress, or is it that I’m just too stupid? I know you speak German, and you know I speak Spanish besides English. Let’s do a net meeting one day. I will speak in Spanish, and you will shoot the breeze in German. It’ll be a hell of an experience.
I’ve got to finish now. We’ll have a prose-reading session at school at 6 p.m. Do write soon. I’m really looking forward to hearing from you.
Cheers,
Dani
Spot the differences
There are several shared features between the two letters. The themes are the same, the language is mostly the same. One of the main differences is in length – the input text is 307 words, the output text is 326.
After I had typed up Les’s letter, I studied the assessment scale and was struck by how little was said in it about the achievement of the task. Also, as the input text is so long, I was wondering whether candidates would not lift some parts into their own answers. And thus began my discovery that the input letter and the assessment scale were such that they allowed for the engagement of the copying strategy. As I will show, my letter meets most requirements of the task – in fact, if my own judgment of my own letter is any reliable indication, I would have scored 20 points out of the 25. (But do not take my word for it – I will provide those details in just a moment).
What is striking about my letter is that it has little, if any, interactive value at first glance – ideas and questions raised by Les are addressed but never fully engaged with. The interaction between Les and Dani is minimal. Yet there is interaction, and thus the letter can be regarded as a reply.
I have just passed this segment of the entrance exam – and the way I have done it is simply by copying several bits from the input text and performing some basic transformation techniques. The style is mostly informal, with the vocabulary displaying a wide range and accuracy. Sentence and clause structures are spot on, paragraphs indented, coherence present. I write about the themes mentioned in Les’s letter: there is some discussion of hospitalization, of male and female ideals, and of language learning.
Remember that in the task we know very little about the relationship of Les and Dani – we don’t even know their genders, which is why you will see all these awkward “s/he”s here. How would Les react to a letter like this? Maybe s/he will realize that s/he is getting this letter because s/he did something like this in a previous letter to Dani, when s/he did not fully address Dani’s questions. There are several ifs and maybes and perhapses. The reason – the task is in a vacuum, and it is placed there such that it is only Dani’s task to respond. What about Les? Why doesn’t s/he respond to what Dani has written?
The power relationship appears stereotypical and unequal: Les, the English pen-friend is requesting information – and gives little on her/his own accord. We get to know about the tonsil operation and the poetry reading session. But these two events are simply there in the letter because Les wanted to write about them – not because Dani has asked Les to discuss these topics. The two main topics, ideals and language learning, are the bits that are supposed to elicit information from Dani – whose role is that of the student on the periphery who may help the person in the center write an article. Note the discourse that displays this unequal power relationship. First, the parenthetical remark about the spelling of “Süni”; second, the reference to “your part of the world.”
What I am saying is this – when spotting the difference between the two letters and making a decision on how I, the hypothetical candidate, performed on the task, you will need to consider how I react to this letter – in the real world of the world, and the real world of the test. Both are realities – the former is my daily experience and values; the latter: my current experience and what I am trying to do with my values.
Hamp-Lyons’s point is worth noting:
Only recently have we come to understand that all writing ... is personal. Each writer brings the whole of himself or herself to the task at hand. In interpreting a task and creating a response to it, each writer must create a “fit” between his or her own world and the world of the essay test topic.... If [the process of attending to task expectations] breaks down, the writer will replace the task with a different or a related one, but will not respond to the topic intended. (1990, p. 77)
Consider this: I wanted to sound sarcastic. In being told that my letter was to be written to an “imaginary English pen-friend,” I imagined a person that is narrow-minded and never answers my questions. And when I saw Les’s letter, I did see traces of this persona: there are no answers to questions in a letter to which Les now responds – and the weak joke about spelling “Süni” and the reference to my “part of the world” make me choose the strategy: Hey, Les, cut this out, will ya? I’m a real person who’s had enough of your communication that focuses on me giving you advice. Why pretend you need it? How on earth are you gonna use my views of the ideal men and women in the third millennium? Wait a minute! What is the third millennium? Who’s gonna be around for as long as that? We can’t test whatever we say, can we? Is your editor crazy or somethin’? Was there one ideal of men and women in the second millennium? Surely there were hundreds, thousands, maybe millions. And where is this ideal supposed to be? In what culture? In Hungary? Europe? Asia?
Again, maybe all I wanted to communicate to Les was that I was being sarcastic. As markers, university teachers would have no way of knowing – candidates do not insert footnotes to their test letters telling markers, “I wrote this piece because I felt this input letter was making fun of my culture.” No. What candidates do is try to understand the task and do as best they can to achieve success – based on what they are told about how success can be achieved.
The letter writing task of the 2001 entrance exam is one that calls attention to the need to reform it – as I have shown so far, in terms of its rubrics and input letter. But there is a third aspect to factor in: how the scripts are assessed.
1199 words
I have no evidence that any one candidate employed the strategy I have outlined. I do have evidence of a lot of lifting and of the results of a small-scale analysis of the marks given by nine raters at the English Departments of the University of Pécs. I obtained 53 of the 244 test papers taken in Pécs in 2001. As the sample is small for any conclusive conclusions, what I will report here will be the raw rank order of the marks awarded on the seven assessment criteria applied in the analytic scoring.
The criteria and their corresponding mark ranges were the following:
1. Purpose and audience: 0 to 2
2. Vocabulary and spelling: 0 to 5
3. Structures: 0 to 5
4. Organisation, layout and punctuation: 0 to 5
5. Content and task achievement. Topic 1: candidate’s hospital treatment: 0 to 2
6. Content and task achievement. Topic 2: male and female ideals of the future: 0 to 3
7. Content and task achievement. Topic 3: successful language learning: 0 to 3
The 53 scripts’ marks showed the following rank order (given in percentages of maximum marks):
Purpose, audience and style 86.75
Content and task achievement. Topic 1 70.5
Content and task achievement. Topic 2 59
Content and task achievement. Topic 3 55.66
Organisation, layout and punctuation 52.8
Vocabulary and spelling 48.2
Structures 44
As the 53 scripts were randomly chosen, the percentage scores and the rank order should be treated with caution; with a different set of 53 scripts or with all the scripts, I may have found different results. However, it seems clear that the tendency is to perform higher on criteria that had less weight – note that purpose and the three content point criteria have two or three as maximum points. The three criteria whose maximum marks were 5 appear at the bottom of the rank order list.
Also, within the content points, these 53 candidates had a higher chance of getting higher marks on the first topic, which they were supposed to treat in less detail than the other two.
The result that over 80 percent of the maximum mark was awarded for purpose, audience, and style is indication that, assuming high intra- and inter-rater reliability, candidates generally do not have a problem with writing informal letters.
But how were these marks given in the rating sessions?
A three-page document, “Marking instructions,” accompanied the candidates’ scripts as they were delivered from the entrance exam committee to the department. I saw no validation mark on any of these instructions: they were photocopied booklets with no identification of where they were issued. No stamps or other proof of validity appeared on the marking instruction copy I used.
What is equally dubious is the functionality of the instructions. It is a 1199-word document with an introduction, the seven criteria, the bands, and their descriptors. For lack of space, I cannot reproduce all of the scale here. Instead, I will include one band and its descriptor for each criterion so you can decide if the letter I wrote meets the requirements for those categories. Ready? Here are the bands that I think my letter would have been assigned by markers. If you agree with the mark, appearing before each descriptor, put a tick next to it. If you disagree, cross out the mark.
Purpose, audience and style
2 Predominantly informal style with an occasional formal element in vocabulary possible. The conventions of letter writing fully observed.
Vocabulary and spelling (0-5 points)
5 Excellent vocabulary throughout the letter; a high degree of lexical sophistication; accurate use of a wide range of vocabulary items; practically no inaccuracies; fully idiomatic. Spelling is accurate and error-free.
Structures (0-5 points)
5 Practically error-free; max. 3 minor inaccuracies. Consistent and appropriate variety of sentence structure; short and long, simple and complex sentences as appropriate for the informal style of the letter.
Organisation, layout and punctuation (0-5 points)
5 Fully coherent; well-organised; there is a smooth flow to the text; correct use of paragraphs, linking words and pronouns; no inaccuracies; accurate punctuation.
Content and task achievement – Topic 1: candidate’s hospital treatment (0-2 points)
1 Topic described or dealt with in some detail but with not enough sophistication.
Content and task achievement – Topic 2: male and female ideals of the future (0-3 points)
1 Topic described in some detail but with little sophistication; hardly any arguments are supported.
Content and task achievement – Topic 3: successful language learning (0-3 points)
1 Topic described in some detail but with little sophistication; hardly any arguments are supported.
I have given my script 20 points, out of the total 25. This is an 80 percent score, which is rather high when we consider that I hardly wrote anything myself and thus have given little idea to the marker about my writing ability. If you agree with all marks, well, we have high inter-rater reliability (or maybe not – my marks may have influenced your rating). If you have disagreed with a few or all marks, it would be good to know whether you think the marks should be even higher or lower than mine. Please send your comments to the editors of NyelvInfo or address them direct to me at my joe@btk.pte.hu email address.
But whether we agree or disagree on the marks, one aspect of these bands seems clear: there is no mention in the marking instructions (or the rubrics) of lifting. Words and ideas and full sentences can be incorporated in the answer with no penalty. Remember – as a candidate, I was not told that I was not supposed to do it – all I was told were those three hints about what will make my work successful (names having male and female versions, my address, and topic coverage in the three paragraphs).
The only criteria where lifting could have been spotted and penalized are the three task achievement categories. Yet none of the descriptors for topic 1 or topics 2 and 3 consider this issue. For a more detailed analysis, let’s look at what the marker was to follow when assessing the second topic, ideals:
6. Content and task achievement – Topic 2: male and female ideals of the future (0-3 points)
Does the letter include discussion of the topic raised in the second paragraph of the input letter?
0 Topic ignored or dealt with in one or two short sentences, using simple and/or short inaccurate language.
1 Topic described in some detail but with little sophistication; hardly any arguments are supported.
2 Topic described in greater detail with some sophistication; most of the arguments are supported.
3 Topic dealt with fully; all the arguments are supported; preferably, some counter-arguments are also anticipated and refuted.
A zero had to be assigned when the topic was ignored or it was dealt with only in one or two sentences that had simple or inaccurate structure. (Note that this idea in itself is problematic because the marker has already assessed structures. This overlap does not add to the value and functionality of the scale.) I agree that I aimed at no sophistication at all when writing about ideals in my letter – but I described the topic is “some” detail and made sure I supported no arguments. As I never intended in an informal letter to support arguments or to describe the topic in much more detail, I would be content with band 1, knowing that I have already earned the points that I wanted to earn.
To illustrate the style and content of the 2001 marking instructions, I have performed a word frequency analysis and have this to report, indicating a need to make them more rater friendly.
The 1199-word document really contains only 336 different words – showing very high levels of redundancy.
The ten most frequently occurring content words are the following (with frequencies in parentheses):
topic (20)
errors (17 )
vocabulary (14)
inaccuracies (13)
severe (12)
punctuation, variety (10)
used (9)
arguments, spelling (8)
More troubling than redundancy is the construct of the scale. Organization, layout, and punctuation, for example, are assessed as one single criterion, even though there is no evidence that so diverse elements of the writing ability would form a coherent unit. People who can, for example, punctuate their texts well by following a standard known to them may have difficulty with the coherence of their texts. In other words, even though one may well be able to punctuate the chunk “this is a very interesting idea isnt it” one may not be able to make this idea a coherent part of a larger unit of text. In fact, when considering the first three bands (0, 1, and 2) of this criterion, one could argue that it is perfectly possible that a script would be assigned 0 marks on its coherence (“coherence is inadequate”), 1 mark on its layout (“paragraphs used but link between the various sentences/paragraphs often unclear”) and 2 marks on punctuation (“only minor punctuation errors”). Yet the texts I have quoted in parentheses appear for different bands under this collective of organization, layout, and punctuation. How was this criterion used in the marking? I have serious doubts about the reliability of the marks on it.
One more aspect of this problem needs to be discussed before, after all this criticism, I put forth suggestions. This aspect is the idea of connecting paragraphs in this particular task. As you will remember, the candidate read Les’s letter. If you took the time to thumb through back to that part of the article, you would see that there is absolutely no thematic or cohesive connection between any two of her/his paragraphs either. No wonder – pen-friends hardly ever produce such discourse. In fact, we could argue that it is precisely this lack of inter-paragraph cohesion and coherence that makes a letter – an informal letter. And to suggest, as the marking instructions do, that for a top mark for this criterion the candidate needs to produce a “smooth flow of text” is to indicate that what is stated for one criterion (purpose, audience, and style) is negated in another. How could anyone connect, in a real letter to real a friend, the topic of hospital treatment to male and female ideals, and this latter to successful language learning? I could, if I really wanted to – but why would I? These are not my ideas, requiring my own planning and development. Rather, they are answers to an inauthentic text that was written to elicit language about three different topics to provide enough vocabulary range. The input text is not coherent on this level.
I have developed an alternative assessment scale to address some its current flaws. Clearly, keeping over 1,000 words in mind or going over them again and again when evaluating a 300-word script will not work very reliably. The redundancies of the wording of the descriptors can be avoided if a team of markers work together and study other forms of assessment.
May 23, 2002
We don’t know yet when the new entrance exam written test will be given – what I suggest is that it would be a good idea to make that test more valid and reliable. Part of any such reform will have to address the marking instructions – they will have to be simplified for markers and its essential elements made available to candidates so they know how their scripts will be assessed.
To provide impetus for that reform, I have written this article and will provide recommendations. Besides, I have developed a framework for an alternative assessment scale that may be used as a starting point for designing an instrument that can be used in such large-scale assessment as the entrance exam.
The rationale is this. The scale should
be easy to use
be introduced after marker training
contain only positive statements about a script
be based on a theoretically sound construct of writing.
The first draft of the alternative scale is given below – in the form of a checklist. I am not suggesting it is the perfect solution. Far from it. But it may be the beginning of a solution. The scale is for the 2001 letter task and includes as much as possible of the original instructions – even the requirement that there be transition between paragraphs, a feature we have seen may be questionable in this type of text.
Alternative marking instructions
These instructions are in a checklist form. The list is divided into two parts. The first part is for the content, the second for the formal and stylistic features of the letter.
Part One. Consider the following features in relation to the relevant sections (paragraphs) of the letter. Each feature represents a positive aspect of writing. Put a tick next to a feature if it is present in the relevant section of the letter. See the example (Q):
Q Handwritten letter
-----------
XA Hospital treatment dealt with
XB Hospital treatment described in detail
YA Ideals dealt with
YB Ideals described in detail
YC Arguments for ideals
ZA Successful language learning dealt with
ZB Successful language learning described in detail
ZC Arguments for successful language learning
If there are fewer than three ticks, don’t assess the letter further.
Part Two. Put a tick next to each of the following features if it characterizes at least half the letter. Each feature is positive.
AA Basic verbs
AB Idiomatic verbal phrases
BA Basic nouns
BB Idiomatic noun phrases
C Wide range of vocabulary
D Accurate spelling
EA Simple structures
EB Complex structures
EC Accurate structures
FA Appropriate paragraphs
FB Link between sentences
FC Link between paragraphs
FD Fully coherent
FE Effective cohesive devices
FF Smooth text
G Effective communication
H Informal elements
I have piloted this checklist format on two colleagues. I have asked them to evaluate three scripts and to evaluate the scale itself. The two markers, who have had a long and wide teaching and marking experience, have reported that the format appeared rater-friendly and suggested that it may be useful after modifications and marker training.
Recommendations for the writing task next year
The way the writing task has been designed is in line with its specifications (Országos Felsõoktatási Felvételi Iroda, 2001). It does include one or two tasks – in the past several years there has always been one task. It could be considered whether two shorter tasks should be introduced, after piloting. There has been a lot of debate about why piloting for entrance exam tasks is impractical (Dávid, 1996, p. 82)– none of which seems relevant for the writing component, and has in fact been carried out within the school-leaving exam reform (cf. Szabó, Gróf & Alderson, 2000). There are many colleagues teaching writing skills at high schools, colleges and universities in this country – they would welcome the opportunity of integrating pilot tasks into their syllabuses and thus contribute to the construct and content validity of the exam. A letter writing task and one that tests, for example, skills in interpreting a chart for a specific audience will provide more information on candidates’ skills than does a single task.
As regards the letter task, the specifications for the rubrics, the input text, and the assessment scale have to be re-designed. The construct of writing abilities has to be defined and corresponding tasks and texts developed and piloted (Burch, 1999; Weir, 1990). As a minimum requirement, the input text has to be significantly shorter than the output text. Besides the problem of lifting, a long input text violates validity concerns; that is, if the task is to test writing abilities, how much reading can be regarded as not interfering with the direct test of writing? (Alderson, Clapham & Wall, 1995, p. 60). The rubrics should contain information about what markers look for. This is easier than one would think. Currently, the answer sheet is a blank sheet of paper. At the bottom of the reverse side of it, the criteria could be printed next year so that candidates knew what aspects of their scripts are evaluated. This information is no secret – but is revealed to them only after they have taken the orals. Then, if they wish to inspect their papers and consider appealing the marking, they are shown the scale, complete with descriptors for bands. The criteria and their individual weights can be, I would think, given to them when they most need them: as they plan and draft and finalize their letters.
The input text could be made more authentic. One way to achieve this would be by searching for teenagers’ and students’ magazines, of which there are plenty, and choosing a few real writing situations. For example, one could look for the advice columns or competitions of these magazines and invite candidates to write to those audiences. If this is done, more control may need to be introduced for task achievement – as we have seen, candidates could have passed, with flying colors, the 2001 writing task even if they did little more than copying.
As far as the assessment scale is concerned, we need to begin work on how to make it more functional and thus more reliable. Redundancy in the wording has to be trimmed down – or rather, avoided altogether. The cognitive load of a 1,000-word scale applied to a 300-word script is high indeed. Also, overlaps between criteria are to be checked for and eliminated.
The new school-leaving exam will issue a passport to universities for candidates who have earned the necessary points. But it is still in the works. Until 2005, there will be thousands and thousands of candidates who will seek entry to the English programs of Hungarian universities. Who gets in and how is partly determined by the writing task. Let’s return to Longwood and Kámospatak to see what is going on in the communication of Leses and Danis so that future candidates may benefit from our work.
References
Alderson, J. C., Clapham, C., & Wall, D. (1995). Language test construction and evaluation. Cambridge: Cambridge University Press.
Burch, C. B. (1999). Writing for your portfolio. Boston: Allyn & Bacon.
Dávid, G. (1996). The entrance exam: Lifting the shroud. Novelty, 3 (3), 75-87.
Grabe, W., & Kaplan, R. W. (1996). Theory and practice of writing: An applied linguistic perspective. London: Longman.
Halápi M., & Király Zs. (1999). University and college entrance requirements. In H. Fekete, É. Major, & M. Nikolov (Eds.), English language education in Hungary: A baseline study (pp. 55-62). Budapest: British Council Hungary.
Hamp-Lyons, L. (1990). Second language writing: Assessment issues. In B. Kroll (Ed.), Second language writing: Research insights for the classroom (pp. 69-87). Cambridge: Cambridge University Press.
Heltai P. (1996). That infamous school leaving/entrance test. Novelty, 3 (3), 52-59.
Horváth J. (1997). “Typing matters”: On the face validity of the 1997 entrance test. Testing Matters, 2 (3), 17-22.
Országos Felsõoktatási Felvételi Iroda. (2001). Honlap [Online]. Available www.felvi.hu
Szabó, K., Gróf, Sz., & Alderson, C. (2000). The writing test. In J. C. Alderson, E. Nagy & E. Öveges (Eds.), English language education in Hungary: Examining Hungarian learners’ achievements in English (pp. 100-118). Budapest: British Council Hungary.
Weir, C. J. (1990). Communicative language testing. New York: Prentice Hall.
Horváth József
University of Pécs, Dpt of English Applied Linguistics
A story
Wednesday, May 23, 2001. In the morning, the written test of the English entrance exam gets underway. Candidates receive their test papers, which include a component called Multiple Choice Test and another one, Composition. For the former, they have two hours, and its maximum score is 75 points. For the latter, they have one hour, and its maximum score is 25. Dictionaries may be used for the composition task.
When they are finished with the first task, candidates begin work on the letter writing part. The instructions say:
PART B – COMPOSITION
Read the letter from your imaginary English pen-friend carefully, then WRITE A REPLY TO IT (300-350 words).
WHAT YOU NEED TO KNOW FOR A SUCCESSFUL COMPLETION OF THE TASK:
1. The names in the letter have both male and female versions, you can choose either;
2. Your address is: 3176 Kámospatak, Hegyalja u. 11.
3. The three paragraphs in the letter deal with separate topics. You are expected to write about the topics in paragraphs 2 and 3 at greater length than about the one in paragraph 1.
Let’s consider what candidates may do now. First, they have received a letter from their imaginary pen-friend. Some may have such pals, other may not. Whatever the case, as the reply letter is expected to be written to an imaginary friend, the nature of their friendship is not made concrete by the task as this is not specified by the rubrics or, as we shall see, by the input letter.
Second, they have to write a letter of 300-350 words. How candidates will go about ensuring that their letters will not exceed or go below the range is hard to say – some will surely count the words in each paragraph as they draft their answers.
Third, they are told what they need for success. The air becomes electrified as candidates read this section. They get to know that there are three prerequisites for success – that is, passing the writing test. I will examine each of these requirements.
Names in the letter, candidates are told, can be either male or female names. As we will see later, there are seven names in the input letter: Elm, Castle, Longwood, England, Dani, Süni, and the English pen-friend, Les. Fine, some candidates may say. England and Süni, then, may be either male or female names. They do not know yet how this will be helpful, but they may consider it.
Another aspect of success is the address. If I know this I will be successful, some candidates may mumble to themselves, and they readily put the address in the top right corner of their answer sheets – the same place where the English pen-friend’s address appears.
And there is one more hint – about the topics of the input letter. Candidates have to say less about the first topic than about the second and third. This info, too, registers in many candidates’ mind, and now they can start the exploration of the input letter.
Testing writing
But before we take that path with them and discover what they are doing and how they are doing it, let me stop here to consider the framework of the writing task. As is clear to anyone who has prepared students for the entrance exam or those of us who have read candidates’ scripts, this task is a direct test of writing that aims, according to the specifications, to elicit language use for a communicative purpose. It is part of the written exam besides the language use and reading components (subsumed under the name Multiple Choice Test because these parts use the MC test technique) because universities need valid and reliable information on how candidates perform on a productive test.
The validity of the test has not been doubted, but when it comes to reliability, there are problems (Heltai, 1996, pp. 54-56; about entrance requirements, see Halápi & Király, 1999). As this is a subjectively marked test, using human raters, we cannot assume, without evidence to the contrary, that the dozens, maybe even scores of markers working at various universities will mark the same scripts the same way or that each of them would mark the same scripts the same way a second time. Strangely, no large-scale inter-rater and intra-rater reliability studies have been conducted on what many consider the highest-stakes exam of English in Hungary. But such an investigation will have to be carried out sooner rather than later – the entrance exam as we know it today will change in 2005 when the new school-leaving exam is introduced.
In this paper, I will not deal with intra- and inter-rater reliability. I will, instead, focus on the construct and content validity of the test in arguing that the rubrics and the input letter of the 2001 entrance exam had serious flaws (see also Horváth, 1997 for another aspect of the validity of an earlier entrance exam). I will also aim to assess the functionality of the assessment scale. My aim: to help contribute to a better test in 2002 and beyond. (Those interested in the underlying theory of testing writing will find Grabe & Kaplan, 1996 a useful resource.)
We can now return to the candidates and to our discovery of what they are doing.
Two letters, the worlds
The candidates now begin reading the input letter, to which they have to respond. It was sent by the pen-friend, Les, who had this to say:
Elm Cottage
25, Castle Road
Longwood
LA4 7JK
England
18 May 2001
Dear Dani,
I was sorry to hear from Süni (I hope I’ve spelt it right this time) that you’ve just come out of hospital. What exactly happened to you? Süni didn’t seem to know the details. I remember how uncomfortable I felt when I had to spend three days in hospital when I was having my tonsils out. I hope it was a less upsetting experience for you!
As you know, I write articles for the school magazine and my editor has just asked me to write one with the title ‘The New Man and the New Woman of the Third Millennium’. I am supposed to collect ideas on what young people think male and female ideals of the future will be like and also the reason for their preferences. I would appreciate it if you would share your ideas with me on this subject – I need to hear views from as many people as possible and I have very little contact with your part of the world, unfortunately.
You won’t believe it, but I’ve started to learn Spanish! I’ve been going to Spain on holiday with my parents for the past four years and now, at last, I’ve decided I’m fed up with just being the ‘strong silent type’ when it comes to communicating with the locals. The problem is I seem to be making slow progress, or is it that I’m just too impatient? As you are such an experienced foreign language learner yourself, could you give me a few tips on how I could become as proficient in Spanish as you are in English?
I’ve got to finish now. We’ll have a poetry-reading session at school at 5 p.m. Do write soon. I’m really looking forward to hearing from you.
Cheers,
Les
Having carefully read the letter, candidates begin to underline key words in various paragraphs so they can respond. Knowing what the three steps to success are, an imaginary candidate may write the following script. Note that this is a letter I have written, on the basis of what I have read in the rubrics, the input letter, and the assessment scale.
Kámospatak
Hegyalja u. 11.
3176
Hungary
23 May 2001
Dear Les,
I was sorry to hear that you felt uncomfortable when you had to spend three days in hospital when you were having your tonsils out. It was an upsetting experience for you! I felt uncomfortable, too, when I had to spend six days in hospital when I was having my appendix out. It was a lousy experience for me, too.
As you know, I also write features for the school journal, and my editor has just asked me to write one with the title ‘The Old Man and the Old Woman of the Second Millennium’. What coincidence! Maybe we should write an article together. I am supposed to collect ideas on what young people think male and female ideals of the past were like and also the reason for their ideas. I would appreciate it if you would share your ideas with me on this subject – I need to hear views from as many people as possible, and I have very little contact with your part of the world, unfortunately. I will give you a buzz and we can talk about it.
You won’t believe it, but I’ve started to study German! I’ve been going to Germany on holiday with my buds for the past five years and now, at last, I’ve decided I’m fed up with just being the ‘strong silent type’ when it comes to communicating with the locals. The problem is I seem to be making slow progress, or is it that I’m just too stupid? I know you speak German, and you know I speak Spanish besides English. Let’s do a net meeting one day. I will speak in Spanish, and you will shoot the breeze in German. It’ll be a hell of an experience.
I’ve got to finish now. We’ll have a prose-reading session at school at 6 p.m. Do write soon. I’m really looking forward to hearing from you.
Cheers,
Dani
Spot the differences
There are several shared features between the two letters. The themes are the same, the language is mostly the same. One of the main differences is in length – the input text is 307 words, the output text is 326.
After I had typed up Les’s letter, I studied the assessment scale and was struck by how little was said in it about the achievement of the task. Also, as the input text is so long, I was wondering whether candidates would not lift some parts into their own answers. And thus began my discovery that the input letter and the assessment scale were such that they allowed for the engagement of the copying strategy. As I will show, my letter meets most requirements of the task – in fact, if my own judgment of my own letter is any reliable indication, I would have scored 20 points out of the 25. (But do not take my word for it – I will provide those details in just a moment).
What is striking about my letter is that it has little, if any, interactive value at first glance – ideas and questions raised by Les are addressed but never fully engaged with. The interaction between Les and Dani is minimal. Yet there is interaction, and thus the letter can be regarded as a reply.
I have just passed this segment of the entrance exam – and the way I have done it is simply by copying several bits from the input text and performing some basic transformation techniques. The style is mostly informal, with the vocabulary displaying a wide range and accuracy. Sentence and clause structures are spot on, paragraphs indented, coherence present. I write about the themes mentioned in Les’s letter: there is some discussion of hospitalization, of male and female ideals, and of language learning.
Remember that in the task we know very little about the relationship of Les and Dani – we don’t even know their genders, which is why you will see all these awkward “s/he”s here. How would Les react to a letter like this? Maybe s/he will realize that s/he is getting this letter because s/he did something like this in a previous letter to Dani, when s/he did not fully address Dani’s questions. There are several ifs and maybes and perhapses. The reason – the task is in a vacuum, and it is placed there such that it is only Dani’s task to respond. What about Les? Why doesn’t s/he respond to what Dani has written?
The power relationship appears stereotypical and unequal: Les, the English pen-friend is requesting information – and gives little on her/his own accord. We get to know about the tonsil operation and the poetry reading session. But these two events are simply there in the letter because Les wanted to write about them – not because Dani has asked Les to discuss these topics. The two main topics, ideals and language learning, are the bits that are supposed to elicit information from Dani – whose role is that of the student on the periphery who may help the person in the center write an article. Note the discourse that displays this unequal power relationship. First, the parenthetical remark about the spelling of “Süni”; second, the reference to “your part of the world.”
What I am saying is this – when spotting the difference between the two letters and making a decision on how I, the hypothetical candidate, performed on the task, you will need to consider how I react to this letter – in the real world of the world, and the real world of the test. Both are realities – the former is my daily experience and values; the latter: my current experience and what I am trying to do with my values.
Hamp-Lyons’s point is worth noting:
Only recently have we come to understand that all writing ... is personal. Each writer brings the whole of himself or herself to the task at hand. In interpreting a task and creating a response to it, each writer must create a “fit” between his or her own world and the world of the essay test topic.... If [the process of attending to task expectations] breaks down, the writer will replace the task with a different or a related one, but will not respond to the topic intended. (1990, p. 77)
Consider this: I wanted to sound sarcastic. In being told that my letter was to be written to an “imaginary English pen-friend,” I imagined a person that is narrow-minded and never answers my questions. And when I saw Les’s letter, I did see traces of this persona: there are no answers to questions in a letter to which Les now responds – and the weak joke about spelling “Süni” and the reference to my “part of the world” make me choose the strategy: Hey, Les, cut this out, will ya? I’m a real person who’s had enough of your communication that focuses on me giving you advice. Why pretend you need it? How on earth are you gonna use my views of the ideal men and women in the third millennium? Wait a minute! What is the third millennium? Who’s gonna be around for as long as that? We can’t test whatever we say, can we? Is your editor crazy or somethin’? Was there one ideal of men and women in the second millennium? Surely there were hundreds, thousands, maybe millions. And where is this ideal supposed to be? In what culture? In Hungary? Europe? Asia?
Again, maybe all I wanted to communicate to Les was that I was being sarcastic. As markers, university teachers would have no way of knowing – candidates do not insert footnotes to their test letters telling markers, “I wrote this piece because I felt this input letter was making fun of my culture.” No. What candidates do is try to understand the task and do as best they can to achieve success – based on what they are told about how success can be achieved.
The letter writing task of the 2001 entrance exam is one that calls attention to the need to reform it – as I have shown so far, in terms of its rubrics and input letter. But there is a third aspect to factor in: how the scripts are assessed.
1199 words
I have no evidence that any one candidate employed the strategy I have outlined. I do have evidence of a lot of lifting and of the results of a small-scale analysis of the marks given by nine raters at the English Departments of the University of Pécs. I obtained 53 of the 244 test papers taken in Pécs in 2001. As the sample is small for any conclusive conclusions, what I will report here will be the raw rank order of the marks awarded on the seven assessment criteria applied in the analytic scoring.
The criteria and their corresponding mark ranges were the following:
1. Purpose and audience: 0 to 2
2. Vocabulary and spelling: 0 to 5
3. Structures: 0 to 5
4. Organisation, layout and punctuation: 0 to 5
5. Content and task achievement. Topic 1: candidate’s hospital treatment: 0 to 2
6. Content and task achievement. Topic 2: male and female ideals of the future: 0 to 3
7. Content and task achievement. Topic 3: successful language learning: 0 to 3
The 53 scripts’ marks showed the following rank order (given in percentages of maximum marks):
Purpose, audience and style 86.75
Content and task achievement. Topic 1 70.5
Content and task achievement. Topic 2 59
Content and task achievement. Topic 3 55.66
Organisation, layout and punctuation 52.8
Vocabulary and spelling 48.2
Structures 44
As the 53 scripts were randomly chosen, the percentage scores and the rank order should be treated with caution; with a different set of 53 scripts or with all the scripts, I may have found different results. However, it seems clear that the tendency is to perform higher on criteria that had less weight – note that purpose and the three content point criteria have two or three as maximum points. The three criteria whose maximum marks were 5 appear at the bottom of the rank order list.
Also, within the content points, these 53 candidates had a higher chance of getting higher marks on the first topic, which they were supposed to treat in less detail than the other two.
The result that over 80 percent of the maximum mark was awarded for purpose, audience, and style is indication that, assuming high intra- and inter-rater reliability, candidates generally do not have a problem with writing informal letters.
But how were these marks given in the rating sessions?
A three-page document, “Marking instructions,” accompanied the candidates’ scripts as they were delivered from the entrance exam committee to the department. I saw no validation mark on any of these instructions: they were photocopied booklets with no identification of where they were issued. No stamps or other proof of validity appeared on the marking instruction copy I used.
What is equally dubious is the functionality of the instructions. It is a 1199-word document with an introduction, the seven criteria, the bands, and their descriptors. For lack of space, I cannot reproduce all of the scale here. Instead, I will include one band and its descriptor for each criterion so you can decide if the letter I wrote meets the requirements for those categories. Ready? Here are the bands that I think my letter would have been assigned by markers. If you agree with the mark, appearing before each descriptor, put a tick next to it. If you disagree, cross out the mark.
Purpose, audience and style
2 Predominantly informal style with an occasional formal element in vocabulary possible. The conventions of letter writing fully observed.
Vocabulary and spelling (0-5 points)
5 Excellent vocabulary throughout the letter; a high degree of lexical sophistication; accurate use of a wide range of vocabulary items; practically no inaccuracies; fully idiomatic. Spelling is accurate and error-free.
Structures (0-5 points)
5 Practically error-free; max. 3 minor inaccuracies. Consistent and appropriate variety of sentence structure; short and long, simple and complex sentences as appropriate for the informal style of the letter.
Organisation, layout and punctuation (0-5 points)
5 Fully coherent; well-organised; there is a smooth flow to the text; correct use of paragraphs, linking words and pronouns; no inaccuracies; accurate punctuation.
Content and task achievement – Topic 1: candidate’s hospital treatment (0-2 points)
1 Topic described or dealt with in some detail but with not enough sophistication.
Content and task achievement – Topic 2: male and female ideals of the future (0-3 points)
1 Topic described in some detail but with little sophistication; hardly any arguments are supported.
Content and task achievement – Topic 3: successful language learning (0-3 points)
1 Topic described in some detail but with little sophistication; hardly any arguments are supported.
I have given my script 20 points, out of the total 25. This is an 80 percent score, which is rather high when we consider that I hardly wrote anything myself and thus have given little idea to the marker about my writing ability. If you agree with all marks, well, we have high inter-rater reliability (or maybe not – my marks may have influenced your rating). If you have disagreed with a few or all marks, it would be good to know whether you think the marks should be even higher or lower than mine. Please send your comments to the editors of NyelvInfo or address them direct to me at my joe@btk.pte.hu email address.
But whether we agree or disagree on the marks, one aspect of these bands seems clear: there is no mention in the marking instructions (or the rubrics) of lifting. Words and ideas and full sentences can be incorporated in the answer with no penalty. Remember – as a candidate, I was not told that I was not supposed to do it – all I was told were those three hints about what will make my work successful (names having male and female versions, my address, and topic coverage in the three paragraphs).
The only criteria where lifting could have been spotted and penalized are the three task achievement categories. Yet none of the descriptors for topic 1 or topics 2 and 3 consider this issue. For a more detailed analysis, let’s look at what the marker was to follow when assessing the second topic, ideals:
6. Content and task achievement – Topic 2: male and female ideals of the future (0-3 points)
Does the letter include discussion of the topic raised in the second paragraph of the input letter?
0 Topic ignored or dealt with in one or two short sentences, using simple and/or short inaccurate language.
1 Topic described in some detail but with little sophistication; hardly any arguments are supported.
2 Topic described in greater detail with some sophistication; most of the arguments are supported.
3 Topic dealt with fully; all the arguments are supported; preferably, some counter-arguments are also anticipated and refuted.
A zero had to be assigned when the topic was ignored or it was dealt with only in one or two sentences that had simple or inaccurate structure. (Note that this idea in itself is problematic because the marker has already assessed structures. This overlap does not add to the value and functionality of the scale.) I agree that I aimed at no sophistication at all when writing about ideals in my letter – but I described the topic is “some” detail and made sure I supported no arguments. As I never intended in an informal letter to support arguments or to describe the topic in much more detail, I would be content with band 1, knowing that I have already earned the points that I wanted to earn.
To illustrate the style and content of the 2001 marking instructions, I have performed a word frequency analysis and have this to report, indicating a need to make them more rater friendly.
The 1199-word document really contains only 336 different words – showing very high levels of redundancy.
The ten most frequently occurring content words are the following (with frequencies in parentheses):
topic (20)
errors (17 )
vocabulary (14)
inaccuracies (13)
severe (12)
punctuation, variety (10)
used (9)
arguments, spelling (8)
More troubling than redundancy is the construct of the scale. Organization, layout, and punctuation, for example, are assessed as one single criterion, even though there is no evidence that so diverse elements of the writing ability would form a coherent unit. People who can, for example, punctuate their texts well by following a standard known to them may have difficulty with the coherence of their texts. In other words, even though one may well be able to punctuate the chunk “this is a very interesting idea isnt it” one may not be able to make this idea a coherent part of a larger unit of text. In fact, when considering the first three bands (0, 1, and 2) of this criterion, one could argue that it is perfectly possible that a script would be assigned 0 marks on its coherence (“coherence is inadequate”), 1 mark on its layout (“paragraphs used but link between the various sentences/paragraphs often unclear”) and 2 marks on punctuation (“only minor punctuation errors”). Yet the texts I have quoted in parentheses appear for different bands under this collective of organization, layout, and punctuation. How was this criterion used in the marking? I have serious doubts about the reliability of the marks on it.
One more aspect of this problem needs to be discussed before, after all this criticism, I put forth suggestions. This aspect is the idea of connecting paragraphs in this particular task. As you will remember, the candidate read Les’s letter. If you took the time to thumb through back to that part of the article, you would see that there is absolutely no thematic or cohesive connection between any two of her/his paragraphs either. No wonder – pen-friends hardly ever produce such discourse. In fact, we could argue that it is precisely this lack of inter-paragraph cohesion and coherence that makes a letter – an informal letter. And to suggest, as the marking instructions do, that for a top mark for this criterion the candidate needs to produce a “smooth flow of text” is to indicate that what is stated for one criterion (purpose, audience, and style) is negated in another. How could anyone connect, in a real letter to real a friend, the topic of hospital treatment to male and female ideals, and this latter to successful language learning? I could, if I really wanted to – but why would I? These are not my ideas, requiring my own planning and development. Rather, they are answers to an inauthentic text that was written to elicit language about three different topics to provide enough vocabulary range. The input text is not coherent on this level.
I have developed an alternative assessment scale to address some its current flaws. Clearly, keeping over 1,000 words in mind or going over them again and again when evaluating a 300-word script will not work very reliably. The redundancies of the wording of the descriptors can be avoided if a team of markers work together and study other forms of assessment.
May 23, 2002
We don’t know yet when the new entrance exam written test will be given – what I suggest is that it would be a good idea to make that test more valid and reliable. Part of any such reform will have to address the marking instructions – they will have to be simplified for markers and its essential elements made available to candidates so they know how their scripts will be assessed.
To provide impetus for that reform, I have written this article and will provide recommendations. Besides, I have developed a framework for an alternative assessment scale that may be used as a starting point for designing an instrument that can be used in such large-scale assessment as the entrance exam.
The rationale is this. The scale should
be easy to use
be introduced after marker training
contain only positive statements about a script
be based on a theoretically sound construct of writing.
The first draft of the alternative scale is given below – in the form of a checklist. I am not suggesting it is the perfect solution. Far from it. But it may be the beginning of a solution. The scale is for the 2001 letter task and includes as much as possible of the original instructions – even the requirement that there be transition between paragraphs, a feature we have seen may be questionable in this type of text.
Alternative marking instructions
These instructions are in a checklist form. The list is divided into two parts. The first part is for the content, the second for the formal and stylistic features of the letter.
Part One. Consider the following features in relation to the relevant sections (paragraphs) of the letter. Each feature represents a positive aspect of writing. Put a tick next to a feature if it is present in the relevant section of the letter. See the example (Q):
Q Handwritten letter
-----------
XA Hospital treatment dealt with
XB Hospital treatment described in detail
YA Ideals dealt with
YB Ideals described in detail
YC Arguments for ideals
ZA Successful language learning dealt with
ZB Successful language learning described in detail
ZC Arguments for successful language learning
If there are fewer than three ticks, don’t assess the letter further.
Part Two. Put a tick next to each of the following features if it characterizes at least half the letter. Each feature is positive.
AA Basic verbs
AB Idiomatic verbal phrases
BA Basic nouns
BB Idiomatic noun phrases
C Wide range of vocabulary
D Accurate spelling
EA Simple structures
EB Complex structures
EC Accurate structures
FA Appropriate paragraphs
FB Link between sentences
FC Link between paragraphs
FD Fully coherent
FE Effective cohesive devices
FF Smooth text
G Effective communication
H Informal elements
I have piloted this checklist format on two colleagues. I have asked them to evaluate three scripts and to evaluate the scale itself. The two markers, who have had a long and wide teaching and marking experience, have reported that the format appeared rater-friendly and suggested that it may be useful after modifications and marker training.
Recommendations for the writing task next year
The way the writing task has been designed is in line with its specifications (Országos Felsõoktatási Felvételi Iroda, 2001). It does include one or two tasks – in the past several years there has always been one task. It could be considered whether two shorter tasks should be introduced, after piloting. There has been a lot of debate about why piloting for entrance exam tasks is impractical (Dávid, 1996, p. 82)– none of which seems relevant for the writing component, and has in fact been carried out within the school-leaving exam reform (cf. Szabó, Gróf & Alderson, 2000). There are many colleagues teaching writing skills at high schools, colleges and universities in this country – they would welcome the opportunity of integrating pilot tasks into their syllabuses and thus contribute to the construct and content validity of the exam. A letter writing task and one that tests, for example, skills in interpreting a chart for a specific audience will provide more information on candidates’ skills than does a single task.
As regards the letter task, the specifications for the rubrics, the input text, and the assessment scale have to be re-designed. The construct of writing abilities has to be defined and corresponding tasks and texts developed and piloted (Burch, 1999; Weir, 1990). As a minimum requirement, the input text has to be significantly shorter than the output text. Besides the problem of lifting, a long input text violates validity concerns; that is, if the task is to test writing abilities, how much reading can be regarded as not interfering with the direct test of writing? (Alderson, Clapham & Wall, 1995, p. 60). The rubrics should contain information about what markers look for. This is easier than one would think. Currently, the answer sheet is a blank sheet of paper. At the bottom of the reverse side of it, the criteria could be printed next year so that candidates knew what aspects of their scripts are evaluated. This information is no secret – but is revealed to them only after they have taken the orals. Then, if they wish to inspect their papers and consider appealing the marking, they are shown the scale, complete with descriptors for bands. The criteria and their individual weights can be, I would think, given to them when they most need them: as they plan and draft and finalize their letters.
The input text could be made more authentic. One way to achieve this would be by searching for teenagers’ and students’ magazines, of which there are plenty, and choosing a few real writing situations. For example, one could look for the advice columns or competitions of these magazines and invite candidates to write to those audiences. If this is done, more control may need to be introduced for task achievement – as we have seen, candidates could have passed, with flying colors, the 2001 writing task even if they did little more than copying.
As far as the assessment scale is concerned, we need to begin work on how to make it more functional and thus more reliable. Redundancy in the wording has to be trimmed down – or rather, avoided altogether. The cognitive load of a 1,000-word scale applied to a 300-word script is high indeed. Also, overlaps between criteria are to be checked for and eliminated.
The new school-leaving exam will issue a passport to universities for candidates who have earned the necessary points. But it is still in the works. Until 2005, there will be thousands and thousands of candidates who will seek entry to the English programs of Hungarian universities. Who gets in and how is partly determined by the writing task. Let’s return to Longwood and Kámospatak to see what is going on in the communication of Leses and Danis so that future candidates may benefit from our work.
References
Alderson, J. C., Clapham, C., & Wall, D. (1995). Language test construction and evaluation. Cambridge: Cambridge University Press.
Burch, C. B. (1999). Writing for your portfolio. Boston: Allyn & Bacon.
Dávid, G. (1996). The entrance exam: Lifting the shroud. Novelty, 3 (3), 75-87.
Grabe, W., & Kaplan, R. W. (1996). Theory and practice of writing: An applied linguistic perspective. London: Longman.
Halápi M., & Király Zs. (1999). University and college entrance requirements. In H. Fekete, É. Major, & M. Nikolov (Eds.), English language education in Hungary: A baseline study (pp. 55-62). Budapest: British Council Hungary.
Hamp-Lyons, L. (1990). Second language writing: Assessment issues. In B. Kroll (Ed.), Second language writing: Research insights for the classroom (pp. 69-87). Cambridge: Cambridge University Press.
Heltai P. (1996). That infamous school leaving/entrance test. Novelty, 3 (3), 52-59.
Horváth J. (1997). “Typing matters”: On the face validity of the 1997 entrance test. Testing Matters, 2 (3), 17-22.
Országos Felsõoktatási Felvételi Iroda. (2001). Honlap [Online]. Available www.felvi.hu
Szabó, K., Gróf, Sz., & Alderson, C. (2000). The writing test. In J. C. Alderson, E. Nagy & E. Öveges (Eds.), English language education in Hungary: Examining Hungarian learners’ achievements in English (pp. 100-118). Budapest: British Council Hungary.
Weir, C. J. (1990). Communicative language testing. New York: Prentice Hall.
posted by Unknown at
Friday, November 23, 2001
