February 2017
TESOL HOME Convention Jobs Book Store TESOL Community


ARTICLES
INCREASING AUTHENTICITY IN ESP AND LSP ASSESSMENT: INSIDERS' VIEWS
Margaret van Naerssen, Independent Consultant, Pennsylvania, USA

We know that involving workplace/subject matter specialists (insiders) is a best practice in English for ESP/Language for Specific Purposes (LSP) when we develop customized training for the needs of specific populations. As Douglas (2001) notes, when doing needs analyses we commonly mine the target situation for target language use (TLU), but we can do more.

This article revisits Jacoby’s 1998 concept of Indigenous Assessment Criteria (IAC) which shows us how insiders’ intuitive insights can help us drill deeper into the communicative situation for greater authenticity in our ESP/LSP programs and assessments (Douglas, 2000, 2001).

What Are Indigenous Assessment Criteria?

In his very thorough article on IAC and language testing, Douglas (2001) asks where IAC come from. Let’s begin with Jacoby’s initial observations.

Linguist Sally Jacoby, collecting data for her doctoral research (1998), sat in on conference presentation rehearsals by physicists (native and nonnative speakers) to study the evaluative feedback on presentations by each other and by mentors. She noticed that feedback was not on the linguistic aspects of accuracy and style, commonly used as performance assessment criteria in second language testing (Jacoby & McNamara, 1999).

Instead—professional standards (criteria) of excellence were generally applied across the board, irrespective of the presenter’s native/nonnative speaker status. Comments were part of professional development. What is acceptable communication in a specific community? What is valued in the professional culture? Such feedback promotes professional socialization (Jacoby, 1998; also on socialization see van Naerssen & Brennan, 1993).

Drawing on her ethnolinguistic perspective, Jacoby classified these types of intuitive comments as “indigenous assessment criteria” (IAC). These are criteria used by subject or workplace specialists when assessing communication performances of apprentices in the specialty field. Staying within an ethnolinguistic perspective, “insider” has become a more accessible synonym for “indigenous,” at least in the context of assessment: insider assessment criteria (IAC).

Jacoby drew criteria for assessment of communication skills from community-based individual interactions, not from linguists/testers imposing standards from outside. She went beyond even the formal standards for performance within a specific job context (e.g., supervisor checklists, professional standards). She captured informal evaluative comments regarding performance (Douglas, 2000, 2001).

What Interested the Language Testing Community About IAC?

The value of identifying IAC was quickly recognized by the international language testing community, especially by like-minded language testing experts working in specific purposes contexts. Douglas (2000, 2001) reports on the work of an Australian language tester who then went on to collaborate with Jacoby, (i.e., Jacoby and McNamara). McNamara had actively employed such an approach in test development in Australia, in work with medical practitioners. Douglas and Myers had studied the criteria used by veterinary professionals in assessing the communication skills of those becoming veterinarians.

How Were IAC Introduced to the TESOL ESP Interest Section Community?

After becoming aware of Jacoby’s ongoing research, the TESOL ESP Interest Section (ESPIS) invited her to present on her work at a TESOL convention in an ESPIS academic session as part of exploring ways of linking research and practice. In 1998, Jacoby also further explained IAC for ESP practitioners.

It is not enough to ask specialists directly about criteria as this may not capture some of the detailed insider criteria a specialist might use. We need to look at what insiders say and do all the time as part of their professional culture.

These insider criteria can be identified through detailed observations of evaluative communication in a workplace/ professional culture between trainers and trainee/ novices/ newcomers to a field or to workplace tasks. The researcher focuses on the routine responses (verbal and non-verbal) a trainer makes to routine workplace/ professional. (TESOL, 2000, Section VI)

On a very sad note, Sally Jacoby died of cancer in 2007. Her work on IAC, however, has been kept alive, especially in the language community, by those concerned with ESP/LSP.

Douglas (2001) notes that IAC are not intended to replace other forms of language assessment. Rather, “our LSP tests will stand a much better chance of being appropriate in the specific purpose context as perceived by subject specialists if they are grounded in assessment criteria derived from an analysis of the target language use (TLU) domain” (p. 185). Jacoby, McNamara, and Douglas also note that as IAC are situation specific, it is more difficult to translate these to assessments that can be valid for wider use without research on their reliability.

Where Else Are IAC Approaches Found?

As occasionally happens with insightful ideas, similar perspectives have developed independently of Jacoby. Below are two ESP cases (Landa and Lockwood, from TESOL, 2000) that illustrate other places where IAC-types of insights have been found. No doubt others also may have independently taken such a perspective.

Landa (TESOL, 2000), in his ESP work with pharmacists, looked at the judging in the Annual Counseling Competition sponsored by the American Pharmaceutical Association. He examined the criteria used in competition to promote effective communications with pharmacy clients.

Lockwood (TESOL, 2000) reported on accessing supervisor feedback on agent–customer telephonic interactions. In this case study, we can see how insider feedback, which reflected good customer service, was incorporated into both training and language testing.

A large bank in a major world city has a number of foreign customers. English is the primary language used with these customers. The bank had telephone recordings of such interactions which could be examined. After numerous communication problems, the bank assigned special telephone lines and customer service agents to handle such customers. A training program was proposed.

The workplace language testing specialists worked closely with the supervisors to incorporate, into the testing instrument, what they, as professionals, felt were the criteria reflecting good customer service. Teams of raters included a supervisor and a workplace language trainer.

The above examples come from personal communications for case studies which appeared

in the 2000 TESOL Best Practices in Workplace Language Training document. Unfortunately, these were not included in the shorter Effective Practices in Workplace Language Training, published by TESOL in 2003 and 2014.

What Might IAC Look Like?

Below are some examples of IAC on the overall performance in physicists’ presentations. Some may apply in other fields as well, but they may also differ. There may also be individual variation among the insider evaluators.

  • Designing visuals to accompany the talk which are coherent and legible
  • Keeping to the time limit
  • Avoiding verbosity
  • Articulating the significance of the topic to the profession
  • Stating arguments and labeling visuals clearly
  • Stating arguments and labeling visuals clearly
  • Making effective, convincing arguments
    (Source: Table 1, Jacoby and McNamara, 1999, as cited in Douglas 2001: 177).

How Does an IAC Approach Apply to ESP/LSP Program Development?

As can be seen in Lockwood’s (TESOL, 2000) example, use of an insider assessment criteria approach is not restricted to language testing. In strong ESP/LSP program development, ideally identifying IAC should become a part of doing an initial needs analysis. This then provides the input for the development of criterion-based performance objectives, including prioritized IAC. These criterion-based objectives then shape the training based on real-life situations. Assessment of performance in such an ESP/LSP program would be based on the criterion-based objectives, which would include the prioritized IAC. We all would agree that there should be a match between what is taught and what is tested.

Concluding Remarks

As time passes in our professional fields, too often valuable insights from our past are lost or forgotten. Also, similar insights might appear in different forms as “new discoveries.” To provide a more coherent sense of our profession, it is valuable to revisit important insights that might still apply today. While use of IAC still lives in ESP/LSP testing, I hope the insiders’ intuitive views are also remembered when we do needs analyses in our ESP/LSP program development. Collaboration between ESP/LSP practitioners and workplace/subject matter specialists should be a key practice for increasing authenticity in our programs. Thus, I felt it would be valuable to revisit Jacoby’s concept of indigenous (insider) assessment criteria (IAC) to consider how it can enrich our work.

References

Douglas, D. (2000). Assessing languages for specific purposes. Cambridge, United Kingdom: Cambridge University Press.

Douglas, D. (2001). Language for specific purposes assessment criteria: Where do theycome from? Language Testing,18(2), 171–185.

Jacoby, S. (1998). Science as performance: Socializing scientific discourse through the conference talk rehearsal. (Unpublished doctoral dissertation). University of California, Los Angeles, Los Angeles, California.

Jacoby, S., & McNamara, T. (1999). Locating competence.English for Specific Purposes, 18(3), 213–41.

TESOL Workplace Language Training Task Force. (2000). Section VI: Conduct an Instructional Needs Assessment, Best Practice 4. TESOL Best Practices in Workplace Language Training, unpublished document. TESOL International Association: Alexandria, Virginia.

van Naerssen, M. & Brennan, M. (1993). Language socialization in professional cultures: Language for Specific Purposes. Cahiers de APLIUT. Paris: Association des professeurs de langues des instituts universitaires de technologies.


Margaret van Naerssen (PhD, applied linguistics) is an EFL/ESL teacher trainer and a linguistics consultant in forensic cases involving nonnative English speakers. She was the third ESPIS chair and the codirector of the English for Science & Technology Center in Beijing (UCLA & Chinese Academy of Sciences). She enjoys teamwork on needs analyses and materials development.

« Previous Newsletter Home Print Article Next »
In This Issue
LEADERSHIP UPDATES
ARTICLES
ABOUT THIS COMMUNITY
Tools
Search Back Issues
Forward to a Friend
Print Issue
RSS Feed
Like What You See? Want to Share Your Story?
Please consider submitting an article for the next newsletter. We want to know what is going on in your corner of the world! Submit your idea to Co-Editor, Kevin Knight.