We know that involving workplace/subject matter specialists
(insiders) is a best practice in English for ESP/Language for
Specific Purposes (LSP) when we develop customized
training for the needs of specific populations. As Douglas (2001) notes,
when doing needs analyses we commonly mine the target situation for
target language use (TLU), but we can do
This article revisits Jacoby’s 1998 concept of Indigenous Assessment Criteria (IAC) which shows us how insiders’
intuitive insights can help us drill deeper into the communicative
situation for greater authenticity in our ESP/LSP programs and
assessments (Douglas, 2000, 2001).
What Are Indigenous Assessment Criteria?
In his very thorough article on IAC
and language testing, Douglas (2001) asks where IAC come from. Let’s begin with Jacoby’s initial
Linguist Sally Jacoby, collecting data for her doctoral
research (1998), sat in on conference presentation rehearsals by
physicists (native and nonnative speakers) to study the evaluative
feedback on presentations by each other and by mentors. She noticed that
feedback was not on the linguistic aspects of accuracy and style,
commonly used as performance assessment criteria in second language
testing (Jacoby & McNamara, 1999).
Instead—professional standards (criteria) of excellence were
generally applied across the board, irrespective of the presenter’s
native/nonnative speaker status. Comments were part of professional
development. What is acceptable communication in a specific community?
What is valued in the professional culture? Such feedback promotes
professional socialization (Jacoby, 1998; also on socialization see van
Naerssen & Brennan, 1993).
Drawing on her ethnolinguistic perspective, Jacoby classified
these types of intuitive comments as “indigenous assessment criteria” (IAC).
These are criteria used by subject or workplace specialists when
assessing communication performances of apprentices in the specialty
field. Staying within an ethnolinguistic perspective, “insider” has
become a more accessible synonym for “indigenous,” at least in the
context of assessment: insider assessment criteria (IAC).
Jacoby drew criteria for assessment of communication skills
from community-based individual interactions, not from linguists/testers
imposing standards from outside. She went beyond even the formal
standards for performance within a specific job context (e.g.,
supervisor checklists, professional standards). She captured informal
evaluative comments regarding performance (Douglas, 2000,
What Interested the Language Testing Community About IAC?
The value of identifying IAC was quickly recognized by the
international language testing community, especially by like-minded
language testing experts working in specific purposes contexts. Douglas
(2000, 2001) reports on the work of an Australian language tester who then
went on to collaborate with Jacoby, (i.e., Jacoby and McNamara).
McNamara had actively employed such an approach in test development in
Australia, in work with medical practitioners. Douglas and Myers had studied the criteria used
by veterinary professionals in assessing the communication skills of
those becoming veterinarians.
How Were IAC Introduced to the TESOL ESP Interest Section Community?
After becoming aware of Jacoby’s ongoing research, the TESOL
ESP Interest Section (ESPIS) invited her to present on her work at a
TESOL convention in an ESPIS academic session as part of exploring ways
of linking research and practice. In 1998, Jacoby also further explained
IAC for ESP practitioners.
It is not enough to ask specialists directly about criteria as
this may not capture some of the detailed insider criteria a specialist
might use. We need to look at what insiders say and do all the time as
part of their professional culture.
These insider criteria can be identified through detailed
observations of evaluative communication in a workplace/ professional
culture between trainers and trainee/ novices/ newcomers to a field or
to workplace tasks. The researcher focuses on the routine responses
(verbal and non-verbal) a trainer makes to routine workplace/
professional. (TESOL, 2000, Section VI)
On a very sad note, Sally Jacoby died of cancer in 2007. Her
work on IAC, however, has been kept alive, especially in the language
community, by those concerned with ESP/LSP.
Douglas (2001) notes that IAC are not intended to replace other
forms of language assessment. Rather, “our LSP tests will stand a much
better chance of being appropriate in the specific purpose context as
perceived by subject specialists if they are grounded in assessment
criteria derived from an analysis of the target language use (TLU) domain”
(p. 185). Jacoby, McNamara, and Douglas also note that as IAC are
situation specific, it is more difficult to translate these to
assessments that can be valid for wider use without research on their
Where Else Are IAC Approaches Found?
As occasionally happens with insightful ideas, similar
perspectives have developed independently of Jacoby. Below are two ESP
cases (Landa and Lockwood, from TESOL, 2000) that illustrate other
places where IAC-types of insights have been found. No doubt others also
may have independently taken such a perspective.
Landa (TESOL, 2000), in his ESP work with pharmacists, looked
at the judging in the Annual Counseling Competition sponsored by the
American Pharmaceutical Association. He examined the criteria used in
competition to promote effective communications with pharmacy clients.
Lockwood (TESOL, 2000) reported on accessing supervisor
feedback on agent–customer telephonic interactions. In this case study,
we can see how insider feedback, which reflected good customer service,
was incorporated into both training and language testing.
A large bank in a major world city has a number of foreign
customers. English is the primary language used with these customers.
The bank had telephone recordings of such interactions which could be
examined. After numerous communication problems, the bank assigned
special telephone lines and customer service agents to handle such
customers. A training program was proposed.
The workplace language testing specialists worked closely with
the supervisors to incorporate, into the testing instrument, what they,
as professionals, felt were the criteria reflecting good customer
service. Teams of raters included a supervisor and a workplace language
The above examples come from personal communications for case studies which appeared
in the 2000 TESOL Best Practices in Workplace Language Training
document. Unfortunately, these were not included in the shorter Effective Practices in Workplace Language Training, published by TESOL in 2003 and 2014.
What Might IAC Look Like?
Below are some examples of IAC on the
overall performance in physicists’ presentations. Some may apply in
other fields as well, but they may also differ. There may also be
individual variation among the insider evaluators.
- Designing visuals to accompany the talk which are coherent and legible
- Keeping to the time limit
- Avoiding verbosity
- Articulating the significance of the topic to the profession
- Stating arguments and labeling visuals clearly
- Stating arguments and labeling visuals clearly
- Making effective, convincing arguments
(Source: Table 1, Jacoby and McNamara, 1999, as cited in Douglas 2001: 177).
How Does an IAC Approach Apply to ESP/LSP Program Development?
As can be seen in Lockwood’s (TESOL, 2000) example, use of an insider assessment criteria approach is not restricted to language testing. In strong ESP/LSP
program development, ideally identifying IAC should become a part of
doing an initial needs analysis. This then provides the input for the
development of criterion-based performance objectives, including
prioritized IAC. These criterion-based objectives then shape the
training based on real-life situations. Assessment of performance in
such an ESP/LSP program would be based on the criterion-based
objectives, which would include the prioritized IAC. We all would agree
that there should be a match between what is taught and what is
As time passes in our professional fields, too often valuable
insights from our past are lost or forgotten. Also, similar insights
might appear in different forms as “new discoveries.” To provide a more
coherent sense of our profession, it is valuable to revisit important
insights that might still apply today. While use of IAC still lives in
ESP/LSP testing, I hope the insiders’ intuitive views are also
remembered when we do needs analyses in our ESP/LSP program development.
Collaboration between ESP/LSP practitioners and workplace/subject
matter specialists should be a key practice for increasing authenticity
in our programs. Thus, I felt it would be valuable to revisit Jacoby’s
concept of indigenous (insider) assessment criteria (IAC) to consider how it
can enrich our work.
Douglas, D. (2000). Assessing languages for specific
purposes. Cambridge, United Kingdom: Cambridge University
Douglas, D. (2001). Language for specific purposes assessment criteria: Where do theycome from? Language Testing,18(2), 171–185.
Jacoby, S. (1998). Science as performance: Socializing scientific discourse
through the conference talk rehearsal. (Unpublished doctoral
dissertation). University of California, Los Angeles, Los Angeles,
Jacoby, S., & McNamara, T. (1999). Locating competence.English for Specific Purposes, 18(3), 213–41.
TESOL Workplace Language Training Task Force. (2000). Section VI: Conduct an Instructional Needs Assessment, Best Practice
4. TESOL Best
Practices in Workplace Language Training, unpublished document. TESOL
International Association: Alexandria, Virginia.
van Naerssen, M. & Brennan, M. (1993). Language
socialization in professional cultures: Language for Specific Purposes. Cahiers de APLIUT. Paris:
Association des professeurs de langues des instituts universitaires de
Margaret van Naerssen (PhD, applied linguistics)
is an EFL/ESL teacher trainer and a linguistics consultant in forensic
cases involving nonnative English speakers. She was the third ESPIS
chair and the codirector of the English for Science & Technology
Center in Beijing (UCLA & Chinese Academy of Sciences). She
enjoys teamwork on needs analyses and materials development.