Jen Ouellette-Schramm and Jen Vanek
We are happy to bring you the 2019 Volume 35(2) Fall Issue of MinneTESOL Journal! This issue features 2018 Harold B. Allen Award winner Dr. Martha Bigelow’s invited piece on the importance of educators understanding pre-resettlement experiences of refugee students, an observation seen first-hand in her visit to Ali Addeh refugee camp in Djibouti. We are also pleased to share an invited piece by Dr. Ann Mabbott, who describes recent experiences bringing her ESL expertise abroad to help experienced ESL educators consider opportunities to apply professional experiences internationally. The remainder of this issue is focused on a critical yet arguably underrepresented area of ESL in adult education, with authors writing about their work spanning literacy to college classrooms.
In her article, “‘It’s not something you can imagine—it’s something you can watch’: Elders’ indigenous knowledge in beginning language and literacy classrooms,” Lisa Vogl shares an important yet rare perspective on how indigenous learning experiences shape literacy experiences among Somali elders—that of these elders themselves. Based on findings from her qualitative research, she recommends strategies for building on prior indigenous learning experiences.
In their viewpoint piece, “College Readiness Academy: Increasing equity by removing barriers to higher education,” Madie Spartz, Norzin Waleag, Elizabeth Fontaine and Merle Greene make a case for more collaboration between higher and adult education in Minnesota. They also describe how College Readiness Academy supports adult ESL learners in accessing college.
Finally, adult ESL teacher educator Suzanne Gilchrist McCurdy reviews the second edition of the book, Teaching adult English language learners: A practical introduction, by Betsy Parrish of Hamline University.
Lastly, as the Journal continues to grow and prepares for a new phase of leadership, we offer the article, How to Get Involved in MinneTESOL Journal. This complements our session of the same title at the 2019 MELEd Conference.
Please read, enjoy, and share the research and practice ideas in this issue of MinneTESOL Journal!
Jen Ouellette-Schramm and Jen Vanek
You know that MinneTESOL Journal is a resource for academically rigorous, yet practitioner-friendly and relevant articles related to English language education in Minnesota and surrounding states. You know that you can access and share articles through the Journal’s open-access, fully online and free platform. But did you know that whether you are a new or seasoned teacher, teacher educator, or emerging expert in the field, you can partner with MinneTESOL Journal to develop and share your expertise?
As a practitioner, you can work with the editors to craft what is working well in your classroom into an evidence-based practice article. As a teacher educator, you can leverage MinneTESOL Journal articles as thought-provoking springboards for discussion or models for linking theory to practice. Furthermore, MinneTESOL Journal now offers the options of blind peer review or editorial board review for both our Spring and Fall issues. Increased opportunities for blind peer review provide an excellent opportunity for graduate students to take their scholarship to the next level. MinneTESOL Journal’s peer review process is rigorous yet supportive of new authors. If a submission shows potential for publication, our senior editors work closely with authors, mentoring them through revisions and toward publication.
MinneTESOL Journal currently has opportunities for ambassadors to share the Journal’s work, and reviewers to provide rich guiding feedback on manuscript submissions. The Journal will also be moving into a new phase of leadership, with opportunities to join the editorial team. Please contact senior editors Jen Vanek or Jen Ouellette-Schramm regarding how you would like to get involved!
In this article, Dr. Bigelow describes her first-hand experience learning about multilingual resources and educational practices found in a refugee camp in Djibouti. Dr. Bigelow reflects on the importance of educators truly understanding the pre-resettlement experience of their learners and marvels that linguistically and culturally relevant multilingual resources not often found in schools in the U.S. are in use in refugee camps in the Horn of Africa.
Key words: refugee, East Africa, SLIFE, Djibouti
The proverb in my title is a carpe diem sort of Somali proverb that I learned in Djibouti, written here in Somali and in Arabic on a map in my office in Peik Hall at the University of Minnesota. It captures the urgency I feel when I think about education for refugee-background youth all over the world, but especially those here in Minnesota. It inspires me.
In Minnesota, we have an enormous amount of experience with refugee-background students, starting in the 1970s when so many Hmong families were given their well-earned asylum here. In fact, we are looked to across the nation for our expertise with the widely diverse needs of this slice of our English learner population. Note, for instance, the consistently strong showing at our annual and always homespun MinneSLIFE conference (http://minneslife.org/), that is now drawing participants and presenters from around the world. We can see Minnesota leadership on teaching adults who are new to print on the pages of the Literacy Education and Second Language Learning for Adults (LESLLA) symposia proceedings, starting back in 2005 (https://www.leslla.org/). We seem to be continually learning, re-learning, and un-learning about culturally responsive pedagogies that help refugee-background English learners develop print literacy and the myriad other skills they seek in order to reach their goals as they make new lives in the diaspora. There is no end to what educators need to know about teaching their refugee-background students, but it seems that one of our most significant gaps is what we know about our students’ pre- and during-migration experiences. We need to nuance, trouble, de-essentialize, and complicate our notions of our students’ pre-resettlement experiences in order to discover, amplify, and leverage their many gifts and assets for joyful, productive classroom learning experiences.
At the same time, it’s impossible for most of us to comprehend the pre-resettlement experiences of the refugee-background students in our classrooms. We listen to and read personal accounts of the experiences of displacement—so varied, but often terrible and so protracted.1 Narratives often repeat—girls with less access to education, child soldiers, lack of educational materials, enormous class sizes, injuries, violence, and illnesses suffered from displacement and en route to a refugee camp and also after arrival to the camp. That students will have had inadequate, limited or interrupted formal schooling is an overarching assumption among us when we meet a refugee-background student from the Horn of Africa (the Horn), and particularly from Dadaab refugee camp. Camps in Djibouti are different, which I learned during a recent visit. These camps, and the Djiboutian multilingual educational system in general, gave me pause, making me imagine a more multilingual Minnesota and the possible need for different first educational encounters with newcomers from the Horn. Djibouti showed me that we all need to explore, in our own way, the pre-resettlement experiences of our refugee-background students, but travel to the Horn is transformative.
I have learned some important lessons about newcomers through the locally-produced, community-engaged, and highly collaborative research I’ve done with teens from the Horn, their teachers, and my co-authors. We have discovered, for example, that even though the newcomers have physically crossed (many) borders, they have still been shaped by their pre-resettlement experiences. These experiences matter as we seek to understand language learning in post-migration contexts. We have seen how boys often adjust to the ordinary routines of formal schooling in the U.S. fairly easily because of previous schooling and print literacy, even if interrupted, and their prior schooling and literacy often converts into status in classrooms (Bigelow & King, 2014). Reasons for a gender gap can easily be traced to reports such as those produced by the UNHCR, clearly documenting the need to prioritize refugee girls’ education in pre-resettlement contexts. When inequities occur pre- and during-migration, it is clear that these inequities will follow youth into their places of resettlement. The dynamics among refugee-background youth, in the powerful social context of a newcomer classroom, will shape opportunities to engage in learning. As newcomers are adjusting to the routines and new cultural and linguistic practices of their classrooms (King & Bigelow, in press), they are growing new identities as cultural beings, as capable problem-solvers, and as students through family and community relationships as well as at school (Bigelow, in press). The more we know about pre- and during-migration experiences, the more we can legitimize the knowledge and skills of our refugee-background youth to be supportive of these educational processes (Bigelow, 2010; Bigelow & King, 2016; Bigelow, Vanek, King, & Abdi, 2017; King & Bigelow, 2018; King, Bigelow, & Hirsi, 2017).
In March this year, I traveled to Djibouti with two remarkable and visionary educators—Jill Watson (St. Olaf College) and Ryan Krominga (Faribault Public Schools)—to visit schools and a refugee camp and learn about the Horn, a region that is so important to Minnesota. Jill and I had been dreaming about such a trip for at least ten years and the idea of partnering with a school district to create ways educators could learn about pre-resettlement contexts started to take shape. Jill started working on this project in 2015 and plans to take a small group of students and educators from Minnesota to Djibouti to learn about the Horn and pre-resettlement experiences, and to collaborate with educators there. But before that, she needed to go herself and invited me to join her. Ryan became an enthusiastic partner because of the many students Faribault Public Schools has from the Horn. We hope that more educators from Faribault Public Schools will go in the future.
Why Djibouti? We wanted to visit East Africa, particularly a Somali homeland, considered by many to be a pre-colonial Somali ethnic heritage area. We wanted to visit a safe, small place that might afford us opportunities to learn about the Horn and about refugee camps similar to those we had long heard about in Minnesota. It is logistically and culturally easy for foreigners to travel to Djibouti and the scale and pace of the capital city is remarkably manageable. As safety in the region increases, we hope to broaden our collaborations; but in the interim, we envisioned Djibouti as a place for us and other educators to begin.
As we learned about Djibouti, Jill and I were stunned that we did not recall ever hearing about or meeting anyone from Ali Addeh refugee camp. We have Djiboutians in Minnesota, but they are few and likely lumped together with people from Kenya, Ethiopia, or Somalia. In other words, people who identify as ethnically Somali are considered Somali, even if they are from Djibouti or elsewhere. Data from the Census Bureau’s Population Estimates Program calculates that in 2017 there were no Djiboutians or Eritreans in Minnesota, a fact easy to refute given our acquaintances from these countries. Estimates for people with Somali (52,333), Ethiopian (21,996), and Kenyan (4,277) ancestry are also likely low, but do reflect our sense of the proportions of people claiming these nationalities, which often intersect more with ethnicity in the case of people who are have a Somali identity. This is because people of Somali ethnicity come from all of the countries of the Horn and now from many large Diaspora communities worldwide (e.g., UK, Netherlands, Norway, Sweden, Denmark).
Djibouti is a small country with a population of only about one million people and a total area of approximately 9,000 square miles. It is the only French speaking country in the region and is ethnically Somali (60%) and Afar (35%) with most people speaking one or both of those two languages in addition to French and Arabic. The language of instruction in Djibouti in French and other languages such as English and Arabic are taught as foreign languages. The Ministère de l’Éducation enthusiastically produces multilingual materials for K-12 Djiboutian students. This is a photo of third grade textbook in Somali and Afar designed specifically for Djiboutian students who are in French medium schools. With materials such as this, elementary age students’ cultural and linguistic identities are affirmed at school, as they learn through French.
We also saw materials designed to make reading enjoyable in the mother tongue. This is an example of a book of Somali folktales. The pictures on the cover of the book make it easy for children to link to stories told in their homes. These examples, of which there are many more, beg the question: How is it possible for a small country such as Djibouti to produce so many multilingual materials, when our wealthy country and state cannot?
Education in Djibouti also includes a well-developed and similarly controlled education system at the refugee camps within its borders. The Ministère de l’Éducation produces materials for the schools in the camps and offers administrative oversight as well as support.
To give readers some background about the camps, according to the UNHCR, there are approximately 27,601 refugees in Djibouti and the Ali Addeh refugee camp has approximately 15,676 refugees mainly from Somalia, Ethiopia, Yemen, and Eritrea. (This is small compared to, for example, the 212,936 registered refugees and asylum seekers in Dadaab Refugee Complex in 2019.) There are two other smaller camps in Djibouti—Holl Holl and Obock—and many refugees live outside the camps, including in Djibouti City. As a point of comparison, Minneapolis Public Schools has approximately 36,000 students and Saint Paul Public Schools has approximately 38,000 students.
When we went to Ali Addeh, we had the opportunity to visit various K-12 schools that most, if not all, children in the camp attend.
The education practices in the Djiboutian refugee camps had many admirable and progressive policies, for example:
While French is used for instruction in the rest of the country, the director of Ali Addeh camp explained that English was selected as the language of instruction in all subject areas, to better prepare students for life after resettlement. The English materials specifically designed for the children in the Ali Addeh camp were of the same quality as those produced for the Djiboutian children. This is an example of a page from a civics book for elementary age children:
The materials consider the realities of life in the refugee camp in ways that seem, from the perspective of outsiders, culturally appropriate. Children graduate from the schools in the camp as they await resettlement. We heard from educators that the majority are not resettled and even the best students are not able to find employment. This desperation is part of a system far beyond the control of teachers, the camp administration, the Djiboutian government—it’s a global problem for which we’re all responsible.
We can learn from Djibouti. There are clear lessons learned for our context, even without more and much needed further study:
One of the most important questions I was left with is why don’t we offer multilingual materials for all of our small populations? We have legislation that supports multilingualism broadly in Minnesota and immersion programs are able to gather and write curricula across all grades and subjects. But materials in the languages of our immigrant groups are not available; even our most commonly represented linguistic minority groups do not benefit from any coordinated, systematic, or professional effort to provide them with materials in their languages and that reflect their lives. It must be that Djibouti, despite their small size and limited resources, believes in the importance of students’ languages and cultures being included in their schooling experiences. In other words, it seems that the barrier in the U.S. is largely ideological, not financial. In so many multilingual contexts, ideologies such as the following still prevail:
The children in the refugee camps in Djibouti are studying hard, and learning English to a very high level of proficiency. There is hope that English is a pathway to a better future for them. We don’t know if they will ever be resettled, or that they’ll be resettled to a place where English will help them make healthy, successful, and happy lives. Optimistically and responsibly, the Djiboutian Ministère de l’Éducation continues to educate the refugee children in ways that reflect how they would educate their own children, in ways that they hope will resonate with the children’s lives, and that will serve them in their very uncertain futures. What if we all took up this very serious goal of teaching multilingual children multilingually?
The U.S. is on a path to no longer being a receiving country for refugees. The number of refugees resettled in the U.S. has been very low, particularly given our resources and size. Under the current federal administration, the U.S. plans to admit a maximum of 18,000 refugees in 2020, down from 30,000 in 2019, the lowest number since 1980 when Congress created the U.S. refugee resettlement program. Compare this number to 207,116 in 1980 or 60,191 in 2008, nationwide. Data in the following table from the U.S. Department of State on the top 10 languages spoken by refugees in the U.S. reflect the loss of cultural and linguistic diversity in the U.S.
The following list of top 10 languages spoken by K-12 students in Minnesota (besides English) in the 2018-2019 academic year reflects our proud and strong immigrant and refugee communities:
Between October 2018 and September 2019, Minnesota welcomed only 848 refugees according to the Department of State Refugee Processing Center.
According to the UNHCR, there are now 70.8 million forcibly displaced people worldwide. There is more need than ever to receive refugees. But as the number of refugees resettled in the U.S. decreases, so do services for refugee-background youth and families. It will take many years to re-establish the newcomer programs and services that are disappearing, thus making it extremely challenging to become a place once again that has the capacity to resettle refugees as well as we have, for so many decades.
I believe that our current federal refugee resettlement policies are taking us in the wrong direction and will have a profound and negative impact on our state—educationally, culturally, and linguistically. Minnesota has the tradition and the capacity to help address the worldwide refugee crisis. We must resist compassion-less federal policies and isolationist ideologies that ignore the level of displacement that is occurring in so many places in the world and stand in solidarity with our refugee-background communities.
Bigelow, M. (2010). Mogadishu on the Mississippi: Language, racialized identity, and education in a new land. New York: Wiley-Blackwell.
Bigelow, M. (in press). The case of a Somali teenage girl with limited formal schooling: Seeing assets and poking holes in deficit discourse. In A. Cooper, & A. Ibrahim (Eds.), Black voices matter: Black immigrants in the United States and the politics of race, language, and multiculturalism. New York: Peter Lang, International Academic Publishers.
Bigelow, M., & King, K. (2014). Somali immigrant youths and the power of print literacy. Writing Systems Research, 6(2), 1-16.
Bigelow, M., & King, K. (2016). Peer interaction while learning to read in a new language. In M. Sato, & S. G. Ballinger (Eds.), Peer interaction and second language learning (pp. 349-375). Philadelphia: John Benjamins.
Bigelow, M., Vanek, J., King, K., & Abdi, N. (2017). Literacy as social (media) practice: Refugee youth and home language literacy at school. International Journal of Intercultural Relations, 60, 183-197.
King, K., & Bigelow, M. (2018). The language policy of placement tests for newcomer English learners. Educational Policy, 32(7), 936-968.
King, K., Bigelow, M., & Hirsi, A. (2017). New to school and new to print: Everyday peer interaction among adolescent high school newcomers. International Multilingual Research Journal, 11(3), 137-151.
King, K., & Bigelow, M. (in press). The hyper-local development of translanguaging pedagogies. In E. Moore, J. Bradley, & J. Simspon (Eds.), Translanguaging as transformation: The collaborative construction of new linguistic realities. Tonawanda, NY: Multilingual Matters.
Hamline Professor Emeritus Ann Mabbott shares how experienced ESL teachers can take the skill set they have developed here to teach abroad.
Key words: English Language Specialist, teach abroad, academic language, content-based instruction
Would you like to teach overseas? For the first time or once again? ESL professionals are all enthusiastic about languages and cultures, and in helping language learners to acquire English. In addition, most of us have an interest in traveling internationally, and our experience teaching English to immigrants and refugees in the United States can be foundational for international work. In this article, I will discuss employment opportunities, ones that are relatively well paid, with the U.S. State Department for experienced ESL teachers. I will describe how I used the work I have been doing in the United States with teaching academic language to support content instruction in an international context, with bilingual teachers in Spain. Finally, I will provide some links and resources to additional overseas English teaching positions for teachers of varying levels of experience.
The English Language Program Office in the U.S. State Department offers two types of international teaching opportunities for ESL professionals. The first, the English Language Fellows program, is for experienced English teachers with an M.A. or equivalent experience who are able to teach abroad at universities and other academic institutions for 10 months. It allows you to take dependents with you, and provides a stipend for dependents. The prospect of teaching in a country such as Nepal or Malaysia for 10 months has always appealed to me, and if it does to you, look into this opportunity.
The second program is called English Language Specialists. The Specialist program sends English teaching experts to countries who request them to provide professional development lasting from two weeks to three months. Candidates apply to be an English Language Specialist, and then are accepted into a pool of “experts.” Once a country makes a request for a particular type of project, representatives of the country are sent several resumes to consider. The host country chooses individuals to interview over Skype, and then invites their top candidate to come. The types of projects that Specialists deliver vary widely, from using drama in English instruction to proving workshops on communication skills for UN Peace Keeping Troops.
In my case, I finally had time to pursue my desire to do more teaching overseas when I retired from my faculty position at Hamline University. I presented the English Language Specialists program a resume which included my long-time work teaching content teachers (math, science, social studies) how to teach content and the supporting academic language needed at the same time, along with credentials in language proficiency assessment and online course development. So far, I have gotten invitations to teach in Spain and Russia.
Spain, the country I worked in first, had been starting many bilingual immersion programs as a way to provide young people more international opportunities and to support the internationalization of its economy. Together with the cultural attaché office in the U.S. embassy in Madrid, representatives of the bilingual programs in Spain put together a program of conference presentations and teaching at four different universities in Spain, which they wanted the Specialist to provide.
One of the things that I had to convince myself of and state publicly at the beginning of every talk I gave in Spain is why what I could bring from my Minnesota immigrant and refugee education perspective was equally relevant in bilingual immersion schools in Spain. Most of the k-12 students with whom the Spanish teachers work are not immigrants or refugees, but what we have in common is the need to teach content in a language in which the students are not yet fluent. We arrived at two goals:
Fortunately, I was able to draw on the work of the Hamline University’s English Learner in the Mainstream (ELM) project, which was funded by the U.S. Department of Education in 2016. In both Spain and in Minnesota, we are changing the roles of both teacher educators (putting a new emphasis on complementary language and content instruction) and ESL teachers (insisting that one of their roles be coaching their mainstream colleagues).
The theoretical foundations of the ELM work that I drew on for my work in Spain are:
Instructional coaching, although it is listed last among the foundations, is crucial, and even newer to teachers in Spain than the U.S. We all tend to come from a background of independent teachers who reign in their own classrooms, and so teachers have to learn to trust each other as supportive professionals working together for the benefit of students.
Accessible instruction includes multi-modal teaching practices that make it possible to understand content even when one’s language proficiency is limited. Systemic functional linguistics urges teachers to think about the function of the academic texts they are using, and to teach at the:
Most content teachers, since they are not trained to be language experts, tend to think of language needs in terms of new vocabulary, or words that are bolded in textbooks, and need to be encouraged to expand their teaching beyond semantics. They also need to learn about syntax and discourse patterns for their disciplines.
Fortunately, ELM provides wonderful resources for teaching about academic language and how to write language objectives that support content instruction that everyone can access. One approach to writing them are the following steps:
Both content teachers in Spain and content teachers here do not have pedagogical linguistics training, and it is up to us as ESL professionals to support them. ELM also provides a worksheet with more support for writing academic language objectives.
One example I like to use when helping teachers learn how to write language objectives comes from a ninth-grade social studies class I observed in Minneapolis. The steps above are followed with the social studies example below in Figure 1.
When working in Spain, I changed the context from the U.S. to a comparison between per capita consumption in Spain compared to that of India, making the instruction culturally relevant for the context in which I was teaching. I also used bull fighting, a Spanish cultural tradition which is very controversial in Spain right now, as the setting for a discussion about the different types of academic or professional language. After showing a picture of a bullfighter, I had teachers write about the picture from the perspectives of the bullfighter, the bull breeder, a veterinarian, an animal rights activist, a journalist, and the tourism minister. We analyzed the language that each different professional might use, and determined what function each was fulfilling, as well as what kinds of language would need to be taught to fulfill those various functions.
Interaction is the last foundation that my work here and in Spain both addressed. As ESL professionals, we all know that academic language is not acquired efficiently when students do not get a chance to use it in writing and speaking (Swain, 1995; Zwiers & Crawford, 2011). An outstanding example of interaction in a content class on environmental issues that includes a large number of English learners can be found in Sarah Horowitz’s Teaching Channel video, Teaching Evidence-Based Academic Discussions. I ask teachers to watch the video and then identify:
When students have a lower level of proficiency, Watson’s (2015) RISA oral interaction approach can be quite effective. RISA stands for routine, integrated, structured, and academic. They are similar to the dialogues we had to memorize in world language classes, but the topics of conversation are academic, rather than social. They are done routinely, integrated into content objectives, with the structure provided, and academic in nature. An example that Watson provides is in Figure 2.
The dialogue can be about any topic in the curriculum, and by reciting it, and then reversing roles, students can learn the necessary academic vocabulary and language structures, as well as the pragmatics involved in oral interaction. As students become more proficient, the support structure provided can gradually diminish. Since many of the students in Spanish classrooms have lower levels of English proficiency, Spanish teachers loved the RISA dialogues.
The last thing I shared with the teachers in Spain is the ELM classroom observation form. As in the United States, classroom observations in Spain tend to not address some of the crucial issues for second language classrooms. The ELM observation form has sections on language proficiency of the English learners, access to instruction, language objectives that support the content, and interaction. It can also be revised to meet needs particular to a school or classroom. I encourage anyone who has an opportunity to work with content teachers to use the form for recording classroom observations and discussing how instruction might be modified to meet the needs of English learners.
What I have learned by teaching abroad is that the work we are doing here with immigrants and refugees is very transferable to instruction in a second language in other countries. In fact, I just got back from my second experience as an English Language Specialist. This time the trip was to Russia, where the context was largely for university-level classes that are taught in English. My major assignment was teaching academic language and content at the same time, but several additional requests came up. Some teachers wanted to know how to differentiate instruction in a content class for students with different levels of English proficiency. Differentiation is another area with which American ESL teachers have a lot of experience, as they are often teaching classes with varied proficiency levels. Russian teachers were also interested in understanding higher-order thinking skills and applying them to critical reading, writing, and speaking. Finally, if you have written an M.A. or doctoral thesis, those skills are very much in demand as well. Many university-level Russian teachers are being asked to publish for the first time, and are not familiar with reading or writing either pedagogical or research papers. If you are interested in sharing your skill set, consider applying to be an English Language Fellow or Specialist with the U.S. State Department. My assignments were in Spain and Russia, but there are similar requests from all over the world. Below I also list other international opportunities for experienced ESL professionals to consider.
Aguilar, E. (2013). The art of coaching. San Francisco: Jossey-Bass.
Brisk, M. E. (2015). Engaging students in academic literacies: Genre-based pedagogy for k-5 classrooms. New York: Routledge.
Cummins, J. (2008). BICS and CALP: Empirical and theoretical status of the distinction. In B. Street, & N. H. Hornberger (Eds.), Encyclopedia of Language and Education (2nd ed.) (pp. 71-83). New York, NY: Springer Science + Business Media LLC.
Delpit, L. (1988). The silenced dialogue: Power and pedagogy in educating other people’s children. Harvard Educational Review, 58(3), 280-299
Dutro, S., & Moran, C. (2003). Rethinking English language instruction: An architectural approach. In G. G. Garcia (Ed.), English learners: Reaching the highest level of English literacy (pp. 227-258). Newark, DE: International Reading Association.
Gay, G. (2000). Culturally responsive teaching: Theory, research, and practice. New York, NY: Teachers College Press.
Gibbons, P. (2015). Scaffolding language, scaffolding learning: teaching English language learners in the mainstream classroom. Portsmouth, N.H: Heinemann.
Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language and meaning. London: Edward Arnold.
Heath, S. B. (1983). Ways with words: Language, life, and work in communities and classrooms. New York, NY: Cambridge University Press.
Knight, J. (2007). Instructional coaching: A partnership to improving instruction. Thousand Oaks, CA: Corwin.
Ladson-Billings, G. J. (1995). Toward a theory of culturally relevant pedagogy. American Education Research Journal, 35, 465-491.
Paris, D. (2012). Culturally Sustaining Pedagogy: A Needed Change in Stance, Terminology, and Practice. Educational Researcher, 41(3), 93-97.
Schleppegrell, M. J. (2004) The language of schooling: A functional linguistics perspective. Mahwah, NJ: Lawrence Erlbaum Associates.
Swain, M. (1995). Three functions of output in second language learning. In G. Cook, & B. Seidlhofer (Eds.), Principle and practice in applied linguistics: Studies in honour of H. G. Widdowson (pp. 125-144). Oxford: Oxford University Press
Watson, J. (2015) Watson Educational Consulting: Effective instruction and assessment for diverse learners. Accessed 11 November, 2019 at https://www.watsoneducationalconsulting.com
Wong-Fillmore, L., & Snow, C. E. (2000). What teachers need to know about language. Washington, DC: Center for Applied Linguistics. Retrieved from http://www.elachieve.org/images/stories/eladocs/articles/Wong_Fillmore.pdf.
Zwiers, J. (2008). Building academic language: Essential practices for content classrooms. San Francisco, CA: Jossey-Bass.
Zwiers, J., & Crawford, M. (2011). Academic conversations: Classroom talk that fosters critical thinking and content understandings. Portland, ME: Stenhouse Publishers.
Suzanne Gilchrist McCurdy
“Teaching Adult English Language Learners: A Practical Introduction,” Betsy Parrish’s long-awaited 2nd edition to her 2004 textbook. 388 pages, paperback, $39.00.
Key words: English as a second language, adult education, teacher preparation, textbooks
Many adult ESL practitioners have a copy of Betsy Parrish’s first book, Teaching Adult ESL (2004). It’s probably well-used. Mine is. It accompanied me through my time as a graduate student, then as a teacher, and now as a teacher educator, always giving me practical, concrete advice for the classroom. In many ways, however, the field of adult ESL is unrecognizable from what it was 15 years ago. We have advanced at a rapid pace to a more professionalized, standards-based field. We have widened our curricular focus significantly from one of local survival skills to a much broader frame, including career and college readiness skills, the navigation of digital spaces, and meaningful community involvement. Adult ESL providers have expanded options for delivering instruction, including distance education and blended learning. We have much more access to resources and to our global colleagues than we ever have, further enriching and broadening our understanding of adult ESL. With this forward movement we need a new book, and we have it in the much anticipated 2nd edition of Parrish’s book: Teaching Adult English Language Learners: A Practical Introduction (2019).
The book is divided into an introduction and ten chapters. Chapter 1 introduces the reader to the multifaceted field of adult ESL, including the diversity in the learner population with regard to background, level, strengths and needs, as well as the variety of organizational contexts in which an adult ESL practitioner may find themselves. Chapter 2 offers a solid and accessible introduction to the history of second language acquisition methodologies, key theorists, and their influence on current research-based teaching principles. Chapter 3 explores how to combine rigorous, integrated-skill instruction with meaningful and student-centered teaching principles. Theory is made practical in Chapters 4-5 as Parrish guides the reader through the development and classroom application of contextualized practice activities using examples from real classrooms representing levels across the spectrum. Chapters 6-7 lead the reader through lesson planning, creating principled and inclusive learning environments, and managing the complexity of multilevel groups. Parrish fully updated Chapter 8, which prepares readers to select, adapt, and supplement materials and resources for a variety of contexts, including digital tools. The wide variety of assessment types and their purposes are addressed in Chapter 9, from classroom-based and authentic assessment to standardized tests, as well as teacher self-assessment and opportunities for professional growth. Chapter 10 on standards and accountability is completely revised and includes current frameworks from different parts of the globe.
Much of what made the previous edition such an accessible resource for new teachers and a mainstay in teacher education classrooms remains in this volume. Parrish grounds each section with a pre-reading task, inviting the reader to reflect on their own experiences and knowledge before diving into the content. Theoretical content and teaching principles are made accessible through vignettes, testimonials, and activity ideas that Parrish draws from her own experience and from practitioners around the world. Each chapter ends with thoughtful activities which allow the practitioner to apply their learning to their teaching context.
Although the second edition retains the same structure and voice of the earlier volume, much of the content has been adjusted or completely revised in response to shifts in adult education requirements and wider societal changes of the past 15 years. One such change woven through each chapter is Parrish’s ongoing argument for increased rigor and the importance of exercising learners’ critical thinking skills at all levels of instruction. Especially helpful in this regard are the classroom activity examples throughout the book in which Parrish explicitly identifies how to integrate these components into new or already existing materials.
Parrish’s focus on rigor and critical thinking goes hand-in-hand with the adult ESL field’s shift to standards-based curricula. National-level standards are a reality in many adult ESL settings across the world. New and experienced practitioners will find reassurance in Chapter 10 that these standards need not be at odds with a learner-centered approach. To this end, Parrish walks the reader through a series of tasks, using samples from the content standards frameworks of the U.S., Canada, Australia, and the U.K. These tasks systematically ask the reader to unpack not only how standards address college and career readiness, but also how they overlap with life-skills and unique learner needs, illustrating that even learner-generated materials can be used to target the standards. At first blush, standards are daunting. Parrish’s approach gives practitioners across the globe the tools they need to feel confident navigating a standards framework while addressing learner needs.
The ability to navigate digital spaces has become as much of a necessity for learner success as print literacy, expanding and reshaping what it means to integrate technology into curricula. Parrish acknowledges this shift in a fully updated and expanded section dedicated to digital learning in Chapter 8. Drawing from the expertise of current teacher educators, practitioners, and researchers, Parrish encourages teachers to help learners build toward digital competence, teaching skills with transferability from one digital space to another. Parrish presents several examples of integrating digital literacy into language learning at all levels, in and outside of class.
This book will appeal to multiple audiences. Adult ESL teacher educators and their students will find it a valuable core text for ESL methods, TEFL/TESL certificate, or ESL continuing education courses. Adult education professional development programs can guide professional learning and discussions using tasks and reflections found in the text. It would also be valuable to a new or practicing teacher as independent study—the tasks and reflections are designed to be used by readers in vastly different contexts. Readers will appreciate the end of chapter recommended resources for more in-depth reading. Parrish’s engagement with and in support of teachers does not end with the book. With a nod to the power of the internet for disseminating new information, and cognizant of how rapidly web resources change, Cambridge.org makes available lists of free online resources for each chapter, which are updated regularly by Parrish herself. You need to create a Cambridge account, but it is free with the choice to opt out of email notifications from the publisher.
As a teacher educator, Teaching Adult English Language Learners: A Practical Introduction is akin to having Parrish as a co-teacher in the classroom. For my current students, teachers in training, the book serves as a mentor. For those already in the ESL classroom, it will be a trusted colleague. My personal copy of the second edition has already become a dog-eared book; erupting with sticky-notes and sporting pages bright with the neon color of a highlighter. True signs of a book that gives practical, timely advice every time you need it.
Parrish, B. (2004). Teaching adult ESL: A practical introduction. New York, NY: McGraw-Hill.
Parrish, B. (2019). Teaching adult English language learners: A practical introduction (2nd ed.). Cambridge, U.K.: Cambridge University Press.
Prior knowledge is central to memory and learning, but many older refugees come to the classroom with ways of knowing and experiences much different from what they find in Western teaching contexts. Somali elders offer insight.
Key words: indigenous knowledge, Somali, elders, beginning language and literacy, LESLLA
Almost six years ago, I was working in an affordable housing nonprofit as a coordinator of ESL programs. I was struck by the number of elder learners in the program—their regular attendance in a beginning literacy class impressed upon me a vibrant example of who adult learners are. They often said, “Teacher! Students—no good!,” but this ran counter to what I knew must be vast reserves of prior learning experiences which, because I did not speak Somali, I knew nothing about. I wanted to honor their vitality of mind and spirit in learning through a qualitative study on their prior learning experiences and indigenous knowledge (Vogl, 2019). The stories they shared with me described important lessons that were altogether invisible to me, and which I would like to share in this piece: elder learners’ prior learning experiences, the interplay of indigenous knowledge and cultural dissonance in beginning language and literacy classrooms, and the profound impact of isolation and trauma among refugee learners.
Becoming familiar with the knowledge and experiences our students bring to the classroom is one of the central tenets of adult education theory (Tarone, Bigelow, & Hansen, 2009; TESOL International Association, 2018). This can be most challenging in beginning language and literacy classrooms, where the teacher may not always share a language or cultural background similar to their students’. In this paper, I share narratives from elder learners in order to highlight their knowledge and prior learning experiences. Their knowledge is indigenous knowledge, which is passed down orally through generations, and is specific to the place where they lived.
I focus on one particular subset of learners—Somali elders aged 65 and older participating in a beginning language and literacy classroom in Minnesota. Seven participants had gone to school in the U.S. for more than two years, and of those, six were currently attending school at the time of the study. Of the remaining participants, five had attended school in the U.S. fewer than six months and were not currently attending school, while two had never attended school in the U.S. (Vogl, 2019). In order to understand what happens to the place-based knowledge that these learners possess when they must leave their country, I highlight important findings from a qualitative study of interviews and observations with an instructor and elders (Vogl, 2019), which explored their prior learning experiences in order to illuminate what indigenous knowledge is and how culture dissonance, trauma, and isolation can impact elder learners, in elders’ own words. Finally, while the main findings of this study explored indigenous knowledge and cultural dissonance, I will briefly discuss some approaches that may be useful for teachers to apply with learners of any age.
Language learning can be challenging at any age, and the apocryphal belief is that it may become impossible the older you get. But language learning does not stop after childhood (de Bot & Makoni, 2005), and many who register for adult ESL classes are elders. Further compounding our understanding of elders’ needs is the fact that the majority of research on the experiences of refugees in education is focused primarily on younger refugees (Haines, 2010). The research which does focus on elder learners is largely focused on high-income white populations, while minority populations are underrepresented (von Doetinchem de Rande, 2012).
The elders interviewed in this study had not attended school in their home country and were not literate in their first language. In fact, it was not until 1972, when Somali elders of the present day had already reached adulthood, that Somali became the national language and language of instruction. Formal education was limited and intended to further colonization of the country by Britain and Italy (Abdi, 1998). The elders who participated in this study had not gone to school, but instead grew up learning from their elders, in different environments and lifestyles (Vogl, 2019).
Students’ relation to indigenous knowledge is dynamic and subject to shifts in lifestyle and occupation. This is true even for elder learners who, in the case of this group of participants, reached adulthood before being subjected to the upheaval and trauma following the civil war and refugee resettlement process. With this background information, we are able to set the context for indigenous knowledge and how it shapes elder learners’ perception of learning experiences.
Many people know things which have aspects reminiscent of indigenous knowledge. For example, perhaps your grandmother showed you how to preserve fruits you grow in the summer and save them for the winter; for others, indigenous knowledge may be the stories and lessons your extended family taught you about the land they grew up on or the neighborhood where they lived. But for most of us who grew up in the U.S., the knowledge we have comes from books, like the canon of American literature or legends of our founding fathers. This knowledge was taught to us in formal settings, and, thanks to standardized education, much of what we know is the same no matter where we live (Semali & Kincheloe, 1999). For elder refugees from Somalia, their knowledge is not found in textbooks—it was passed down orally through generations, and it is specific to the place where they lived (Ahmed, 2014).
As I was having conversations with elder participants, I was most interested in how I, as a teacher, could understand more about their indigenous knowledge as the prior learning experiences that are so valuable to adult learners in any language class. But, indigenous knowledge as an epistemological concept is difficult for outsiders, as I was in this situation, to nail down and to understand. Often, I felt I was at an impasse in the conversations with elders and didn’t know how to ask more questions to find out more. Luckily, one elder explained the differences between text-based learning and indigenous knowledge systems with textbook-like precision:
“You can practice with your parents. You look at your brothers and sisters and mothers and fathers, and your neighbors. How to butcher, how to milk, how to build and move. It’s not something you imagine—it’s something you can watch.”
This quote shows us that, rather than formal classroom learning experiences, elder learners’ indigenous knowledge was based in experiential learning through apprenticeship, and was grounded in the places and lifestyles of elders before coming to the U.S. As educators, we want to know more about our students’ past learning experiences when teaching new skills. Whether our students grew up in the city or in the country, learning through formal or informal education, we must become more familiar with what they know. For many elder refugees, indigenous knowledge is that wealth of information educators need to draw on, and elder learners attest that it is of equal value to text-based learning:
“Everything about the country, you can write. If you don’t know anything, you are a dark person.”
[Do you think country people are dark people?]
“Country is country, city is city, the education is a different way. Everybody knows what they know. No, everything is good.”
[In reference to images of nomadic lifestyles:]
“This is education. This is a building that they build. This is their house. Sometimes they move. Sometimes they build. This is learning. See? It’s learning.”
Most elders interviewed in this study grew up in rural communities rather than cities, and so I explored how learning was different there and how elder learners compared it to formal schooling. The quotes above demonstrate that elders recognize the value in indigenous knowledge and its relevance to formal learning experiences (“Everything about the country, you can write.”). Further, in indigenous societies, elders are living repositories of indigenous knowledge, and they often become teachers themselves (Dubus, 2010). Elders as teachers offer a deep, reflective sensibility and insight into the precariousness and continual conditionality of life (Deloria, 2006; Gadow, 1983). It’s clear that elders have so much to contribute, not only as learners, but as teachers themselves. Thus, student-centered teaching strategies are essential to engaging elders in a way that enables them to share their prior learning experiences.
As a primary tenet of adult learning theory, students’ prior success in learning is paramount to success in the acquisition of new skills (Vinogradov, 2013). However, when students perceive a difference between their prior learning experiences and what they experience in the classroom, this can present cultural dissonance. Cultural dissonance, as was identified by the elder learners in this study, concerns perceived differences in priorities between the students’ ways of knowing and learning and the Western-style, formal educational framework used in the U.S., which can cause a general sense of discord and conflict when confronted with a new cultural model for learning (Marshall & DeCapua, 2013).
Each elder began the interview with a story about their indigenous knowledge and expertise and success in learning prior to coming to the United States. But while each elder identified this knowledge as valid and their learning experiences as successful, many later conceded their belief when asked to talk about their experiences as students in the U.S. And for many, this meant that their prior knowledge could not be valuable to them in their current experience: “English and Somali are different. I didn’t even go to duqsi. All the things I’ve learned wouldn’t help me in English class.”
The profound impact this has on elder learners is real. I was struck in particular by one elder’s account of why she quit going to school. As one elder described to me, “pen and pencil” learning is the only kind that counts as real education:
“He’s an educated person– he’s a teacher. Me, I’m not educated.
[You don’t see all the things you learned as being a resource in your life?]
When you ask me about learning, I always think pen and pencil. But me, I don’t think it was learning. If you want to learn something you have to learn from your brain. I think, “Why do I bother my teacher? I’m not going to bother my teacher.”
[Why do you feel like you’re wasting the teacher’s time?]
People keep going and I’m going behind. And I will stay home. That’s what I decided.
When elders see a difference between the expectations and practices of Western-style, formal education and everything they have learned in their own lives, this presents cultural dissonance. These perceived differences follow elders into the classroom, impacting their self-esteem and raising their affective filter, making it too stressful to learn and affecting their attendance.
“Nobody knocks on your door in America. I can’t tell my problems, whatever I have inside.”
In a culture which values individualism and independence like in the United States, elders’ vitality is often measured by how well elders are able to live independently and remain self-sufficient (Clark, 1991). On the other hand, Somali culture, as well as many indigenous cultures throughout the world, is collectivist. While the primary focus of this study was on indigenous knowledge and prior learning experiences, a full portrait of elders’ experiences revealed the profound impact of isolation and trauma on elder refugees.
One elder I interviewed was a favorite classmate and neighbor among everyone we knew. As a gregarious, caring person who kept up with all her neighbors, she taught me by example how living in a community is so important to people at any age, and something which is markedly absent in American society. She described the differences between individualist and collectivist societies in this profound indictment:
Why don’t you know your neighbors? Somali people and our religion say you have to know your neighbors. What did they eat last night? If they didn’t eat, you have to feed them. You have to eat together—that’s what being neighbors means. In America, the only way that they don’t know each other is because there is no trust. When one of their American neighbors dies, how do they go?
Many of the elders in this study, like the speaker above, attested to the profound importance of community. Through his student-centered approach, the teacher in the building where these elder learners lived demonstrated how English class can be a part of the fabric of the community, where neighbors participate in learning together. A student-centered approach is essential to becoming a part of the community. The elder learners in this study were celebrated and welcomed as major participants in class, taking turns at the whiteboard and leading activities.
In addition to isolation, past trauma has a profound impact on elders’ sense of self and current experiences as learners:
[So you’re good at math…]
If you ask me now I forget. They used to tell me, “Come on, come on—you are the expert at adding together.” But now I don’t know because of the civil war. Our life was so bad, that’s why we forget a lot.
Several elder participants reported that they were bothered by thinking about the past, or “thinking too much.” In a report by the Minnesota Department of Health (2014), “thinking too much” was identified as a possible indicator of trauma and PTSD affecting functionality, and this was corroborated by the testimony of several elders in this study. Recognition of trauma is a first step towards incorporating the systemic changes needed to adequately serve students who have been affected by it.
A student-centered centered approach values prior knowledge—and when it comes to indigenous knowledge, our students are the experts. Elders as Fonts of Knowledge (Watson, 2015) is one such approach that has shown success. Calling on elders’ expertise may also require educators to “flip the classroom,” and Marshall and DeCapua’s (2013) Mutually Adaptive Learning Paradigm provides excellent guideposts for validating students’ background knowledge when learning new skills.
As the interviews in this study revealed, elder refugees’ lived experiences have important implications for educators. We know that ignoring indigenous knowledge has profound consequences on students in general, and elders as underserved students in particular, and its solution lies in curriculum development and innovation. But, indigenous knowledge is not a “show-and-tell” project, nor is it available as ready-made school curricular materials. It will require ground-breaking research to reach all students.
Abdi, A. A. (1998). Education in Somalia: History, destruction, and calls for reconstruction. Comparative Education, 34(3), 327-40.
Ahmed, F. (2014). Exploring halaqah as research method: a tentative approach to developing Islamic research principles within a critical ‘indigenous’ framework. International Journal of Qualitative Studies in Education, 27(5), 561-583.
Clark, P. G. (1991). Ethical dimensions of quality of life in aging: Autonomy vs. collectivism in the United States and Canada. Gerontologist, 31(5), 631-639.
de Bot, K., Makoni, S. (2005). Language and aging in multilingual contexts. Clevedon: Multilingual Matters.
Deloria, V. (2006). The world we used to live in: Remembering the powers of the medicine men. Golden, CO: Fulcrum Pub.
Dubus, N. (2010). “I feel like her daughter not her mother”: Ethnographic trans-cultural perspective of the experiences of aging for a group of Southeast Asian refugees in the United States. Journal of Aging Studies, 24(3), 204-211.
Gadow, S. (1983). Frailty and strength: The dialectic in aging. The Gerontologist, 23(2), 144-147.
Haines, D. (2010). Safe haven?: A history of refugees in America. Sterling, VA: Kumarian Press.
Marshall, H. W., & DeCapua, A. (2013). Making the transition to classroom success: Culturally responsive teaching for struggling language learners. Ann Arbor: University of Michigan Press.
Minnesota Department of Health. (2014). Mental health screening recommendations for newly arrived refugees in Minnesota. St. Paul, MN: Minnesota Department of Health. Accessed 16 November, 2019 at https://www.health.state.mn.us/communities/rih/guide/mhguidelines14.pdf
Semali, L. M., & Kincheloe, J. L. (1999). What is indigenous knowledge?: Voices from the academy. Hoboken: Routledge.
Tarone, E., Bigelow, M., & Hansen, K. (2009). Literacy and second language oracy. Oxford: Oxford University Press.
TESOL International Association. (2018). The Six principles of exemplary teaching of English Learners. Accessed 1 November 2019 at http://www.tesol.org/the-6-principles
Vinogradov, P. (2013). Defining the LESLLA teacher knowledge base. Proceedings
from the 8th LESLLA Symposium (pp. 9-24). Jyvaskyla, Finland: Low Educated Second Language and Literacy Acquisition.
Vogl, L. (2019) Indigenous knowledge and literacy acquisition: A qualitative study of low-literate elder refugees’ educational backgrounds and cultural dissonance. Proceedings from the 13th LESLLA Symposium (pp. 160-174). Portland, Oregon: Low Educated Second Language and Literacy Acquisition.
von Doetinchem de Rande, S. (2012). Education for the elders in the U.S. Opportunities and challenges in the context of the demographic change. REPORT 4/2012 (35. Jg.), 35-54. Retrieved from https://www.die-bonn.de/doks/report/2012-aelterer-mensch-01.pdf
Watson, J. (2015) Instructional approaches that set SLIFE up to succeed (and are good for everybody else): Structured oral interaction and elders as fonts of knowledge. Paper presented at the 2015 Minnesota English Learner Education (MELEd) Conference, Bloomington, MN. Accessed 16 November, 2019 at https://www.slideshare.net/JillWatsonPhD/jill-watson-instructional-approaches-that-set-slife-up-to-succeed-and-are-good-for-everybody-else-structured-oral-interaction-and-elders-as-fonts-of-knowledge-meled-2015
Madie Spartz, Norzin Waleag, Elizabeth Fontaine, and Merle Greene
College Readiness Academy (CRA) is an innovative program model providing both academic instruction and wrap-around advising services to college-bound Adult Basic Education (ABE) students. This article explains CRA’s rationale and methods for implementation, including a sample lesson plan, outcomes, and potential for replication.
Key words: College readiness, developmental education, equity, remediation, navigation
College Readiness Academy (CRA), offered by the International Institute of Minnesota (the Institute), is an innovative model with proven success in preparing college-bound adults for college-level coursework. CRA helps bridge the achievement gap that many first-generation, college-bound new students experience in higher education. By fully integrating intensive academic skill-building with proactive navigation services, CRA helps students overcome academic and personal obstacles to college success. The overall goal of CRA is to provide a seamless transition to college by offering an alternative to the post-secondary, remedial (sometimes called developmental) education system, which can increase college retention and graduation rates, while also saving time and money. In this article, the authors—CRA instructors and navigators—highlight the challenges that led to the development of CRA, describe the classes and navigation system, and demonstrate how the CRA model might work for students
The CRA Partnership was launched in 2015 in response to the frustrations expressed to us faced by graduates of the Institute’s nursing assistant training program as they attempted to enter college nursing programs. The stories of struggle had a common theme: students had been assigned to mandatory developmental education (ESL, reading, and writing classes offered at colleges), and they were spending many semesters in these courses. This, in turn, created a barrier to enrolling directly in the college-level prerequisite courses needed for nursing programs. This anecdotal evidence has been echoed in national studies as well. The U.S. Department of Education found that 68% of all public two-year students are placed into remedial coursework (Chen, 2016, p. 16). Moreover, students entering remedial education very often do not make it beyond that point. The students placing into these classes are “74% more likely to drop out of college than first-time, full-time, non-developmental students” (Schak, Metzger, Bass, McCann, & English, 2017, p.7). Barry and Dannenberg (2016) indicated that students who take even one remedial class at post-secondary institutions are far less likely to graduate with an associate or bachelor’s degree than students who begin taking college-level classes.
This is not only an issue of graduation rates. Developmental education classes are tuition-based, so students unfamiliar with the post-secondary system in the US might assume them to be college-level, credit-bearing coursework. In some cases, students have used critical financial aid resources for these classes. Prospective CRA students at the Institute often ask us about the remedial coursework: “Will this class listed on my transcript give me credit like college classes do?” The message is clear: many of our students have mistakenly believed that remedial classes award credits that count toward their degrees. Because they typically do not, such developmental classes can be a costly alternative to the free preparation available through Adult Basic Education (ABE) in Minnesota.
The populations most affected by placement in remedial education programs are students of color, first-generation college students, English language learners, and those from lower socioeconomic backgrounds (Academic and Student Affairs, 2018a). Among all beginning postsecondary students, an estimated 58 percent of Hispanic students, 57 percent of black students, 39 percent of Pell Grant recipients, and 40 percent of first-generation college students enrolled in a developmental course between 2010 and 2014 (King, McIntosh, & Bell-Ellwanger, 2017).
Recognizing the challenges that face students who place into remedial courses, Minnesota State created a Developmental Education Strategic Roadmap, which outlines Minnesota State’s collective initiative for revamping the Developmental Education system (Academic and Student Affairs, 2018b). They hope to “re-imagine how students are placed into developmental or college-level courses, as well as how students can successfully complete required developmental-level courses and subsequent college-level gateway courses, enabling them to be on-track in the first year of pursuing their academic program” (p. 3). In fact, Strategic Goal 3.3 charges Minnesota State colleges to “Establish and/or strengthen partnerships with Adult Basic Education, community organizations, and/or other student support services (e.g. TRIO) to provide support for students in developmental education”(p. 2). Goal 5.3 requests they “Establish and/or strengthen bridging options that facilitate student placement into college-level courses (i.e., partnership with Adult Basic Education, summer bridge program, boot camp, course placement assessment prep, etc.)” (p. 2). Moving in this direction will improve partnerships with ABE and make college more affordable to students placing into remedial levels. One solution already in place within ABE is College Readiness Academy (CRA), which provides free college preparation in addition to ongoing navigation throughout the students’ college experience until graduation.
CRA is an effective and free approach that integrates academic skill-building focused on reading, writing, digital literacy, and study skills with intensive navigation (see Figure 1). The navigation typically includes guidance on issues like paying for school, campus culture, and overcoming barriers to progress.
Currently, CRA classes take place at three sites: The International Institute of Minnesota, Hubbs Center for Lifelong Learning, and Saint Paul College. Over the past three years, CRA graduates have moved on to college bypassing an average of 1.5 semesters of developmental education. Of the college-bound students who have completed CRA since 2015, our program has tracked that over 150 have enrolled in college-level courses and programs. Moreover, 90% of these students have persisted beyond the first semester in college, and their average GPA is above 3.5. Additionally, CRA navigators have helped students apply for and earn over $290,000 in college scholarships. CRA students are graduating with a certificate or associate or bachelor’s degree, majoring in a range of programs from nursing to accounting to computer science. By helping to bridge the opportunity gap through academic preparation and navigation support, CRA has increased access to higher education and ultimately to jobs and careers with family-sustaining wages.
In addition to the internal data tracked on students, we have also conducted external evaluations. The most recent evaluation in 2019 conducted by external consultants, The Improve Group, found that “CRA students earn more credits, are more likely to stay enrolled at college, and are more likely to graduate than Developmental Education students. CRA students may be more likely to succeed because of the academic skills and familiarity with the college campus they develop during the program. CRA students are also more likely to stay the course, in part because of the intensive support they receive from CRA navigators” (p. 3). Another interesting statistic is the comparison in credits earned by students who were able to earn a C or higher. And lastly, the evaluation also showed that “On average, CRA students enroll in more credits over their academic careers than [Developmental Education] students, resulting in higher per-student revenue for the colleges” (p. 5).
The evidence reported by the Improve Group (2019) and our own experience with learners indicates that CRA is a viable solution to the disparities in educational attainment and income by primarily serving low-income students of color, including New Americans.
There are a number of defining characteristics of CRA that contribute to the success of our students academically, including skills integration in class, more class time, a cohort model, and integrated learning objectives.
CRA lessons are organic in terms of skill-building because all of the academic components—literacy, oral, computer, and study skills—are fully integrated. Speaking activities, for example, reinforce new vocabulary and give students time to process reading content in preparation for writing tasks. There is a long list of measurable integrated skills objectives that students are assessed to achieve:
See Appendix A for a step-by-step lesson plan from a CRA classroom to see how these components are integrated into lessons.
In CRA, students have abundant face-to-face time with instructors: about twelve hours/week. CRA’s concentrated in-class time is a significant benefit to a student population that needs extra guidance with the opportunity to work in a quiet, focused atmosphere. When students undertake out-of-class assignments, extra class time allows the instructor to monitor their first steps, confirm they are on the right track, or redirect when they are not, before they commit time to a misguided task.
Another important feature of CRA is its strong student cohort. The Power of a Cohort and Collaborative Groups published by the National Center on the Study of Adult Learning and Literacy explored the use of cohorts in ABE and ESL settings, recommending that educators develop an “enduring and consistent learner cohort, employing practices by which students are regularly invited to engage in collaborative learning” (Drago-Severson, Helsing, Kegan, Popp, Broderick, & Portnow, 2001, p. 22).
CRA implements the cohort model; through managed enrollment and an intensive 12-hour-a-week schedule, students work with the same group of learners every day. Moreover, a significant proportion of class time is spent in pair work or small group activities. During this time, students negotiate meaning and share insights with peers, fostering group cohesion. Also, the classroom is configured such that 4-6 students sit around each table, which also encourages collaboration. And another way that CRA exemplifies the cohort model is by making sure the reading and writing tasks are based on socially relevant topics, encouraging students to have conversations that go deeper and build both self-knowledge and stronger relationships with peers. Individual and group presentations on these topics help students from diverse cultures to recognize their similarities.
All of this fosters a greater motivation and learning: “… cohort experiences seem to facilitate academic learning, increased feelings of belonging, broadened perspectives, and, at least by our participants’ report, learner persistence” (Drago-Severson, et al., 2001, p. 22). We continue to see evidence of the effectiveness of a strong learner cohort when students discuss assignments the moment they walk into class, when they stay after class to help one another, when they share personal stories, and when their relationships continue into college.
The inability to navigate college systems is a major obstacle to student success. In response, CRA incorporates proactive college navigation to help students prepare for college and complete their degrees. College navigators work with students step-by-step from their initial CRA class until they graduate from college. Unlike college advisors, who usually have hundreds—if not thousands—of students, college navigators usually work with approximately 50 students at a time (Robinson, 2016). This allows navigators to offer students the intensive case management they need as well as the wrap-around services that enable them to stay in school.
The navigators are present in class both as supplemental instructors and as participants on a weekly basis, assisting the instructor, interacting with students, and monitoring their progress. This allows them to build a meaningful relationship with each student by becoming familiar with their goals and life circumstances. Supplementing classroom navigation with effective individual support is only possible when there is a reasonable navigator/student ratio.
“For me the problem with college is not the concepts. It’s how to better understand what I had to do. Without CRA I think it would have taken me a long, long time.”
-Gnande, CRA Graduate
College navigators’ support to students includes:
“I learned in classes to express myself and to use my critical thinking skills to solve problems. The navigator is always available to answer questions and helps to put in place the college process.”
-Christelle, CRA graduate
An obvious first step to establishing a high-quality college readiness program is to build a team of experienced instructors and navigators. CRA instructors have had combined experience in adult basic education, ESL instruction, and higher education. Navigators typically bring experience with college access organizations or social work. Regardless of professional background, it is critical for instructors and navigators to be flexible, dynamic, creative educators and problem-solvers. Working with a diverse adult population at a variety of sites necessitates a willingness to adjust practices to best fit the needs of the particular group. In order to build trust between navigators and students, navigators are fully integrated into the classroom.
In order for a program to be viable and to improve over time, it is imperative to collect comprehensive data, including student educational background information, progress, and outcomes. Funders will want detailed information on demographics, success rates, and many other metrics, so a uniform method is needed for tracking pertinent information. A customer relationship manager (CRM) is a beneficial tool for data tracking and synthesis. A strong program will also perform frequent self-evaluation and adjust methods accordingly; a CRM is useful for this practice as well. Some examples of frequently accessed metrics include:
Finally, one of the most important elements of creating a successful, sustainable college readiness program is building and maintaining relationships and institutional trust with local colleges, which requires significant time and effort. This process is ongoing and constantly evolving, but the more support a college can give the program, the more impact the program can have. The relationship is symbiotic: CRA provides colleges with a steady stream of prepared, motivated students; in turn, the college can afford CRA students unique benefits, including increased academic support, flexibility with entrance requirements, and exemption from college entrance exams. Perhaps most importantly, students will gain trust in the college and become fully confident in their identity as college students and, ultimately, graduate.
Academic and Student Affairs. (2018a). Developmental education plan report to the legislature. Minnesota State. Accessed 1 November 2019 at http://www.minnstate.edu/system/asa/docs/Minnesota%20State%20-%20Developmental%20Education%20Plan%20Report%20to%20the%20Legislature%20-%202.15.18.pdf
Academic and Student Affairs. (2018b). Developmental education strategic roadmap: Minnesota State’s strategic plan for developmental education redesign. Minnesota State. Accessed 1 November 2019 at http://www.minnstate.edu/system/asa/studentaffairs/academicreadiness/docs/Developmental-Education-Strategic-Roadmap.pdf
Barry, N. G., & Dannenberg, M. (2016). Out of pocket: The high cost of inadequate high schools and high school student achievement on college affordability. Education Reform Now. Accessed 1 November 2019 at http://edreformnow.org/wp-content/uploads/2016/04/EdReformNow-O-O-P-Embargoed-Final.pdf
Chen, X. (2016). Remedial course-taking at U.S. public 2- and 4-year institutions: Scope, experiences, and outcomes (NCES 2016-405). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Accessed 1 November 2019 at https://nces.ed.gov/pubs2016/2016405.pdf
Drago-Severson, E., Helsing, D., Kegan, R., Popp, N., Broderick, M., & Portnow, K. (2001). The power of a cohort and of collaborative groups. NCSALL. Accessed 1 November 2019 at http://www.ncsall.net/index.php@id=254.html
King, J. B., McIntosh, A., & Bell-Ellwanger, J. (2017, January). Developmental education challenges and strategies for reform. U.S. Department of Education. Accessed 1 November 2019 at https://www2.ed.gov/about/offices/list/opepd/education-strategies.pdf
Robinson, W. (2016). Community college academic advisors and the challenge of access in the age of completion (Unpublished doctoral dissertation). Iowa State University, Ames, IA. Accessed 1 November 2019 at https://lib.dr.iastate.edu/cgi/viewcontent.cgi?article=6062&context=etd
Schak, O., Metzger, I., Bass, J., McCann, C., & English, J. (2017). Developmental education challenges and strategies for reform. U.S. Department of Education. Accessed 1 November 2019 at https://www2.ed.gov/about/offices/list/opepd/education-strategies.pdf
The Improve Group. (2019). College Readiness Academy (CRA) 2018-19 Evaluation Findings. St. Paul, MN: The Improve Group.
Example of Step-by-Step of Skills Integration in CRA Classroom
Authentic reading: Adapted from “The Perils of Polygamy”, published in The Economist – December 23, 2017
This article claims that polygamy is responsible for instability and violence in South Sudan. The instructor abbreviated it somewhat and added footnotes for essential vocabulary that could not be understood from context. The class approached the article like this:
This article describes a procedure for training second-language writing raters to use scoring rubrics, and presents ideas for practical adaptation or research projects associated with the training procedure.
Imagine a novice language teacher doing the following:
These descriptors are examples from a commonly used rubric for assessing second language writing (Jacobs et al., 1981), in which the example descriptors above represent only 5% of all the descriptors that a rater is supposed to be familiar with while reading and rating compositions.
In programs in which graduate teaching assistants (TAs) need to be trained over a short period of time to rate second language writers’ essays and make course placement decisions, rater training must ensure that the novice raters quickly become familiar with the descriptors used in a scoring rubric. Regardless of the specific rubrics used, raters are typically expected to demonstrate knowledge of pre-determined descriptors (such as some knowledge of subject, adequate range, limited development of thesis, mostly relevant to topic but lacks detail) and perform the task of applying this knowledge consistently. Such expectations may not be met easily. Joe, Harmes, and Hickerson (2011) show, for example, that lack of transparency in rating scale descriptors can be a factor influencing raters’ performance. At the same time, rater-related factors can also add to the challenge in achieving reliable outcomes in rating (see Barkaoui, 2011 for a comprehensive review).
In this article, first, I will briefly review studies on raters and rater training. Then, I will describe a gap in published studies on rater training. Finally, I will introduce a rater training procedure currently implemented at a four-year university in the U.S. Midwest, for two purposes:
Among several rater-related factors that could influence raters’ interpretation and application of rating scales, raters’ experience (novice vs. experienced) seems to be the most frequently researched factor (Barkaoui, 2011; Greer, 2013; Hamp-Lyons, 1989; Harsch & Martin, 2012; Joe et al., 2011; Weigle, 1994). Research findings on this issue are mixed, contrary to what might be assumed (i.e. the more experienced, the more proficient). In fact, the mixed findings might be a function of the complexity in the way raters’ experience interacts with non-rater factors. In Barkaoui (2011), for example, with regard to severity in rating, novice raters and experienced raters behaved more similarly when using analytic scales than they did when using holistic scales. In other words, raters’ experience impacts ratings differently depending on the type of rating scales used.
The intriguing nature of raters’ experience as a factor was documented with great detail in Joe et al. (2011), which explored rater cognition based on data collected through verbal protocols. In this study, eight faculty experts (experienced raters) and eight undergraduate students (inexperienced raters) participated in rating oral speech performances. Both groups were trained to use an analytic scoring rubric, which included 39 features comprising ten competency dimensions relevant to the construct of the speech performances to be evaluated. The study found that inexperienced raters started out paying attention to rubric features more consistently than did experienced raters, who were found to pay attention to construct-irrelevant features (i.e. features not listed in the rubric) at a higher rate than inexperienced raters did. Over time, however, inexperienced raters attended to rubric features less and less while attending to construct-irrelevant features (such as the use of note cards or memorable thesis statement, which were not included in the rubric) more and more.
Such findings are quite alarming in that not all changes exhibited by raters as they become more experienced seem to be in the expected direction – i.e. experience is generally expected to be a positive factor. Some researchers even suggest that raters should not be selected based on teaching experience as it is not a significant factor (Royal-Dawson & Baird, 2009). Although research findings about rater experience might be mixed, many researchers have emphasized the importance of rater training in enhancing the quality of raters’ performance (Greer, 2013; Lovorn & Rezaei, 2011; Weigle, 1994). The following section will review studies on rater training.
One of the aims in most rater training involves monitoring rater behavior associated with rater-related factors such as experience, rating style, or rating preferences, and then providing feedback accordingly to achieve the ultimate goal of increasing inter-rater reliability (i.e. different raters performing similarly to one another). On the one hand, studies such as Pufpaff, Clarke, and Jones (2015) and Weigle (1994) reported that rater training did not improve inter-rater reliability. On the other hand, Weigle emphasized the point that “rater training cannot make raters into duplicates of each other, but it can make raters more self-consistent” (p. 32). This statement, then, naturally leads to the question as to what factors might contribute to developing rater self-consistency. Although there has not been much research that directly explored this question, several studies have reported the positive effects of rater training on rater performance in different aspects of the rating task.
In Greer (2013), novice raters practiced assessing ESL compositions following a training workbook, which included experienced raters’ feedback on the same compositions that the novice raters were evaluating. After the training, the novice raters reported increased confidence in their rating performance. In another study based on a two-month rater training program (Harsch & Martin, 2012), 13 novice raters completed rigorous weekly assignments consisting of tasks commonly included in rater training such as individual practice and group discussion, using over 1700 writing samples (whittled down from an initial set of over 6000 samples). Although the scope of the study is truly impressive, it is the depth of its rater training that makes it rather unique and remarkable. As a part of their weekly assignments, for example, the novice raters were actively engaged in revising the wordings on the rating scale. In fact, researchers recommend engaging raters in the development of rating scales (Barkaoui, 2010; Stevens & Levi, 2005). Harsch and Martin (2012) concluded that rater agreement increased when a revised rating scale (i.e. revised based on the novice raters’ discussion and input during the training period) was used.
Because the task of rating is replete with a myriad of interacting factors that could influence the process and outcome of rating, research-guided rater training may be essential in most contexts. For example, when training raters, feedback should be provided immediately after rating has occurred (Knoch, 2011). Rater training should promote detailed and analytical understanding of the scoring rubric (Lovorn & Rezaei, 2011; Rezaei & Lovorn, 2010). An eye-movement study on raters’ use of a scoring rubric showed that even the physical layout of the rubric can affect raters’ attention to each category on the rubric (Winke & Lim, 2015). These are just a few examples of published studies that could guide the design of a rater training.
In most published studies that either directly or indirectly report the outcome of rater training (Barkaoui, 2011; Joe et al., 2011; Knock, 2011; Lovorn & Rezaei, 2011; Pufpaff et al., 2015; Weigle, 1994), it appears common to follow variations of the same approach, categorically speaking, in the way raters (regardless whether novice or experienced) are initially introduced to a rating scale (either holistic or analytic). First of all, surprisingly, many studies (Barkaoui, 2011; Knock, 2011; Lovorn & Rezaei, 2011) do not provide sufficient detail regarding exactly how raters are introduced to the rating scales selected in their respective studies. Of the studies (Joe et al., 2011; Pufpaff et al, 2015; Weigle, 1994) that do provide some limited information regarding this part of rater training, the common approach seems to be Present and Clarify/Explain with respect to the descriptors on the rating scale.
This clearly is an example of what is defined as the hierarchical approach: “passing onto raters a predetermined view on how they are to interpret the scale wordings, using pre-assessed scripts (so called ‘master codes’) which are not to be discussed but to be accepted and internalised” (Martin & Harsch, 2012, p. 233). On the one hand, researchers have repeatedly identified the difficulty that raters experience when trying to understand rating scale descriptors (Barkaoui, 2010; Greer, 2013; Hamp-Lyons, 1989; Harsch & Martin, 2012; Joe et al., 2011). On the other hand, as described above, variations of a very top-down approach seem to prevail when it comes to training raters to become familiar with the wordings on the rating scale.
From this perspective, Harsch and Martin’s (2012) study, as reviewed in the preceding section, may be considered as an exception in that the raters in their study were engaged in a series of in-depth tasks attending to, analyzing, and revising the descriptors on the scale. These researchers also emphasize the importance of “reaching consensus about how to interpret scripts with reference to scale descriptors” (p. 233). As mentioned earlier, the rater training in their study spanned over a two-month period. Most readers would agree that a rater training program like that, while both impressive and exemplary, is anything but feasible in most real contexts. The reality of most rater training is likely to resemble the two-hour norming session referred to as a typical rater calibration procedure (Weigle, 1994, pp. 7-8).
Here is a question, then, that a concerned program administrator or teacher educator might ask: Do people use the top-down approach because they are constrained to the typical two-hour calibration procedure (or however many hours it might take but not the luxury of two months)? Acknowledging the “time- and resource-intensive” nature of their approach, Harsch and Martin (2012, p. 244) recommend realistic adaptations using existing rating scale descriptors (i.e. not necessarily attempting to revise the descriptors as their raters did). So it appears that, although there is a huge gap between a deeply engaging, albeit extremely unfeasible, approach to rater training and a more commonly practiced top-down approach, careful retooling of the top-down approach can help fill this gap. In the next section, I will introduce an authentic example of a rater training procedure which is characterized as a rater-centered bottom-up approach. (Readers can rest assured that this procedure will not require two months to try!)
The rater training procedure described here, as an example of a rater-centered bottom-up approach, has been implemented in an authentic test context. The following provides some background information about the context:
|Location||Four-year university in the U.S. Midwest|
|Purpose||Placement decisions for writing courses in English for Academic Purposes|
|Raters||Graduate teaching assistants in an MA-TESL program (1st~4th semester)|
|Target Texts||Academic essays written by second language writers|
|Scoring Rubric||Locally revised version of Composition Profile by Jacobs et al. (1981)|
Table 1. Training Protocol
Training Protocol (Rater-centered Bottom-up Approach)
[I] Individually [SG] Small Groups (3~4) [WG] Whole Group [T] Trainer M: Materials
|Description of Steps in the Training Protocol||Rationale for Each Step|
|1. Activating existing knowledge & expectations about academic writing|
|[SG] Brainstorm & consolidate existing knowledge & expectations about academic writing
[WG] Discuss and summarize
M: Brainstorm sheets
|Step 1 allows each rater to activate existing knowledge; allows each rater to generate his/her own language to describe features of writing; compile & share entire group’s ideas.|
|2. Evaluating a writing sample based on existing knowledge – without any rubric|
|[I] Read Essay #1; Write any/all notable features, good & bad, one feature per sticky-note; Place sticky notes in worksheet; Give a holistic score
[SG] Compare notes placed in individual worksheets; Compare holistic scores
M: Essay #1; Sticky-notes; Worksheet
|Step 2 allows each rater to apply existing knowledge; allows each rater to notice features in the writing with no constraints; exposes raters to writing features noticed by others.|
|3. Familiarization with rating scale descriptors|
|[I] Read rating scale descriptors and criteria
[SG] Discuss & help each other understand concepts & terminologies
[WG] Review & clarify concepts & terminologies
M: Rating scale descriptors & criteria handout; handout on Content-to-Form continuum in writing
|Step 3 introduces descriptor language to raters; helps raters to conceptually align their own language with descriptor language; helps identify & clarify gaps between rater-generated language and descriptor language.|
|4. Matching current knowledge with rating scale descriptors|
|[SG] Discuss each note on sticky-notes; Transfer & match each sticky-note with descriptors in the Descriptor Handout
[WG] Discuss & further clarify descriptors based on questions from SGs
M: Descriptor Handout (one copy for each SG)
|Step 4 allows raters to map their own unconstrained observations onto descriptors; helps identify, discuss, & resolve writing features that are difficult to map onto descriptors; more importantly, helps raters understand descriptors with self-generated concrete examples.|
|5. Practice using rating scale descriptors without scores|
|[I] Read Essay #2; Use the Descriptor Handout to mark relevant descriptors
[SG] Compare individuals’ markings on descriptor handout
M: Essay #2; Descriptor Handout
|Step 5 allows raters to practice using the descriptors directly without scaffolding (i.e. no self-generated descriptive notes as with Essay #1); allows another chance to focus on the descriptors with no burden to score the essay numerically.|
|6. Familiarization with the complete version of rating scale (with score indicators)|
|[T] Introduce the complete version of the rating scale (with score indicators) and explain
[I] Based on markings on Descriptor Handout (from Step 5) and using the rating scale (with score indicators), numerically score Essay #2
[SG] Compare scores for Essay #2
M: Complete version of rating scale
|Step 6 finally exposes raters to the actual rating scale with score indicators; helps raters perform the task of numerical scoring (not exactly the same as mapping observed writing features onto descriptors); helps deal with two different subtasks (i.e. identifying matching descriptors vs. numerical scoring) with more clarity.|
|7. Practice using the complete version of rating scale with a familiar essay|
|[I] RE-read & score Essay #1 using rating scale
[WG] Discuss the results & rationale of rating scale
M: Complete version of rating scale with score indicators
|Step 7 allows raters to apply the rating scale in evaluating a familiar writing sample; provides raters with an opportunity to review & re-assess their own initial evaluation of Essay #1 (performed prior to the introduction of the rating scale).|
|8. Remaining steps in the protocol|
|There are a few more steps in the protocol, which are beyond the focus of this article. Some of the remaining steps are similar to commonly practiced norming procedure, and some steps are specific to the local test context.1|
As Table 1 shows, the training protocol follows a rater-centered bottom-up procedure, which affords the raters step-by-step scaffolding to develop an understanding of and the ability to apply the descriptors on the rating scale. The procedure promotes activating existing knowledge and acquiring new knowledge of technical concepts/terminologies through a sequence of small tasks rather than through top-down imposition of abstract descriptors onto the raters. For many novice raters, learning to use a rating scale with pre-determined descriptors includes an element of language acquisition. It is not a mere coincidence that, in many ways, the procedure introduced here resembles language learning activities based on the task-based language teaching (TBLT) approach, in which language acquisition occurs as a natural part of successful completion of communicative tasks (Van den Branden, 2006).
The rater training procedure introduced here also provides scaffolding for one of the subtasks of rating that present a unique challenge for most raters, namely translating descriptors into numerical scores. Studies have shown that both novice raters (Greer, 2013) and experienced raters (Hamp-Lyons, 1989) find this subtask very difficult. In the procedure described in Table 1, raters are assisted to deal with this challenge in two ways: (1) initial steps in the procedure focus on the descriptors without the ‘burden’ of matching them with numerical scores; and (2) the complete version of the rating scale, a locally revised version of the composition profile by Jacobs et al. (1981), presents numerical scores in subsets to match raters’ judgments based on descriptors.
Unlike the two-month rater training described in Harsch and Martin (2012), which is quite impressive and ambitious, the training procedure introduced in this article is bundled with realistic and practical advantages:
The last two items directly address “the time- and resource-intensive” challenge of Harsch and Martin’s (2012, p. 244) otherwise exemplary rater training model. Hopefully, these practical advantages would encourage many readers of this article to consider employing this training procedure.
Aside from the obvious practical advantages, the most critical advantage of this procedure, at least based on informal observations during several semesters of implementation, involves the change in dynamics and roles between the trainer and the raters in training. As the protocol shows, at each step, raters are actively engaged in small doable tasks either independently or in collaboration with peer raters. Because the steps are sequenced to promote learning-by-doing, the procedure does not require much top-down talk from the trainer.
When this new procedure was first implemented a few semesters ago, raters who had experienced the previous format resembling the Present and Clarify/Explain approach enthusiastically commented that the new procedure felt stress-free, engaging, and helpful. As the trainer in this incidence, I too noticed unexpected changes when first implementing the new procedure. It felt as though I did not have to do anything during the procedure because the raters were doing all the work for themselves!
Although the rater-centered bottom-up training procedure is strongly recommended, the caveat is that it has not been empirically tested. First, interested readers are encouraged to consider field-testing this procedure in their various test contexts. It can be modified to fit the needs and capabilities of each context. One example might be replacing the use of sticky-notes with a digital/online tool to help raters generate, compile, and compare the features they observe in the writing sample they evaluate. This is actually an attractive idea, which can lead to the next point of this discussion, namely research possibilities.
The hands-on aspect of using sticky-notes is actually a very positive and valuable element of the procedure, and it helps raters ease into the sequence of tasks in the procedure. Its non-digital nature, however, has been an obstacle in converting the notes into analyzable data. These rater-generated notes can reveal interesting aspects of rater cognition. Designing empirical studies to capture such data to learn more about rater cognition would not only benefit the field of language education but also the field of education in general. In fact, in the special issue of the journal Educational Measurement: Issues and Practice devoted to rater cognition, Myford (2012) emphasizes that more research on rater cognition is needed.
Other research questions worth exploring including obvious ones such as Is the bottom-up rater training procedure more effective than the more commonly practiced top-down approach? Is there any difference between the two approaches in improving rater self-consistency (emphasized as the main benefit of rater training)? These are just a few examples, and readers are encouraged to pursue their own research questions associated with the rater-centered bottom-up rater training procedure introduced in this article.
This article started by asking readers to imagine a novice rater performing a presumably learnable, but indeed tremendously challenging, task of using a rating scale to make decisions about learners’ proficiency. Anecdotal evidence and informal observations suggest that rater training, for both novice and experienced raters, need not be like that – top-down, opaque, and anxiety-inducing. Instead, a rater-centered bottom-up approach can make the process more transparent and positively engaging. However, for this statement to be generalizable, we need empirical evidence, and this research topic is open to any interested readers.
Barkaoui, K. (2011). Do ESL essay raters’ evaluation criteria change with experience? A mixed-methods, cross-sectional study. TESOL Quarterly, 44, 31057.
Greer, B. (2013). Assisting novice raters in addressing the in-between scores when rating writing. (Master’s thesis). Retrieved from BYU ScholarsArchive.
Hamp-Lyons, L. (1989). Raters respond to rhetoric in writing. In H. W. Dechert & Raupauch (Eds.), Interlingual processes (pp. 229-244). Tubingen: Buner Narr.
Harsch, C. & Martin, G. (2012). Adapting CEF-descriptors for rating purposes: Validation by a combined rater training and scale revision approach. Assessing Writing, 17, 228-250.
Jacobs, H., Zinkgraf, S., Wormuth, D., Hartfiel, V., & Hughey, J. (1981). Testing ESL composition: A practical approach. Rowley, MA: Newbury House.
Joe, J. N., Harmes, J. C., & Hickerson, C. A. (2011). Using verbal reports to explore rater perceptual processes in scoring: a mixed methods application to oral communication assessment. Assessment in Education: Principles, Policy & Practice, 18, 239-258.
Knoch, U. (2011). Investigating the effectiveness of individualized feedback to rating behavior – a longitudinal study. Language Testing, 28, 179-200.
Lovorn, M. G. & Rezaei, A. R. (2011). Assessing the assessment: Rubrics training for pre-service and new in-service teachers. Practical Assessment, Research & Evaluation, 16, 1-18.
Myford, C. M. (2012). Rater cognition research: Some possible directions for the future. Educational Measurement: Issues and Practice, 31, 48-49.
Pufpaff, L. A., Clarke, L., & Jones, R. E. (2015). The effects of rater training on inter-rater agreement. Mid-Western Educational Researcher, 27, 117-141.
Rezaei, A. R. & Lovorn, M. (2010). Reliability and validity of rubrics for assessment through writing. Assessing Writing, 15, 18-39.
Royal-Dawson, L. & Baird, J. (2009). Is teaching experience necessary for reliable scoring of extended English questions? Educational Measurement: Issues and Practice, 28, 2-8.
Stevens, D. D. & Levi, A. (2005). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Virginia: Stylus Publishing, LLC.
Van den Branden, K. (Ed.). (2006). Task-based language education: From theory to practice. Cambridge, UK: Cambridge University Press.
Weigle, S. C. (1994). Using FACETS to model rater training effects. Paper presented at the Language Testing Research Colloquium (Washington, DC).
Winke, P. & Lim, H. (2015). ESL essay raters’ cognitive processes in applying the Jacobs et al. rubric: An eye-movement study. Assessing Writing, 25, 38-54.