Michelle Benegas & Suzanne Gilchrist McCurdy

Welcome to MinneTESOL Journal volume 38, issue 1! We are happy to share a collection of articles that highlight the ingenuity of English language practitioner-scholars from a variety of contexts.

Jeanne Beck and Sebnem Kurt describe a teacher’s journey of creating and implementing a new unit for a middle school English learner and technology curriculum in “Project-Based Language Learning with 3D Printers: Integrating Technology, Language Form/Function, and Assessment into a Middle School CAD Unit.” Beck and Kurt share their perspectives and takeaways from the project as well as easily accessible materials for implementing such a project in readers’ own classroom.

In another report straight from the classroom, Ami Christensen and Kaishan Kong share an innovative project borne from COVID restrictions at their university in “Flipgrid Classroom Conversations: International Virtual Pen Pal Exchange.” The authors detail their process and their learners’ experiences using Flipgrid as a tool to gain and share cultural knowledge and experiences while engaging in authentic language tasks.

Crossing boundaries in “Does Translanguaging Work in Adult ESL?” Alayna Klco argues that pedagogical translation, a translanguaging approach developed for K-12 classrooms, can be adapted for use in adult education. Klco delineates pedagogical translation and the work of Project TRANSLATE and describes modifications that may be used in the adult education context.

Please read, enjoy, and share the research and practice ideas in this issue of MinneTESOL Journal! Stay connected to our MinneTESOL Journal community by following the journal on Facebook and on Twitter (@MnTESOLjournal).

 

DOWNLOAD FULL ARTICLE

Jeanne Beck & Sebnem Kurt

Meaningfully integrating project-based learning, language learning, and technology into the classroom provides 21st century skills for the real world. This article describes how one middle school teacher integrated project-based learning, technology, form/function instruction, and formative assessment into a 3D printing unit.
Keywords: 3D printing, Project-based learning, assessment, Project-based language learning, middle school, curriculum design, technology

Introduction

Project-Based Learning (PBL) is a student-centered approach to teaching and learning that can seamlessly blend language learning and technology integration (Beckett & Slater, 2020). With foundations in Dewey’s experiential learning philosophy (1916) and Kilpatrick’s project method (1918), PBL has become a powerful movement in education, benefiting classrooms through student engagement, collaboration, critical thinking, and problem solving, among other positive effects (Buck Institute for Education, 2022). Infusing PBL with language learning, sometimes referred to as Project-Based Language Learning (PBLL), this approach has paved the way for meaningful second and foreign language learning, documented through research, in countries all around the world (Beckett et al., 2020). Combining relevant apps and web-based tools, PB(L)L projects can powerfully shape learning experiences to bring about authentic learning for students (Alharthi et al., 2021).

Beckett and Slater, well-known PBL/PBLL scholars, argue that projects can engage students in deeper learning with and through technology for learning language, content, and other relevant skills (Beckett & Slater, 2017). They have also correctly stated that PBL with intentional technology integration, language form/function instruction, and assessment has also been limited in the PBL research (Beckett & Slater, 2020; Beck et al., 2021). To help fill this gap, this paper describes a 3D printing PB(L)L unit designed for both native and non-native English speaking middle school students in a technology course at a rural school, which was implemented from 2017-2019. It is hoped that this paper not only highlights the potential that creative technology-infused projects can have on learning, but also inspires other teachers to write about the amazing work that they are doing or have done in their classrooms. 

Literature Review

3D Printing for Authentic Learning

3D printing has come onto the education scene within the last two decades as a high-tech tool for providing engaging STEM (science, technology, engineering, and math) projects, real world skills, teamwork, experimentation, and problem-solving skills (Lacey, 2010). Intrinsically linked with PBL, multiple affordances of 3D printing in educational contexts have been documented through exciting projects including: a project where kindergarten and first graders designed new playground equipment using 3D printing in the USA (Wendt & Wendt, 2015), an interdisciplinary 3D printing program with middle school students in South Korea (Shim & Lee, 2019), and in facilitating a junior/senior high school service-learning project where students 3D printed hands and arms for amputees through E-NABLE (Enabling the Future, 2022; Suchow, 2016). These exemplary studies show that 3D printing PBL is possible at all grade levels and can integrate meaningful content for students.

In addition to 3D printing PBL research involving students, several 3D printing-related studies have involved pre- and in-service teachers. In a survey of teachers concerning their interest in learning about augmented reality (AR), virtual reality (VR), and 3D printing, Trust et al. (2021) found that most of the teachers, who were enrolled in a MOOC (Massive Open Online Course), described wanting to learn how to use and troubleshoot the devices and fit these tools within their teaching pedagogy. Less than one quarter of the teachers, however, indicated interest in how to integrate these devices within their curriculum (Trust et al., 2021). Mixed results concerning 3D printing curricular integration were similarly found with middle school history teachers (Maloy et al., 2017). Teachers found it difficult to integrate 3D printing into their curriculum, and both teachers and students initially found it difficult to use the Computer-Aided Design (CAD) program Tinkercad. After implementing 3D printing projects in their classes, however, their students found the projects fulfilling and some teachers found value in the projects and in altering the “teacher-as-expert/student-as-novice relationship” (Maloy et al., 2017, p. 244). 

In terms of pre-service studies, Novak and Wisdom (2018) highlight a 3D printing project for elementary pre-service teachers in science education where the projects significantly reduced the teachers’ anxiety about teaching science and improved their science interest and perceived competency in K-3 science standards. Moreover, Verner and Merksamer (2015) present a program in which they used the create-design-implement-operate (CDIO) approach to teach pre-service technology teachers and other science and engineering majors about teaching design and manufacturing. These studies with pre-service teachers suggest that future in-service teachers may enter schools with more 3D printing readiness.

3D Printing PBLL for Second and Foreign Language Learning

While less research has been done on 3D printing PBLL in second and foreign language contexts, researchers at the college level have demonstrated its effectiveness. 3D printing PBLL can be seen in a course for international second language graduate students in Japan (Roy, 2017), in an introductory college German class in the USA (Emm & Hawkins, 2020), and in a 3rd year college Russian course in the USA (Enkin et al., 2021). In a research study surrounding a 3D printing course at a 7th-12th grade summer STEM camp, some participants hailed from Italy and the American participants were of diverse ethnic backgrounds, suggesting that language learners might have been involved in the study (Kwon, 2017). Since L1s were not collected, however, second/foreign language learning implications for language learners were not explored. Similar to how the first author (Beck) had only presented on 3D printing in her middle school English language (EL) and technology classroom at conferences and within professional learning communities (i.e., Beck, 2018, 2019), it is very possible that innovative 3D printing work is being done in language classrooms, however, this information is not reaching research communities.

Integrating 3D Printing into the Middle School EL and Technology Curriculum

With the many benefits espoused by PBL research, the first author’s goal was to bring these learning experiences to her classroom. As a half time EL/half time technology teacher at a rural Missouri middle school from 2017-2019, she had the unique opportunity to design the school’s new technology curriculum and develop content for the EL classes. The middle school, a Title I school with approximately 10% ELs, had recently gone 1:1 with Chromebooks and was aiming to expand their students’ 21st century skills. Equipped with a new 3D printer, the teacher was encouraged by administrators to create 3D printing projects for the diverse student body. Designing a CAD and 3D printing unit with both nonnative ELs and native English speaking students in mind, the unit introduced activities that taught or reinforced language learning and were formatively assessed in creative ways.

3D Printing Unit Overview

The 3D printing unit was designed as a two-week unit for the middle school technology course. Created as a service-learning unit, the unit began with the goal of making 3D printed objects to help others. The first part of the unit introduced relevant vocabulary and the real world applications of 3D printers, as well as provided a refresher on the metric system. To aid in CAD design, TinkerCAD was used in the classroom, including its tutorials as well as guided CAD design practice activities structured by the teacher. At the same time, a survey was sent to teachers and school staff in the district to ask what classroom materials or objects they could use for their classrooms or workspaces that could be 3D printed. Through the Google Form, staff members would select an object they could use for their classroom (e.g., pencil holder, name plate, Post-It Note holder, magnet, keychain), it’s shape (e.g., rectangular, oval, hexagonal), color, and if they would like any specific text or images on their object. 

Constraining the project to include requests from teachers/staff members was born out of necessity. In the first iteration of this project no such restrictions were made, which resulted in a few projects that were not quite useful or were unprintable (at least, with the teacher’s level of 3D printing knowledge). Given the amount of time it takes to print a 3D object and how much students want to see their design printed, implementing a student-created Google form in the second iteration of the project onward that narrowed the scope of what students could design proved to be a win-win. While this may seem limiting, students still had many design choices to make as a team. For instance, a teacher might request a blue hexagonal-shaped pencil holder with their name and school logo on it; for this, students would need to decide what dimensions to make the pencil holder, how thick or thin to make the base or the walls of the pencil holder, which font to make the text, where to effectively place the text and school logo, which school logo to use, whether to embed the text/logo so that it looked etched into the surface or have it protrude out, and if there were any value-added design elements the group wanted to add (e.g., adding patterns or borders to the edges of the object).

During the second week, the survey results from the school staff were analyzed and teams of 2-3 were created and drew a conceptual design of what they would be making, spending several days working on the design together. At the end of the two weeks, the 3D designs were submitted to the teacher and the focus shifted to writing a short reflection paper about the project. The reflection writing activity was completed in Google Drawings, where a screenshot of the completed CAD design could be embedded. The reflections and completed 3D prints were placed on display in the middle school trophy cabinets before being given to the school staff members who requested them. In total, this unit was delivered 16 times by the teacher over the course of the two years at the middle school building.

Integrating Form/Function and Assessment

Integrating academic English and the language of technology was important in designing the PBL unit. Specific instruction on language form and function has been described as lacking in PBL research, which can be seen as a missed opportunity for learners (Beckett & Slater, 2020). While it might seem incompatible with a PBL activity at first glance, intentionally integrating form and function can be done in multiple parts of the unit in less intrusive ways through activities that introduce students to vocabulary and grammar (form) and what can be done with language (function), e.g., requesting information, describing processes, or drawing conclusions.

While designing the 3D printing unit, focus was placed on introducing and practicing the new vocabulary and concepts related to 3D printing in both speaking and writing. New words and phrases including filament, CAD program, and SVG image would need to be introduced, modeled and practiced. Moreover, EL students can particularly need additional support with prepositional phrases and direction-related words, for example, telling a classmate when designing a keychain to, “Put the text on top of the rectangular shape” or “Cut a hole out of the rectangle using a cylindrical shape.”

To practice CAD and 3D printer-related language, the teacher developed a 3D printing hyperdoc, an interactive worksheet with links to different websites and videos related to 3D printing. By reading the text, looking at images, and watching short videos online, new words and concepts were introduced and reinforced organically. The hyperdoc activity was modeled by the teacher and completed with a shoulder partner, in which ELs and non-ELs could be grouped together to support each other while completing the activity. As such, the hyperdoc was a simple yet useful formative assessment in the early stages of the unit.

New concepts were also reinforced with a short video on 3D printers that used the same new language and other worksheets. Before watching the video, students were asked to pay attention to specific parts of the video, and then had time to think-pair-share after the video before discussing the answers as a whole class. Additionally, a worksheet on measuring in centimeters and millimeters (Super Teacher Worksheets, 2022) was adopted for practicing the metric system, helping reinforce mathematical principles useful in CAD. The worksheet, the only physical paper used in the entire 9-week course, helped students review centimeter and millimeter measurement by using rulers and measuring objects in the classroom.

Within the 3D design process itself, language form and function were reinforced through focus on the designs that teams were tasked with making as well as the survey that they sent to the teachers. As survey responses were of varying degrees of specificity and clarity, they needed to be read, comprehended, and interpreted so that the staff member’s wishes were honored. Moreover, many of the staff members requested objects with text on them, such as the staff member’s name or famous teacher quotes. While English errors are a part of learning for both native and non-native speakers, it was important that any errors be corrected before printing as reprinting a 3D print would cost multiple hours and waste materials.

The 3D prints themselves were assessed by the teacher in terms of the object’s printability, feasibility of being printed in a timely and resource-friendly manner, and the extent to which students honored the requests of the teachers/staff who requested the print. The rubric categories were: theme (whether the students took the request seriously and created a purposeful object), color scheme (whether students used only 1-3 colors and only the filament colors that were currently available), size (limited to 8cm in height, 2-3mm thickness to reduce print time), detail (adding text, textures, images, etc.), and teamwork. Issues such as text floating in space were accounted for in the “theme” category as these errors, which were routinely checked for during the project-making process, would result in 3D printing errors.

Finally, reflective writing is an important aspect of project work, as the writer can be led to personal discovery and deeper meaning (Rocha, 2006). To integrate technology into the writing process and increase its accessibility, the teacher created a graphic organizer in Google Drawings with four quadrants with questions to answer, plus a space to add a screenshot of the 3D design. The rubric evaluated the amount of information shared, the quality of information shared, and minimal English errors. Displaying the reflection writing next to the 3D prints in the trophy case at school provided students, staff, and visitors a glimpse into the design process.

As a result, these reflections and 3D prints not only provided an assessment for technology-infused PB(L)L (Chen & Hirsch, 2020), but showed evidence of acquisition of technology-related vocabulary and concepts. The prints and write-ups also aided the teams in practicing how to discuss the design process with the 3D print recipients. As a teacher, seeing the students give the printed object to the recipient and explain how they designed it was the most rewarding part of the experience, and the staff and students seemed to enjoy it a lot too.

Conclusion

Authentic technology-infused PB(L)L units can meaningfully combine language, content, and 21st century skills, which can be measured consistently through formative assessment. After running the unit several times, by popular demand, the teacher expanded to create a specific 3D printing PBL unit for English learners who had taken the technology course and wanted to improve their skills further, as well as a before school mini-3D printing course for students who could not enroll in the technology course due to scheduling conflicts. Combined with community coverage from the local newspaper and school awards given for best in 3D design, these units had a great impact on the school, shining a light on the affordances of technology and exciting innovations and careers in STEM.

References

Alharthi, A., Beck, J., Bordbarjavidi, F., Gale, J., Garib, A., Guskaroska, A., Kurt, S., Lestari, F., & Zawadzki, Z. (2021). Effective applications and web-based tools for increasing student engagement in project-based language learning. MIDTESOL Journal, 4. https://midtesol.org/wp-content/uploads/2021/12/5-Effective-Apps-and-Web-based-Tools-for-PBLL-V2.pdf 

Beck, J. (2018, Sept.). 3D Printing and designing in the ESL classroom [conference presentation]. MIDTESOL 2018, Kansas City, MO, USA.

Beck, J. (2019, Mar.). Embedding STEM in the ESL classroom through 3D design and printing in Tinkercad [conference presentation]. TESOL International Convention, Atlanta, GA, USA.

Beck, J., Beckett, G. H., Bordbarjavidi, F., Garib, A., Guskaroska, A., Sollier, N., Yang, J. & Zawadzki, Z. (2021, Aug.). Maximizing technology use in project based learning: Looking back and moving forward [conference presentation]. EUROCALL 2021. 

Beckett, G. H., & Slater, T. (2017, Mar.). A synthesis of project-based language learning: Research-based teaching ideas [conference presentation]. TESOL International Convention, Seattle, WA, USA.

Beckett, G. H., & Slater, T. (Eds.) (2020). Global perspectives on project-based language learning, teaching, and assessment: Key approaches, technology tools, and frameworks. Routledge.

Beckett, G. H., Beck, J., Chen, M., & Guhin, J. (2020, Oct.). Empowerment with project-based language teaching and research [conference presentation]. TESOL Arabia 2020, Online. 

Buck Institute for Education. (2022). PBL Works. https://www.pblworks.org/ 

Chen, M., & Hirsch, R. (2020). A research-based framework for assessing technology-infused PBLL. In G. H. Beckett & T. Slater (Eds.), Global perspectives on project-based language learning, teaching, and assessment: Key approaches, technology tools, and frameworks (pp. 224-243). Routledge. 

Dewey, J. (1916). Democracy and education. Project Gutenberg. https://www.gutenberg.org/files/852/852-h/852-h.htm 

Emm, A., & Hawkins, D. (2020). From bauhaus to makerspace: Meaning design and computer-aided design in introductory German. Die Unterrichtspraxis/Teaching German, 53(2), 138-150. https://doi.org/10.1111/tger.12133 

Enabling the Future. (2022). E-NABLE. http://enablingthefuture.org/ 

Enkin, E., Tytarenko, O., & Kirschling, E. (2021). Integrating and assessing the use of “Makespace” in a Russian cultural studies course: Utilizing immersive virtual reality and 3D printing for Project-Based Learning. CALICO Journal, 38(1), 103-127. https://doi.org/10.1558/cj.40926 

Kilpatrick, W. H. (1918) The project method: The use of the purposeful act in the education process. Teachers College Record, 19(4), 319-335.

Kwon, H. (2017). Effects of 3D printing and design software on students’ overall performance. Journal of STEM Education, 18(4), 37-42. 

Lacey, G. (2010). 3D printing brings designs to life. Tech Directions, 70(2), 17-19.

Maloy, R., Trust, T., Kommers, S., Malinowski, A., & LaRoche, I. (2017). 3D modeling and printing in history/social studies classrooms: Initial lessons and insights. Contemporary Issues in Technology and Teacher Education, 17(2), 229-249.

Novak, E., & Wisdom, S. (2018). Effects of 3D printing project-based learning on preservice elementary teachers’ science attitudes, science content knowledge, and anxiety about teaching science. Journal of Science Education and Technology, 27, 412-432. https://doi.org/10.1007/s10956-018-9733-5 

Rocha, A. G. (2006). The reflective essay. In S. L. Pasqarelli (Ed.), Teaching writing genres across the curriculum (pp. 91-103). Information Age Publishing. 

Roy, D. (2017). Developing a Project-Based CALL environment with technical communication in an exploratory 3D printing context. International Journal of Computer-Assisted Language Learning and Teaching, 7(2), 75-101. https://doi.org/10.4018/IJCALLT.2017040105 

Shim, H. Y., & Lee, H. E. (2019). Multi-converging educational program for design with the usage of 3D printer: Targeted for middle school students. Archives of Design Research, 32(1), 75-87. https://doi.org/10.15187/adr.2019.02.32.1.75 

Suchow, R. (2016). 3D printing & service learning: Social manufacturing as a vehicle for developing social awareness. Journal of Catholic Education, 20(1), 292-300.

Super teacher worksheets. (2022). Metric measurement worksheets: cm and mm. https://www.superteacherworksheets.com/measure-cm-mm.html 

Trust, T., Woodruff, N., Checrallah, M., & Whalen, J. (2021). Educators’ interests, prior knowledge and questions regarding augmented reality, virtual reality and 3D printing and modeling. TechTrends, 65(4), 548-561.

Verner, I., & Merksamer, A. (2015). Digital design and 3D printing in technology teacher education. Procedia CIRP, 36, 182-186. https://doi.org/10.1016/j.procir.2015.08.041 

Wendt, S., & Wendt, J. (2015) Printing the playground: Early childhood students design a piece of playground equipment using 3-D printing technology. Science and Children, 52(5), 43-47.

 

DOWNLOAD FULL ARTICLE

Ami Christensen & Kaishan Kong

Using Flipgrid to facilitate an international virtual pen pal exchange provides learners with opportunities to gain and share cultural knowledge and experiences while engaging in authentic language tasks.
Keywords: Virtual exchange, pen pal, COVID-19, Flipgrid, cross-cultural communication, ELL

How can we connect students outside of the classroom? Pen pal exchange is not a new concept to language teachers and learners. This communicative exchange between learners reflects the essential role of interaction (Long, 1987) and sociocultural aspects (Donato, 1989; Lantolf, 1994) in learning. The development of technology in the 21st century has transformed communication between pen pals from handwritten letters to virtual exchange. International virtual pen pal exchange (IVPPE) allows learners to practice a language with native speakers, share authentic cultural experiences, and explore diverse cultural perspectives (O’Dowd, 2020; O’Rourke, 2007). This paper shares a collaborative virtual pen pal exchange project that engaged two cohorts of university students in intercultural communication on 12 topics via Flipgrid. We describe the project design, share our experiences and student feedback, and offer guidance to language educators with similar pedagogical interests.

Due to the disruption of in-person learning caused by COVID-19, the need for digital tools, such as virtual exchange, became more salient (Guillén et al., 2020). An IVPPE creates a community of learning. Thoughtfully-designed community learning has potential contributions toward students’ increased success, retention, and satisfaction (Greenfield et al., 2013). Adopting intercultural partners enables students in different countries to communicate beyond borders and to develop a feeling of social connectedness in remote settings (Bolliger & Inan, 2012; Garrison & Arbaugh, 2007). IVPPE has been adopted and researched in a wide range of teaching contexts. For instance, Catalano and Barriga (2021) examined their own experiences as teacher educators in the United States and Colombia, respectively, as they planned and facilitated a WhatsApp pen pal exchange meant to develop their teacher candidates’ intercultural competence. In an English as a Foreign Language (EFL) setting, Torres Beltrán and Santana Alarcón (2021) investigated 16 elementary school students’ writing skills development through a pen pal telecollaboration exchange. In Klimova’s (2021) study, the author shared the practice of integrating an IVPPE into online delivery in the Russian language program at the University of Pittsburgh. These studies highlighted that effective virtual exchange between students and their native-speaker partners pushed the learners and teachers out of their comfort zones and nurtured engagement in linguistic, intercultural, and technological learning experiences. Further, the exchanges compelled students to challenge their assumptions and reflect on their own identity and culture. Informed and inspired by the existing scholarship, we were dedicated to creating an interactive, impactful, and intercultural pen pal exchange for our students.

In our context as a university language department, we serve international students who are normally immersed in English language learning on our campus in the Midwestern United States. However, as a result of COVID-19, with some students confined to their home countries and some limited to activity within their dormitories, we began to provide programming virtually. It became clear that our first-year international students were facing a more complex level of uncertainty related to a new culture and different learning styles than those in the past due to COVID restrictions. It was important to offer them high-impact yet low-risk activities to support their transition. We felt the need to identify tools and (re)design lesson plans to foster community and engage students across countries and cultures (TESOL International Organization, 2022, The 6 principles section, no. 4). We created an IVPPE using Flipgrid to encourage “students’ active learning and integrative learning” in both language and culture (Lomicka, 2020, p. 307).

What is Flipgrid?

Flipgrid is an online video discussion platform that allows teachers to create a “grid” to invite students to upload short video responses to “topics” (i.e., prompts in audio, video, or text form) and reply to other students’ responses. Green and Green (2018) argue that Flipgrid supports discussion-based pedagogy and contributes to students’ critical thinking, minority students’ engagement in class, and the classroom culture. Stoszkowski (2018) examined how Flipgrid supported students’ social learning by discussing seven strengths of this app, namely (1) easy access, (2) convenience to record beyond the classroom, (3) encouraging participation, (4) video-based appeal, (5) more efficient formative feedback, (6) tracking engagement, and (7) compatibility with other platforms. Flipgrid has been increasingly adopted by language instructors over the past five years. When in-person learning was disrupted by COVID-19, Flipgrid became even more instrumental in facilitating engagement in virtual settings (Lomicka, 2020).

How did we use Flipgrid to engage students in cross-cultural communication?

What is described here was a collaborative project between two courses at our university. The first course, taught by the first author (Christensen), was part of the Intensive English Program (IEP) and was composed entirely of international students from mainland China. The course focused on developing learning strategies and study skills as well as connecting to the campus and community. The second course, taught by the second author (Kong), was an introduction to Chinese culture called Foreign Culture and Civilization (FLG). While most of the students in FLG were from the United States, there were also a few international students enrolled. Since the IEP students were devoted to improving their English proficiency, and the instructional language in the FLG course was English, the IVPPE was conducted in English.

We intended this collaboration to create an opportunity for both cohorts to meet a peer from another culture and share intercultural experiences on given topics. In addition to cultivating intercultural understanding and empathy, the IEP students also received opportunities to practice the target language, English. Students from each course were randomly assigned several partners and kept the same partners throughout the semester. Students in the FLG course initiated weekly Flipgrid conversations by sharing what they learned from course materials and asking their peers questions. IEP students responded to FLG students’ posts and, as the course progressed, asked their own questions to their FLG partners (see Figure 1). There was no limit to how many responses or comment videos students could post. FLG students wrote weekly reflections and a final reflective paper on the exchange. While this practice was mainly formative, IEP students were required to write a reflection about their experiences twice during their course. Many students also opted to discuss it in other reflections when they could choose their topics. The criteria for this reflective writing focused on the depth of self-reflection, clarity of message, and staying on-topic. During the semester, both cohorts participated in 12 discussion topics (see Table 1).

Figure 1. Sample Weekly Activity Process
Table 1. Discussion Topics

What did the students say about virtual conversations?

At the beginning of the semester, IEP students confirmed that the pandemic was limiting their experiences: “because of the COVID, I can’t go outside to feel the American environment. So, I still know very little about America” (Jane, IEP student). Prior to the IVPPE, IEP students expressed an interest in learning more about U.S. culture: “I want to know more about the extracurricular life activities of American students and some traditional customs or habits. This will allow me to understand them better, have more common topics with them” (Harlan, IEP student). Some IEP students also saw an opportunity to improve their language abilities: “I want to improve my pronunciation and vocabulary from this experience. It is a good chance to improve my English and we can talk about different things each week. I can learn many new words” (Tony, IEP student). Students also mentioned wanting to know more about Americans’ impressions of Chinese culture: “I hope to learn about the differences between American and Chinese culture, and how young people in the United States view Chinese culture” (Kai, IEP student). The interaction with U.S. students that the IEP students hoped for in their initial comments came to be realized as the semester progressed.

Although many of the subjects that students wanted to know more about were covered over the 12 weeks of our project (see Table 1), the topic about the Chinese education system prompted the longest average response from IEP students (see Figure 2). Since education was a very immediate and relatable issue for university students, students in both courses were especially curious and engaged in conversations on this topic: “For me, college life is just studying and entertainment, but I know that many American students need to do part-time jobs to pay for their tuition. They seem to attach great importance to getting an education because of their hard work” (Jane, IEP student). The second most popular topic was about Chinese cultural values. While students discussed differences between cultural values, many also expressed surprise at similarities: “When talking about values, most of my partners, like me, put filial piety first. This surprised me. I originally thought that most Americans regarded independence as very important and did not pay much attention to family; the reality is the opposite” (Harlan, IEP student).

Figure 2. Average Length of IEP Students’ Responses

By the middle of the course, the IEP students expressed a desire to continue speaking with their partners: “After chatting with them, I wanted to learn more about Americans . . . I like to watch the videos from my American partners who takes Chinese culture class. They share with me what they learned in class and some of their cultures, and I enjoy answering their questions and sharing some of my cultures with them” (Irene, IEP student). They also recognized linguistic benefits to the conversations: “One thing going well is that my listening is better than before” (Tony, IEP student).

Reflecting on the semester of exchanges between their partners, all IEP students reported that it was a positive experience for them. They mentioned how the conversations improved their listening and speaking abilities in English. “Last semester, I could not communicate a lot with my American friends because my English listening is very poor. Now, although I often get stuck on some complex words, I can have simple conversations with my American friends” (Johnny, IEP student). Students also shared additional benefits of their semester-long video conversations with FLG partners. “[This] was an invaluable opportunity to interact directly with peers in the United States . . . I have learned a lot from the American students. I believe such communication can play a great role in eliminating the barriers between cultures” (Lee, IEP student). In addition, several IEP students commented on how they appreciated their partners’ growing understanding of Chinese culture. “They have expressed a lot of their understanding of Chinese culture to me, and they have also shown me their love for Chinese culture” (Harlan, IEP student). 

The IEP students valued the exchange a great deal, as evidenced by their comments above. However, their contribution to the success of the FLG students’ positive experience should not be overlooked. All the FLG students applauded their IEP partners’ support in not only broadening their cultural knowledge, but also in building a more ethnorelative view of other cultures. Some of the repeated themes in the FLG cohort’s reflections included: first-hand experiences and knowledge about China, building international friendships, challenging their stereotypes, and growing interest in visiting China. Many students also used specific examples to support their reflections. Mike (FLG student) wrote the following reflection after a unit on gaokao, China’s National College Entrance Examination:

Watching the videos [in class] on …[China’s] education system and learning about Gaokao was interesting but I was happy to hear about their personal experience from the exam for them to tell me how it was so I could understand better. Or the stereotype that all Chinese students do is just study constantly, where that may be the situation for some it isn’t for all of them. My pen pal partners opened my eyes about that as well explaining to me their own personal hobbies and what they like to do with their friends. 

Dana (FLG student) also appreciated the first-hand knowledge gained from her pen pals. She shared, “I enjoyed getting a better insight on current day China as well as how it relates to my own generation being that my partners are the same age. It was fun getting real life examples and stories on what has happened in their own life, and I cannot wait to visit China one day.”

What are the benefits of having an international virtual pen pal?

It was apparent that using Flipgrid for this IVPPE provided abundant opportunities for the IEP students to extend language practice beyond the classroom. With a total of 565 responses, 17,861 views of videos, 1,340 comments, and 967.2 hours of engagement, there is no doubt that this project provided all students with far more opportunities for interaction than traditional face-to-face classes could provide. The IEP students were devoted to authentic intercultural communication (O’Dowd, 2020) during which they comprehended, manipulated, and produced language (Zhang & Luo, 2018). In other words, this IVPPE was able to “create conditions for language learning” (TESOL International Organization, 2022, The 6 principles section, no. 2) and public speaking in a low-risk and less stressful environment. Although Stoszkowski (2018) mentioned that one of the potential barriers to using Flipgrid is students’ concern about saying the right thing or showing their best work, we noticed that the feature of Flipgrid that allows students to record unlimited times until they are satisfied counteracted that concern. IEP students were able to share their best practice in the uploaded video, and this gradually enhanced their confidence. For English language students enrolled in online courses or studying in a place with few opportunities to speak English, utilizing Flipgrid for an IVPPE provides students with authentic language tasks and exposes them to language diversity.

An additional benefit of Flipgrid is that it automatically tracks the total number of responses, views, comments, and engagement on a topic. Data can be easily downloaded, and instructors can invite students to join using links or QR codes. Instructors can easily add a co-lead (partner instructor) who can edit groups/topics and approve responses and comments. These “tracking engagement” and “more efficient formative feedback” (Stoszkowski, 2018) features of Flipgrid made it easy for us to “monitor and assess student language development” (TESOL International Organization, 2022, The 6 principles section, no. 5). The two instructors in this project communicated on a regular basis to exchange observations of students’ participation, which was a contributing factor to the effectiveness of this project.

Additionally, this project allowed the IEP students to gain cultural knowledge through their peers, who were also learning from them. The feeling that they were not only learners but also experts in some areas helped them feel engaged within a cross-cultural community. At the beginning of the project, many IEP students shared their desires to learn more about U.S. culture, such as holiday celebrations and extracurricular life activities. By the end of the project, they cited many specific examples in their reflective writing to show their learning. Some noted that their views of U.S. culture became more complex and nuanced, evidence of challenging their assumptions (Guillén et al., 2020). They also felt the pride and joy of knowing their cross-cultural partners’ appreciation of Chinese culture. Similar results were discovered in Jin’s (2018) study about reflection on one’s own identity and culture through cross-cultural virtual exchange. This mutual learning opportunity was impactful and valuable, especially during difficult times when the pandemic and tension in the U.S.-China relationship caused increased anxiety and uncertainty.

What should I think about before starting an IVPPE?

In this paper, we described a collaborative project to connect two cohorts in an IVPPE using Flipgrid. Throughout the semester, both cohorts initiated and replied to videos on 12 topics. The statistics recorded in Flipgrid and students’ reflections presented linguistic and cultural benefits of virtual exchange. We had a positive experience using IVPPE, and it is our hope that this research-informed project will be adopted by colleagues with similar interests.

For those considering starting an IVPPE with Flipgrid, we offer the following reminders. First, participating in a virtual pen pal project is an engaging way to connect students from diverse classrooms and let them learn from each other. Speaking in front of a class as an individual can be intimidating, especially for shy students or English language learners. Recording a video allows students to think through what they want to say and present their thoughts and ideas without the added pressure of a live audience. English language learners also benefit from viewing videos multiple times to practice listening skills.

Second, to make virtual pen pal exchange effective, it is important for instructors to be thoughtful in creating relatable topics for students and facilitating an online community that allows the students and the instructor “to bond, develop a sense of trust, feel comfortable sharing information,” and to maintain “a sense of connectedness” (Lomicka, 2020, p. 306). It is worth mentioning that one of the drawbacks to using Flipgrid is that it is set up for students to access using Microsoft or Google accounts. If students do not have these accounts, an instructor must generate individual usernames and share them with students. It is important for instructors to provide the students with clear instructions on how to access and respond to topics. Instructors may also want to suggest minimum lengths for videos, especially for the first few exchanges. An additional challenge associated with this virtual pen pal project was the need for instructors to check and make sure students completed their videos. This was relatively easy to do using the search feature in Flipgrid.

Finally, we encourage instructors from any course to utilize Flipgrid intentionally and thoughtfully as a tool to increase engagement and connect their students with a wider community. Using Flipgrid to facilitate an international pen pal exchange provides learners with opportunities to gain and share cultural knowledge and experiences while participating in authentic language tasks.

References

Bolliger, D. U., & Inan, F. A. (2012). Development and validation of the online student connectedness survey (OSCS). International Review of Research in Open and Distance Learning, 13(3), 41-65. https://doi.org/10.19173/irrodl.v13i3.1171 

Catalano, T., & Muñoz Barriga, A. (2021). Shaping the teaching and learning of intercultural communication through virtual mobility. Intercultural Communication Education, 4(1), 75-89. https://doi.org/10.29140/ice.v4n1.443 

Donato, R. (1989). Beyond group: A psycholinguistic rationale for collective activity in second language learning. Dissertation Abstracts International, 49(12), 3701-A. 

Garrison, D. R., & Arbaugh, J. B. (2007). Researching the community of inquiry framework: Review, issues, and future directions. The Internet and Higher Learning, 10(3), 157-172. https://doi.org/10.1016/j.iheduc.2007.04.001 

Green, T., & Green, J. (2018). Flipgrid: Adding voice and video to online discussions. TechTrends, 62(1), 128-130. https://doi.org/10.1007/s11528-017-0241-x 

Greenfield, G. M., Keup, J. R., & Gardner, J. N. (2013). New student orientation: Developing and sustaining successful first-year programs. Jossey-Bass.

Guillén, G., Sawin, T., & Avineri, N. (2020). Zooming out of the crisis: Language and human collaboration. Foreign Language Annals, 53(2), 320-328. https://doi.org/10.1111/flan.12459  

Jin, L. (2018). Digital affordances on WeChat: Learning Chinese as a second language. Computer Assisted Language Learning, 31(1-2), 27-52. https://doi.org/10.1080/09588221.2017.1376687 

Klimova, O. (2021). From blended learning to emergency remote and online teaching: Successes, challenges, and prospects of a Russian language program before and during the pandemic. Russian Language Journal, 71(2), 73-85. https://scholarsarchive.byu.edu/rlj/vol71/iss2/5  

Lantolf, J. P. (1994). Sociocultural theory and second language learning: Introduction to the special issue. The Modern Language Journal, 78(4), 418-420. https://doi.org/10.1111/j.1540-4781.1994.tb02058.x 

Lomicka, L. (2020). Creating and sustaining virtual language communities. Foreign Language Annals, 53(2), 306-313. https://doi.org/10.1111/flan.12456 

Long, M. H. (1987). The experimental classroom. Annals of the American Academy of Political and Social Science, 490, 97-109. 

O’Dowd, R. (2020). A transnational model of virtual exchange for global citizenship education. Language Teaching, 53(4), 477-490. https://doi.org/10.1017/S0261444819000077 

O’Rourke, B. (2007). Models of telecollaboration 1: eTandem. In R. O’Dowd (Ed.), Online intercultural exchange (pp. 41–61). Multilingual Matters.

Stoszkowski, J. (2018). Using Flipgrid to develop social learning. Compass: Journal of Learning and Teaching, 11(2). https://doi.org/10.21100/compass.v11i2.786 

TESOL International Organization. (2022). The 6 principles for exemplary teaching of English learners. https://www.tesol.org/the-6-principles 

Torres Beltrán, J. A., & Santana Alarcón, J. L. (2021). Developing young learners’ EFL writing skills through a pen pal telecollaboration exchange during the COVID-19 Pandemic in two rural institutions in Boyacá. Universidad Pedagógica y Tecnológica de Colombia, Tunja. http://repositorio.uptc.edu.co/handle/001/3814 

Zhang, Y., & Luo, S. (2018). Teachers’ beliefs and practices of task-based language teaching in Chinese as a second language classrooms. Chinese Journal of Applied Linguistics, 41(3). 264-287. https://doi.org/10.1515/cjal-2018-0022 

 

DOWNLOAD FULL ARTICLE

Alayna Klco

Translanguaging pedagogy has largely focused on K-12 learners in content classrooms. However, collaborative translation is one translanguaging literacy practice that is well-suited for adult English as a Second Language (ESL) learners with thoughtful adaptations based on text choice, course objectives, and learner characteristics.
Keywords: Adult Education, Higher Education, SLIFE, Translanguaging, Language Teaching, Multilingualism

Translanguaging has emerged as an increasingly popular framework in English language learner education (García & Li Wei, 2014; Lin & He, 2017). At its strongest variation, translanguaging as an ideology describes a unitary languaging system and calls for learning environments that enable multilingual learners to use their entire linguistic repertoire (García & Li Wei, 2014). To achieve this, several pedagogical interventions such as collaborative dialogue, multilingual writing and word walls, translation, and multilingual listening/visual resources have been suggested for incorporating translanguaging into the classroom (García & Li Wei, 2014). Thus far, translanguaging has been studied most extensively in K-12 content classrooms. However, the benefits of translanguaging are not limited to this setting, and research suggests that adult learners can also benefit substantially from a translanguaging approach (Turnbull, 2019).

Translation is one of the suggested pedagogies for incorporating translanguaging in the classroom. In an effort to shed light on how this translanguaging strategy can be adapted for adult education contexts, this paper will focus specifically on pedagogical translation and the work of Project TRANSLATE (Teaching Reading And New Strategic Language Approaches to Emergent bilinguals, see Figure 1). All data from Project TRANSLATE described in the adaptations below is based on observations and findings from ongoing research conducted in a middle grade classroom in the Midwest. This work has been completed by a team of researchers that includes the author, who has also been an English as a Second Language (ESL) educator for adult language learners for five years.

Figure 1. Project TRANSLATE

Project TRANSLATE uses collaborative translation between the target language and students’ home languages as a literacy intervention. To date, this translanguaging pedagogy has been used only in K-12 content-based settings. However, with some adaptation, the TRANSLATE protocol is rich for use in adult English language learning classrooms.

Translanguaging – One Word, Two Meanings

As an ideology, translanguaging views language as a unitary meaning-making system (García & Li Wei, 2014). Translanguaging intentionally enables multilingual speakers to leverage their entire linguistic and multi-modal repertoires in communication with and about people, objects, places, and spaces (García & Kleifgen, 2019). The goals of translanguaging are to deepen students’ understanding of texts, generate more diverse texts, develop students’ confidence, and foster metalinguistic awareness. García et al. (2021) situate translanguaging as part of asset-based pedagogy and firmly reject the abyssal thinking that places mainstream white English at the top of the language hierarchy. Translanguaging in the classroom is therefore a matter of both educational efficacy and social justice equity (Shohamy, 2011).

As a pedagogy, translanguaging suggests that teachers should leverage students’ full linguistic repertoires when teaching a new language (David et al., 2019) and that multilingual practices should be regularly implemented in the classroom (García et al., 2021). Lin & He (2017) put it even more clearly: “Translanguaging can function as both pedagogical scaffolding strategies and opportunities to negotiate and affirm students’ identities and build teacher-student rapport” (p. 237). However, while many teachers may share a belief in the benefits of students using their full linguistic repertoires, the reality of how that is expressed in the classroom varies greatly.

Oaxaca, Mexico

Some teachers used translanguaging as a scaffold (for example, providing instructions in students’ first language [L1]) whereas others explicitly rewarded the use of translanguaging (for example, in narrative projects) when asked to integrate translanguaging into assessments. 

Studied by Schissel et al., 2021

Basque Country, Spain

University-level content teachers fell into one of three categories based on their use of the target language in the classroom:

  • virtual position (target language only)
  • maximal position (as much target language as possible with some flexibility)
  • optimal position (judicious use of students’ L1 to enhance learning). 

Teachers’ use of translanguaging was typically reflected in their materials, though not necessarily in their assessments. 

Studied by Doiz and Lasagbaster, 2017

Project TRANSLATE

Project TRANSLATE is a collaborative translation strategy for literacy development that uses pedagogical translation as a social practice in which multilingual students’ daily translanguaging practices are mirrored in a classroom setting (David et al. 2019). The TRANSLATE protocol is situated within a guided reading framework (Fountas & Pinnell, 1996), and pedagogical translations are designed to leverage multilingual students’ skills to improve strategic reading, translation, and engagement. This, in turn, develops their English language proficiency as well. The TRANSLATE protocol as detailed by David et al. (2019) is grounded in social practice theory and consists of seven primary steps (see Figure 2).

Figure 2. Project TRANSLATE Protocol

Adapting Pedagogical Translation Across Contexts

What’s different between K-12 and adult ESL when it comes to pedagogical translation?

To date, the TRANSLATE protocol has been exclusively studied in K-12 classrooms. To bring this translanguaging practice to adult ESL education requires several adaptations. Adult ESL education and K-12 content classrooms use fundamentally different input texts in addition to having different course objectives and learner characteristics. 

Different Standards, Different Texts

K-12 objectives for English language learners are typically bound to WIDA standards or other federally or state-mandated standards (Gottlieb, 2016). While teachers may have considerable freedom in their lesson planning, there is some consistency in standards across schools, districts, and states. There may be more variation for adult English learners. English curriculum taught as part of an Adult Basic Education program may be connected to a variety of federal standards such as the National Reporting Service Educational Functioning Levels, English Language Proficiency Standards, College and Career Readiness Standards, or may follow state-specific standards (Gonzalves, 2021). On the other hand, adults learning in an intensive English program that is affiliated with a specific school or institution may adhere to standards established by that program. Finally, English language programs established through non-profit organizations or other community education settings may have little to no formal standards process (Entigar, 2017). Adults in each of these learning environments are likely to have different goals and reasons for learning English – job acquisition, preparing for higher education, or survival English for newcomers to name a few. Using TRANSLATE with adult learners starts with a text that is relevant and engaging for their unique needs.

Adaptations: A good text for collaborative translation is relevant to students, tied to learning objectives, and not directly translatable. In an adult education context, relevancy is driven by the goals of the students. For example, if learners are highly job-focused, consider using authentic materials from a work report or dialogue between colleagues. However, fact-based writing tends to be more directly translatable and leaves fewer opportunities for meaning negotiation, so materials with work-related vocabulary as well as some figurative language and complex structures are ideal. On the other hand, if you are working with adults who are learning English in anticipation of shortly entering an English-speaking college or university, a rich text could address a theme central to existing course materials (including anything from civics to signing a lease!), cultural adaptation, or students’ future studies.

Content Class vs. ESL

Many K-12 English language learners are acquiring English in tandem with grade-specific content matter. Translanguaging in the classroom fosters critical opportunities for multilingual students to express their full content knowledge, regardless of English proficiency. In contrast, adult learners are more likely to be in an ESL class where acquisition of the target language is the primary objective, though this language learning is still often embedded in thematic, content-driven units. In these classes, the structure of adult ESL classes in many ways more closely resembles a K-12 foreign language classroom. This raises the question of how much translanguaging should be used in the classroom. 

In a classroom where English learning is the primary objective, strong translanguaging is no less of an effective resource when applied strategically (Turnbull, 2019). In a study of Japanese young adults learning English as a Foreign Language (EFL) at a national university, Turnbull found that strong translanguaging, with strategic boundaries, enabled the intentional acquisition of specific linguistic features. Students in the strong translanguaging group produced richer student discussions and consistently higher scores for academic and creative writing than those in the weak translanguaging group where students were required to conduct research only in the target language. Strong translanguaging environments also produced richer student discussions, even when the final writing was produced in English (Turnbull, 2019). This study suggests that translanguaging pedagogies such as collaborative translation have significant potential in ESL and EFL classrooms.

Adaptations: As in K-12 classrooms, enabling the use of students’ full linguistic repertoires is key in adult English classes, but ESL classes strongly prioritize language learning in addition to thematic content. To emphasize linguistic connections in the TRANSLATE protocol, teachers can prompt discussion about specific language features, such as negation syntax or object placement, when comparing students’ home languages and the target language in addition to negotiating meaning. If teaching in an online learning environment, collaborative translation could take place using editable documents where students can write and compare translations. Then, meaning negotiation could occur either synchronously through a video call or asynchronously by leaving comments. 

Similarly, instructors can ask students to highlight targeted language features in their home language (e.g., modal verbs) and English to identify and analyze notably similar or different structures between languages. In both in-person and online environments, classes with adults whose English course is also their primary source of target language input should place increased emphasis on reconnecting students with the text while using the target language.

Shifting from K-12 to Adult Education

The demographics of adult English learners may vary dramatically in a single program or even within a classroom. Some adult learners may come in with primarily oral proficiency, emergent literacy in their home language, or interrupted education (Pettitt et al., 2021). As a result, while K-12 schools frequently emphasize literacy-based instruction and learning, that is not necessarily effective for all adult English learners (Pettitt et al., 2021). Instead, adult learners with emergent literacy may perform better when instruction is based heavily on oral proficiency and gradually incorporates print literacy (Tammelin-Laine & Martin, 2014). Teaching that builds on learner strengths and the funds of knowledge that all students bring to the classroom also helps develop a learner-centered education environment (Parrish, 2019).

Adaptations: If working with adult learners with emergent literacy, additional modifications to the TRANSLATE protocol will be necessary. For example, the selected text and target sentences for translation must be available in an oral format. Written translations are valuable for meaning negotiation because they afford opportunities to visually compare translation syntax and consider orthographic patterns as part of meaning negotiation. However, an adapted process could rely on teacher collaboration or technology to create the translation record. For example, in our ongoing study, one teacher used a student dictation approach with Nepali-speaking students who were uncomfortable writing in their home language. To complete the translation and translanguage in oral and written modes, students read a text in English, created a translation based on Google Translate and their knowledge of Nepali, and their teacher wrote their verbal, Nepali, translations on paper phonetically using the English script. 

Video technology similarly creates a translation record that allows students to review and compare their recorded translations as many times as necessary while implicitly increasing students’ technology literacy in the process (Vega-Carrero et al., 2017). Going forward, speech-to-text technology may also provide a scaffold for language learners with emerging print literacy. Additional scaffolding could be used for students with emerging literacy in their home language by reading the text aloud and conducting the entire activity orally. These adaptations can be used as intermediary steps to written translation if print literacy is a core objective. 

The Takeaway

Facilitating translanguaging pedagogy is a key step to supporting equitable cultural and linguistic development for multilingual learners (García & Kleifgen, 2019; García & Li Wei, 2014; Shohamy, 2011). One strategy to try is pedagogical translation, which helps learners develop strong literacy practices and can be well suited for adult learners as well as K-12 contexts. You know your students best, so consider carefully what will be a meaningful text that meets their interests and your curricular objectives. Then, look for opportunities to negotiate meaning and highlight key language differences to spark metalinguistic conversations as students build multilingual connections in this dynamic classroom pedagogy. 

Additional Reading:

References

David, S., Pacheco, M., & Jiménez, M. (2019). Designing translingual pedagogies: Exploring pedagogical translation through a classroom teaching experiment. Cognition and Instruction, 37(2), 252-275. https://doi.org/10.1080/07370008.2019.1580283 

Doiz, A., & Lasagbaster, D. (2016). Teachers’ beliefs about translanguaging practices. In C. M. Mazak & K. S. Carroll (Eds.), Translanguaging in higher education: Beyond monolingual ideologies (pp. 157-176). Multilingual Matters. https://doi.org/10.21832/9781783096657-011 

Entigar, K. E. (2017). The limits of pedagogy: Diaculturalist pedagogy as paradigm shift in the education of adult immigrants. Pedagogy, Culture & Society, 25(3), 347-356. https://doi.org/10.1080/14681366.2016.1263678 

Fountas, I. C., & Pinnell, G. S. (1996). Guided reading: Good first teaching for all children. Heinemann Press.

García, O., Flores, N., Seltzer, K., Li Wei, Otheguy, R., & Rosa, J. (2021). Rejecting abyssal thinking in the language and education of racialized bilinguals: A manifesto. Critical Inquiry in Language Studies, 18(3), 203-228. https://doi.org/10.1080/15427587.2021.1935957 

García, O., & Li Wei. (2014). Translanguaging: Language, bilingualism, and education. Palgrave Macmillan.

García, O., & Kleifgen, J. A. (2019). Translanguaging and literacies. Reading Research Quarterly, 55(4), 553-571. https://doi.org/10.1002/rrq.286 

Gonzalves, L. (2021). A system of erasure: State and federal education policies surrounding adult L2 learners with emergent literacy in california. In D. S. Warriner (Ed.), Refugee education across the lifespan (pp. 271-288). Springer.

Gottlieb, M. (2016). Assessing English language learners: Bridges to educational equity (2nd ed.). Corwin.

Lin, A. M. Y., & He, P. (2017). Translanguaging as dynamic activity flows in CLIL classrooms. Journal of Language, Identity & Education, 16(4), 228-244. https://doi.org/10.1080/15348458.2017.1328283 

Parrish, B. (2019). Teaching adult English language learners: A practical introduction (2nd ed.). Cambridge University Press.

Pettitt, N., Gonzalves, L., Tarone, E., & Wall, T. (2021). Adult L2 writers with emergent literacy: Writing development and pedagogical considerations. Journal of Second Language Writing, 51 (Supplement 2021), 1-5. https://doi.org/10.1016/j.jslw.2021.100800 

Schissel, J. L., De Korne, H., & López-Gopar, M. (2021). Grappling with translanguaging for teaching and assessment in culturally and linguistically diverse contexts: Teacher perspectives from Oaxaca, Mexico. International Journal of Bilingual Education and Bilingualism, 24(3), 340-356. https://doi.org/10.1080/13670050.2018.1463965 

Shohamy, E. (2011). Assessing multilingual competencies: Adopting construct valid assessment policies. The Modern Language Journal, 95(3), 418-429. https://doi.org/10.1111/j.1540-4781.2011.01210.x 

Tammelin-Laine, T. & Martin, M. (2014). The simultaneous development of receptive skills in an orthographically transparent second language. Writing Systems Research, 7(1), 39-57. https://doi.org/10.1080/17586801.2014.943148 

Turnbull, B. (2019). Translanguaging in the planning of academic and creative writing: A case of adult Japanese EFL learners. Bilingual Research Journal, 42(2), 232-251. https://doi.org/10.1080/15235882.2019.1589603 

Vega-Carrero, S., Pulido, M., & Ruiz-Gallego, N. E. (2017). Teaching English as a second language at a university in Colombia that uses virtual environments: A case study. Revista Electrónica Educare, 21(3), 1-21. https://doi.org/10.15359/ree.21-3.9 

 

DOWNLOAD FULL ARTICLE

This article describes a procedure for training second-language writing raters to use scoring rubrics, and presents ideas for practical adaptation or research projects associated with the training procedure.

Imagine a novice language teacher doing the following:

  1. understanding exactly what is meant by rubric descriptors such as some knowledge of subject, adequate range, limited development of thesis, mostly relevant to topic but lacks detail;
  2. looking for features in a second language writer’s composition that appear to match these descriptors, and then, upon finding a match; and
  3. rating the content of the composition as Good.  

These descriptors are examples from a commonly used rubric for assessing second language writing (Jacobs et al., 1981), in which the example descriptors above represent only 5% of all the descriptors that a rater is supposed to be familiar with while reading and rating compositions.  

In programs in which graduate teaching assistants (TAs) need to be trained over a short period of time to rate second language writers’ essays and make course placement decisions, rater training must ensure that the novice raters quickly become familiar with the descriptors used in a scoring rubric.  Regardless of the specific rubrics used, raters are typically expected to demonstrate knowledge of pre-determined descriptors (such as some knowledge of subject, adequate range, limited development of thesis, mostly relevant to topic but lacks detail) and perform the task of applying this knowledge consistently.  Such expectations may not be met easily.  Joe, Harmes, and Hickerson (2011) show, for example, that lack of transparency in rating scale descriptors can be a factor influencing raters’ performance.  At the same time, rater-related factors can also add to the challenge in achieving reliable outcomes in rating (see Barkaoui, 2011 for a comprehensive review).  

In this article, first, I will briefly review studies on raters and rater training.  Then, I will describe a gap in published studies on rater training.  Finally, I will introduce a rater training procedure currently implemented at a four-year university in the U.S. Midwest, for two purposes:

  1. to encourage language program administrators or language teacher educators to adopt and field-test the procedure in their own programs, and
  2. to open up the possibility of empirically testing the procedure using systematically designed research methods.

Are experienced raters consistently better than novice raters?

Among several rater-related factors that could influence raters’ interpretation and application of rating scales, raters’ experience (novice vs. experienced) seems to be the most frequently researched factor (Barkaoui, 2011; Greer, 2013; Hamp-Lyons, 1989; Harsch & Martin, 2012; Joe et al., 2011; Weigle, 1994).  Research findings on this issue are mixed, contrary to what might be assumed (i.e. the more experienced, the more proficient).  In fact, the mixed findings might be a function of the complexity in the way raters’ experience interacts with non-rater factors.  In Barkaoui (2011), for example, with regard to severity in rating, novice raters and experienced raters behaved more similarly when using analytic scales than they did when using holistic scales.  In other words, raters’ experience impacts ratings differently depending on the type of rating scales used.

The intriguing nature of raters’ experience as a factor was documented with great detail in Joe et al. (2011), which explored rater cognition based on data collected through verbal protocols.  In this study, eight faculty experts (experienced raters) and eight undergraduate students (inexperienced raters) participated in rating oral speech performances.  Both groups were trained to use an analytic scoring rubric, which included 39 features comprising ten competency dimensions relevant to the construct of the speech performances to be evaluated.  The study found that inexperienced raters started out paying attention to rubric features more consistently than did experienced raters, who were found to pay attention to construct-irrelevant features (i.e. features not listed in the rubric) at a higher rate than inexperienced raters did.  Over time, however, inexperienced raters attended to rubric features less and less while attending to construct-irrelevant features (such as the use of note cards or memorable thesis statement, which were not included in the rubric) more and more.  

Such findings are quite alarming in that not all changes exhibited by raters as they become more experienced seem to be in the expected direction – i.e. experience is generally expected to be a positive factor.  Some researchers even suggest that raters should not be selected based on teaching experience as it is not a significant factor (Royal-Dawson & Baird, 2009).  Although research findings about rater experience might be mixed, many researchers have emphasized the importance of rater training in enhancing the quality of raters’ performance (Greer, 2013; Lovorn & Rezaei, 2011; Weigle, 1994).  The following section will review studies on rater training.

What do we know from studies on rater training?

One of the aims in most rater training involves monitoring rater behavior associated with rater-related factors such as experience, rating style, or rating preferences, and then providing feedback accordingly to achieve the ultimate goal of increasing inter-rater reliability (i.e. different raters performing similarly to one another).  On the one hand, studies such as Pufpaff, Clarke, and Jones (2015) and Weigle (1994) reported that rater training did not improve inter-rater reliability.  On the other hand, Weigle emphasized the point that “rater training cannot make raters into duplicates of each other, but it can make raters more self-consistent” (p. 32).  This statement, then, naturally leads to the question as to what factors might contribute to developing rater self-consistency.  Although there has not been much research that directly explored this question, several studies have reported the positive effects of rater training on rater performance in different aspects of the rating task.

In Greer (2013), novice raters practiced assessing ESL compositions following a training workbook, which included experienced raters’ feedback on the same compositions that the novice raters were evaluating.  After the training, the novice raters reported increased confidence in their rating performance.  In another study based on a two-month rater training program (Harsch & Martin, 2012), 13 novice raters completed rigorous weekly assignments consisting of tasks commonly included in rater training such as individual practice and group discussion, using over 1700 writing samples (whittled down from an initial set of over 6000 samples).  Although the scope of the study is truly impressive, it is the depth of its rater training that makes it rather unique and remarkable.  As a part of their weekly assignments, for example, the novice raters were actively engaged in revising the wordings on the rating scale.  In fact, researchers recommend engaging raters in the development of rating scales (Barkaoui, 2010; Stevens & Levi, 2005).  Harsch and Martin (2012) concluded that rater agreement increased when a revised rating scale (i.e. revised based on the novice raters’ discussion and input during the training period) was used.

Because the task of rating is replete with a myriad of interacting factors that could influence the process and outcome of rating, research-guided rater training may be essential in most contexts.  For example, when training raters, feedback should be provided immediately after rating has occurred (Knoch, 2011).  Rater training should promote detailed and analytical understanding of the scoring rubric (Lovorn & Rezaei, 2011; Rezaei & Lovorn, 2010).  An eye-movement study on raters’ use of a scoring rubric showed that even the physical layout of the rubric can affect raters’ attention to each category on the rubric (Winke & Lim, 2015).  These are just a few examples of published studies that could guide the design of a rater training.

So what is the gap in published studies on rater training?

In most published studies that either directly or indirectly report the outcome of rater training  (Barkaoui, 2011; Joe et al., 2011; Knock, 2011; Lovorn & Rezaei, 2011; Pufpaff et al., 2015; Weigle, 1994), it appears common to follow variations of the same approach, categorically speaking, in the way raters (regardless whether novice or experienced) are initially introduced to a rating scale (either holistic or analytic).  First of all, surprisingly, many studies (Barkaoui, 2011; Knock, 2011; Lovorn & Rezaei, 2011) do not provide sufficient detail regarding exactly how raters are introduced to the rating scales selected in their respective studies.  Of the studies (Joe et al., 2011; Pufpaff et al, 2015; Weigle, 1994) that do provide some limited information regarding this part of rater training, the common approach seems to be Present and Clarify/Explain with respect to the descriptors on the rating scale.  

This clearly is an example of what is defined as the hierarchical approach: “passing onto raters a predetermined view on how they are to interpret the scale wordings, using pre-assessed scripts (so called ‘master codes’) which are not to be discussed but to be accepted and internalised” (Martin & Harsch, 2012, p. 233).  On the one hand, researchers have repeatedly identified the difficulty that raters experience when trying to understand rating scale descriptors (Barkaoui, 2010; Greer, 2013; Hamp-Lyons, 1989; Harsch & Martin, 2012; Joe et al., 2011).  On the other hand, as described above, variations of a very top-down approach seem to prevail when it comes to training raters to become familiar with the wordings on the rating scale.

From this perspective, Harsch and Martin’s (2012) study, as reviewed in the preceding section, may be considered as an exception in that the raters in their study were engaged in a series of in-depth tasks attending to, analyzing, and revising the descriptors on the scale.  These researchers also emphasize the importance of “reaching consensus about how to interpret scripts with reference to scale descriptors” (p. 233).  As mentioned earlier, the rater training in their study spanned over a two-month period.  Most readers would agree that a rater training program like that, while both impressive and exemplary, is anything but feasible in most real contexts.  The reality of most rater training is likely to resemble the two-hour norming session referred to as a typical rater calibration procedure (Weigle, 1994, pp. 7-8).

Here is a question, then, that a concerned program administrator or teacher educator might ask: Do people use the top-down approach because they are constrained to the typical two-hour calibration procedure (or however many hours it might take but not the luxury of two months)?  Acknowledging the “time- and resource-intensive” nature of their approach, Harsch and Martin (2012, p. 244) recommend realistic adaptations using existing rating scale descriptors (i.e. not necessarily attempting to revise the descriptors as their raters did).  So it appears that, although there is a huge gap between a deeply engaging, albeit extremely unfeasible, approach to rater training and a more commonly practiced top-down approach, careful retooling of the top-down approach can help fill this gap.  In the next section, I will introduce an authentic example of a rater training procedure which is characterized as a rater-centered bottom-up approach.  (Readers can rest assured that this procedure will not require two months to try!)

What does a rater-centered bottom-up rater training procedure look like?

Bottom-up Approach

The rater training procedure described here, as an example of a rater-centered bottom-up approach, has been implemented in an authentic test context.  The following provides some background information about the context:

Location Four-year university in the U.S. Midwest
Purpose Placement decisions for writing courses in English for Academic Purposes
Raters Graduate teaching assistants in an MA-TESL program (1st~4th semester)
Target Texts Academic essays written by second language writers
Scoring Rubric Locally revised version of Composition Profile by Jacobs et al. (1981)

 

Table 1.  Training Protocol

Training Protocol (Rater-centered Bottom-up Approach) 

[I] Individually    [SG] Small Groups (3~4)     [WG] Whole Group     [T] Trainer     M: Materials

Description of Steps in the Training Protocol Rationale for Each Step
1.  Activating existing knowledge & expectations about academic writing
[SG]  Brainstorm & consolidate existing knowledge & expectations about academic writing
[WG]  Discuss and summarize
M:  Brainstorm sheets
Step 1 allows each rater to activate existing knowledge; allows each rater to generate his/her own language to describe features of writing; compile & share entire group’s ideas.
2.  Evaluating a writing sample based on existing knowledge – without any rubric
[I]  Read Essay #1; Write any/all notable features, good & bad, one feature per sticky-note; Place sticky notes in worksheet; Give a holistic score
[SG]  Compare notes placed in individual worksheets; Compare holistic scores
M:  Essay #1; Sticky-notes; Worksheet
Step 2 allows each rater to apply existing knowledge; allows each rater to notice features in the writing with no constraints; exposes raters to writing features noticed by others.
3.  Familiarization with rating scale descriptors
[I]  Read rating scale descriptors and criteria
[SG]  Discuss & help each other understand concepts & terminologies
[WG]  Review & clarify concepts & terminologies
M:  Rating scale descriptors & criteria handout; handout on Content-to-Form continuum in writing
Step 3 introduces descriptor language to raters; helps raters to conceptually align their own language with descriptor language; helps identify & clarify gaps between rater-generated language and descriptor language.
4.  Matching current knowledge with rating scale descriptors
[SG]  Discuss each note on sticky-notes; Transfer & match each sticky-note with descriptors in the Descriptor Handout
[WG]  Discuss & further clarify descriptors based on questions from SGs
M: Descriptor Handout (one copy for each SG)
Step 4 allows raters to map their own unconstrained observations onto descriptors; helps identify, discuss, & resolve writing features that are difficult to map onto descriptors; more importantly, helps raters understand descriptors with self-generated concrete examples.
5.  Practice using rating scale descriptors without scores
[I]  Read Essay #2; Use the Descriptor Handout to mark relevant descriptors
[SG]  Compare individuals’ markings on descriptor handout
M:  Essay #2; Descriptor Handout
Step 5 allows raters to practice using the descriptors directly without scaffolding (i.e. no self-generated descriptive notes as with Essay #1); allows another chance to focus on the descriptors with no burden to score the essay numerically.
6.  Familiarization with the complete version of rating scale (with score indicators)
[T]  Introduce the complete version of the rating scale (with score indicators) and explain
[I]  Based on markings on Descriptor Handout (from Step 5) and using the rating scale (with score indicators), numerically score Essay #2
[SG]  Compare scores for Essay #2
M:  Complete version of rating scale
Step 6 finally exposes raters to the actual rating scale with score indicators; helps raters perform the task of numerical scoring (not exactly the same as mapping observed writing features onto descriptors); helps deal with two different subtasks (i.e. identifying matching descriptors vs. numerical scoring) with more clarity.
7.  Practice using the complete version of rating scale with a familiar essay
[I]  RE-read & score Essay #1 using rating scale
[WG]  Discuss the results & rationale of rating scale
M:  Complete version of rating scale with score indicators
Step 7 allows raters to apply the rating scale in evaluating a familiar writing sample; provides raters with an opportunity to review & re-assess their own initial evaluation of Essay #1 (performed prior to the introduction of the rating scale).
8.  Remaining steps in the protocol
There are a few more steps in the protocol, which are beyond the focus of this article.  Some of the remaining steps are similar to commonly practiced norming procedure, and some steps are specific to the local test context.1

 

As Table 1 shows, the training protocol follows a rater-centered bottom-up procedure, which affords the raters step-by-step scaffolding to develop an understanding of and the ability to apply the descriptors on the rating scale.  The procedure promotes activating existing knowledge and acquiring new knowledge of technical concepts/terminologies through a sequence of small tasks rather than through top-down imposition of abstract descriptors onto the raters.  For many novice raters, learning to use a rating scale with pre-determined descriptors includes an element of language acquisition.  It is not a mere coincidence that, in many ways, the procedure introduced here resembles language learning activities based on the task-based language teaching (TBLT) approach, in which language acquisition occurs as a natural part of successful completion of communicative tasks (Van den Branden, 2006).

The rater training procedure introduced here also provides scaffolding for one of the subtasks of rating that present a unique challenge for most raters, namely translating descriptors into numerical scores.  Studies have shown that both novice raters (Greer, 2013) and experienced raters (Hamp-Lyons, 1989) find this subtask very difficult.  In the procedure described in Table 1, raters are assisted to deal with this challenge in two ways: (1) initial steps in the procedure focus on the descriptors without the ‘burden’ of matching them with numerical scores; and (2) the complete version of the rating scale, a locally revised version of the composition profile by Jacobs et al. (1981), presents numerical scores in subsets to match raters’ judgments based on descriptors.

Unlike the two-month rater training described in Harsch and Martin (2012), which is quite impressive and ambitious, the training procedure introduced in this article is bundled with realistic and practical advantages:

  1. the procedure can be easily adopted and adapted in most rater training contexts;
  2. the sequence will work well with any type of rating scales;
  3. it only takes 2~4 hours to complete the entire protocol; and finally and more importantly,
  4. it does not cost much (i.e. if you can afford lots of sticky-notes). 

The last two items directly address “the time- and resource-intensive” challenge of Harsch and Martin’s (2012, p. 244) otherwise exemplary rater training model.  Hopefully, these practical advantages would encourage many readers of this article to consider employing this training procedure.

Aside from the obvious practical advantages, the most critical advantage of this procedure, at least based on informal observations during several semesters of implementation, involves the change in dynamics and roles between the trainer and the raters in training.  As the protocol shows, at each step, raters are actively engaged in small doable tasks either independently or in collaboration with peer raters.  Because the steps are sequenced to promote learning-by-doing, the procedure does not require much top-down talk from the trainer.  

When this new procedure was first implemented a few semesters ago, raters who had experienced the previous format resembling the Present and Clarify/Explain approach enthusiastically commented that the new procedure felt stress-free, engaging, and helpful.  As the trainer in this incidence, I too noticed unexpected changes when first implementing the new procedure. It felt as though I did not have to do anything during the procedure because the raters were doing all the work for themselves!

What more can be done with this training procedure?

Although the rater-centered bottom-up training procedure is strongly recommended, the caveat is that it has not been empirically tested.  First, interested readers are encouraged to consider field-testing this procedure in their various test contexts.  It can be modified to fit the needs and capabilities of each context.  One example might be replacing the use of sticky-notes with a digital/online tool to help raters generate, compile, and compare the features they observe in the writing sample they evaluate.  This is actually an attractive idea, which can lead to the next point of this discussion, namely research possibilities.

The hands-on aspect of using sticky-notes is actually a very positive and valuable element of the procedure, and it helps raters ease into the sequence of tasks in the procedure.  Its non-digital nature, however, has been an obstacle in converting the notes into analyzable data.  These rater-generated notes can reveal interesting aspects of rater cognition.  Designing empirical studies to capture such data to learn more about rater cognition would not only benefit the field of language education but also the field of education in general.  In fact, in the special issue of the journal Educational Measurement: Issues and Practice devoted to rater cognition, Myford (2012) emphasizes that more research on rater cognition is needed.

Other research questions worth exploring including obvious ones such as Is the bottom-up rater training procedure more effective than the more commonly practiced top-down approach?  Is there any difference between the two approaches in improving rater self-consistency (emphasized as the main benefit of rater training)?  These are just a few examples, and readers are encouraged to pursue their own research questions associated with the rater-centered bottom-up rater training procedure introduced in this article.

This article started by asking readers to imagine a novice rater performing a presumably learnable, but indeed tremendously challenging, task of using a rating scale to make decisions about learners’ proficiency.  Anecdotal evidence and informal observations suggest that rater training, for both novice and experienced raters, need not be like that – top-down, opaque, and anxiety-inducing.  Instead, a rater-centered bottom-up approach can make the process more transparent and positively engaging.  However, for this statement to be generalizable, we need empirical evidence, and this research topic is open to any interested readers.


NOTES:

  1. Due to space limitation, this section has been abridged.  Please feel free to contact the author for more information or samples.

References

Barkaoui, K. (2011). Do ESL essay raters’ evaluation criteria change with experience? A mixed-methods, cross-sectional study. TESOL Quarterly, 44, 31057.

Greer, B. (2013). Assisting novice raters in addressing the in-between scores when rating writing. (Master’s thesis). Retrieved from BYU ScholarsArchive.

Hamp-Lyons, L. (1989). Raters respond to rhetoric in writing. In H. W. Dechert & Raupauch (Eds.), Interlingual processes (pp. 229-244). Tubingen: Buner Narr.

Harsch, C. & Martin, G. (2012). Adapting CEF-descriptors for rating purposes: Validation by a combined rater training and scale revision approach. Assessing Writing, 17, 228-250.

Jacobs, H., Zinkgraf, S., Wormuth, D., Hartfiel, V., & Hughey, J. (1981). Testing ESL composition: A practical approach. Rowley, MA: Newbury House.

Joe, J. N., Harmes, J. C., & Hickerson, C. A. (2011). Using verbal reports to explore rater perceptual processes in scoring: a mixed methods application to oral communication assessment. Assessment in Education: Principles, Policy & Practice, 18, 239-258.

Knoch, U. (2011). Investigating the effectiveness of individualized feedback to rating behavior – a longitudinal study. Language Testing, 28, 179-200.

Lovorn, M. G. & Rezaei, A. R. (2011). Assessing the assessment: Rubrics training for pre-service and new in-service teachers. Practical Assessment, Research & Evaluation, 16, 1-18.

Myford, C. M. (2012). Rater cognition research: Some possible directions for the future. Educational Measurement: Issues and Practice, 31, 48-49.

Pufpaff, L. A., Clarke, L., & Jones, R. E. (2015). The effects of rater training on inter-rater agreement. Mid-Western Educational Researcher, 27, 117-141.

Rezaei, A. R. & Lovorn, M. (2010). Reliability and validity of rubrics for assessment through writing. Assessing Writing, 15, 18-39.

Royal-Dawson, L. & Baird, J. (2009). Is teaching experience necessary for reliable scoring of extended English questions? Educational Measurement: Issues and Practice, 28, 2-8.

Stevens, D. D. & Levi, A. (2005). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Virginia: Stylus Publishing, LLC.

Van den Branden, K. (Ed.).  (2006). Task-based language education: From theory to practice.  Cambridge, UK: Cambridge University Press.

Weigle, S. C. (1994). Using FACETS to model rater training effects.  Paper presented at the Language Testing Research Colloquium (Washington, DC).

Winke, P. & Lim, H. (2015). ESL essay raters’ cognitive processes in applying the Jacobs et al. rubric: An eye-movement study. Assessing Writing, 25, 38-54.

 

DOWNLOAD FULL ARTICLE