Adapting a Checklist of Materials Evaluation for a Self-Access Learning Center in Japan

Materials evaluation calls for a systematic and principled approach. In reality, however, materials evaluation in language-learning self-access centers (SACs) is significantly lacking in good models. This paper reports on a project undertaken by SAC facilitators in Japan to investigate whether/how a pre-evaluation checklist developed a decade ago at a SAC in New Zealand (c.f. Reinders & Lewis, 2006) could be adapted to their target context. A mixed methods approach was employed where data was obtained via a Likert-scale questionnaire and follow-up interviews. The survey was adapted from Reinders and Lewis and enrolled 103 Japanese university students. The interviews were conducted to eight randomly selected survey respondents. Results show that the modeled checklist can be used as a basis with modifications allowing for contextual differences. The results suggest that Japanese learners of English value visually stimulating materials and require more guided support for them to effectively use materials beyond the classroom. The article provides an adapted checklist designed for Japanese learners of English as well as suggestions for future research.

institutional SAC budgets, and take Gardner's (1999) suggestion that resources be systematically evaluated and replaced for their ability (or inability) to support learners.
Building on the accounts of other established SACs, this study investigates our target learners' needs and wants for learning materials, aiming to eventually achieve a more principled, informed materials evaluation. A discussion of the literature in materials evaluation is needed to begin the process.

Materials Evaluation in Classrooms and its Application in Self-Access Contexts
Materials evaluation has been mainly discussed for the classroom context.
Defined as "a procedure that involves measuring the value (or potential value) of a set of learning materials" (Tomlinson, 2013a, p. 21), Tomlinson (2013b) claims that initiating materials evaluation is likely to be "ad hoc" because teachers ordinarily tend to choose new coursebooks by browsing in bookstores or catalogues. However, teachers' (aka evaluators') principles need to be clear so that their evaluation can be systematic and reliable (p. 31), so that a more objective and principled approach can take place in materials evaluation. This should be applied to materials evaluation in SACs.

Teachers/Learners
Language advisors (LAs) /administrative managers (AMs) /learners Like classroom teachers, SAC facilitators need to evaluate various types of materials for their strengths and weaknesses, as well as assess the timing of materials evaluation. SACs tend to accommodate various types of materials for the purpose of catering to a large body of diverse users. In relation to timing in the classroom context, Mishan and Timmis (2015) note above that pre-evaluation seems to be the most common because it is easier to undertake, but can be detrimental because it is likely to rely heavily on teachers' impressions. Tomlinson (2013b) explains that whilst-use evaluation tends to be more "objective and reliable" than making predictions in pre-use evaluation, as teachers measure the value of materials while they are being used, or obtain information by observing materials being used (p. 32). However, this sort of observation alone will not inform us of what is actually taking place in learners' thought processes. Thus, post-use evaluation may be more valuable because "it can measure the actual effects of the materials on the users" (Tomlinson, 2013b, p. 33).
However, it can be time-consuming to carry out.
Through this study, we have surmised that pre-evaluation is the most suitable, because at this time we aim to develop a practical tool that could be used for a large number of materials for non-exclusive target learners. Whilst-and post-evaluations are certain to be developed after a pre-evaluation instrument has been created, however that is beyond the scope of this study. With respect to how evaluation takes place, few empirical examples were found for SAC materials evaluation, while multiple approaches are proposed and implemented in the classroom context. The most common practice among them seems to be developing evaluation criteria. Mishan and Timmis (2015) suggest that criteria that make evaluators' values explicit can be generated, and that subsequent evaluation instruments can be developed. According to them, the criteria are often associated with a "checklist method" (p. 61), which involves developing a checklist that exclusively associates with specific criteria. This requires a long and complex process, not the "easy and quick" impression that the term 'checklist' may convey. Many such models (c.f. Gardner and Miller, 1999;Jones,1993;Lockwood, 1998;Sheerin, 1989), perhaps lacking in rigorous development processes, appear to have critical limitations and need further development because they are too time-consuming, too general, too lengthy, or indicate low reliability (Reinders & Lewis, 2006). Amongst the checklists in the available literature, we found a preevaluation checklist developed by Reinders and Lewis (2006) to be the most rigorous and reliable due to its systematic developmental process. The next section describes the developmental process and how it was modified for the Japanese SAC context.

A Checklist Developed at a University English SAC in New Zealand
Reinders and Lewis (2006)  Prior to the project, they asked SAC facilitators about good features of SAC materials to later find that asking learners directly was a better approach (Reinders & Cotteral, 2001). Then, they administered a four-point Likert scale questionnaire developed from a former project (Reinders and Lewis, 2005), to twenty randomly selected ELSAC users. Table 2 below shows their survey items. Note. N = 20. All numbers in the table represent percentages. The sum of percentages in a row may not turn out to be 100% since these percentages were computed by rounding off. Rating average was calculated by dividing the sum of the total number of the following ratings in each scale in a row by 20: 1: very important, 2: important, 3: a little bit important, 4: not so important. A lower rating average represents the learners' perceptions of higher importance.
(Adapted from Reinders and Lewis, 2006, p. 275) Because the rating average for item 3 ("Look nice") was relatively higher than other items, they concluded that their target learners do not consider materials' 'looking nice' important. Thus, they decided to eliminate the category of visual presentation of learning materials from their checklist. The following (Table 4) is the refined checklist they developed after repeated discussions, trials, and inter-rater reliability was checked.
They made sure to state the specific features to keep them generic to all proficiency levels, and to avoid points that were too subjective or difficult to answer quickly. Table 4 The Checklist Developed by Reinders and Lewis (2006) Features Yes/No/Unsure Comments Selecting the resource -Claims to be suitable for self-access -Clearly describes student level -Needs to be used sequentially Accessing the parts of the resource -An index -A table of contents -A detailed 'map' -A glossary -Chapter previews or summaries The learning process -Information summarized -Examples provided for tasks -Objectives provided for tasks -Keys / answers / criteria for tasks Learning to learn -Notes on the learning process -Shows how to set goals Other features (Adapted from Reinders and Lewis, 2006, p. 277) Whereas we acknowledge the fact that the checklist is dated and SAC resources may have changed and evolved since then, this is the latest and most rigorous checklist found in the literature, especially because it appears that no follow-up studies have been published elsewhere. The next section describes the context of the present study.

The Present Study
This present study finds its context in a university-affiliated SAC, which aims to foster learners' English language acquisition, in Chiba, Japan. Established in 2001, the Self-Access Learning Centre (SALC) incorporates a range of English language learning materials and services, including language advisory sessions, speaking practice and writing support, and student-led learning communities. At the time of this study in 2015, materials and resources numbered approximately 8,000 items and were comprised of didactic materials, authentic materials, in-house materials, a small selection of 'self-access' labeled materials, and EFL materials specifically designed for Japanese L1 learners. The selection and provision of materials are mainly facilitated by Learning Advisors (LAs), who assist in the location of suitable materials and resources and devise general and materials-specific learning strategies for users. Administrative Managers (AMs) are also involved in purchasing and maintenance.
For some time, LAs and AMs operated under a previously established set of policies that mostly relied on facilitators' ad-hoc perceptions and/or the physical condition of materials. This inspired a team of four LAs to develop a more objective criterion-based checklist. As described earlier, developing a rigorous and reliable checklist requires a lengthy process. Therefore, we decided to examine the adaptability of Reinders and Lewis' checklist because it seems to hold the most systematically developed criteria. However, as discussed earlier, our target learners possess similarities but also have different cultural backgrounds and experiences with English learning and use compared to ELSAC learners. Given this, we planned to first administer Reinders and Lewis' 'Good Materials' Questionnaire (Table 2.) to discover whether our target learners hold similar perceptions on good materials to their target learners. The research questions are shown below: 1) Can Reinders and Lewis' checklist be used as a foundation for our checklist? 2) Which features from the modeled checklist are subject to modification? 3) How can the identified features be reformulated to be more suitable for SALC users?

Method
Although the Reinders and Lewis checklist was well developed, we still found a couple of improvements that could be made to their data collection methods. Whilst they administered the survey to 20 students, a larger number of participants should be enrolled to represent a larger population of SAC users. In addition, while they did not obtain qualitative data from the survey participants, we thought that such data would provide useful information in the process of developing a checklist. We decided to conduct follow-up interviews as a means of data collection. As a result, we employed a mixed-methods approach (Creswell, 2015), within which explanatory sequential design was selected. Explanatory sequential design aims for qualitative data to add further information to the findings from quantitative analysis. We administered Reinders and Lewis' questionnaire, translated into Japanese, to explore our target learners' preferences for learning materials in the SALC. At that point, we decided to use the survey without modification because we aimed to examine the distributions of SALC users' responses as opposed to ELSAC users in order to identify which features of the checklist, if any, we should further qualitatively investigate for possible changes. It consisted of six statements to which the respondents were asked to indicate their degree of agreement on a 4-point Likert scale, 1 being "very important", 2 for "important", 3 for "a little bit important", and 4 for "not so important". The survey was administered via SurveyMonkey campus-wide to all enrolled students at the university in Chiba, Japan, to which the SALC belongs. The information about the survey was posted on a university portal site open to all students. A total of 103 learners responded to the questionnaire. After data was gathered, the number of respondents on each Likert point was counted. Then, cross tabulations were performed between corresponding items in the present study and Reinders and Lewis (2006) to see if there were differences in tendencies between these two studies with the aid of a non-parametric test to statistically examine these differences. Non-parametric statistical tests allow researchers to do statistical analysis with data not satisfying parametric assumptions, such as normal distribution of the data and equal variances across the groups, and even with categorical data (e.g., Mizumoto, 2010). While the chi-square test can be a typical choice for cross tabulations performed in the present study, it is not always robust especially when the sample size is small. With a small sample size and cells containing expected numbers less than 5, it is highly recommended to calculate the p value with Fischer's exact test. Since the number of participants in one of the groups was comprised of a small number, and the number of respondents was zero in some cells (e.g., for Items 6 and 7, no SALC students chose "Not so important"), we used Fischer's exact test to obtain more precise p values.
Follow-up semi-structured interviews were conducted with eight randomly sampled respondents. The interviewers mainly asked about the features that marked significantly distinct distributions between the two target parties. The interviews aimed to obtain survey respondents' deeper insights. The interview questions are shown in the Appendix. As all students were foreign language-majors, the interview was conducted and transcribed in English. The data were transcribed and analyzed thematically. provided further support for this tendency. In all seven items, more than 70 percent of the responses indicated "Very important" or "Important." On the other hand, in Item 3, the responses were scattered, and only about half of the responses indicated "Very important" or "Important," while the highest percentage was obtained for the scale "Not so important" among all of the eight items. Interestingly, a comparison of tendencies in SALC learners' responses to each item with those of ELSAC learners (Table 3) uncovered both similarities and differences in these two studies that took place in different SAC environments. A general tendency in the percentages of ELSAC learners suggested the similarity between these two studies: in most of the items, except for Item 3, more than 80 percent of ELSAC learners' responses reflected "Very important" and "Important,"

Survey results
which was slightly higher than the proportion obtained from SALC learners.
At the same time, however, there were noticeable differences between these two studies. First, there were more learners who considered Item 3 important among SALC learners than ELSAC learners (Table 6). Considering the proportions of the responses obtained for "Very important" and "Important," it can be said that about 50% of SALC learners responded that self-access materials should look nice. Conversely, more than two-thirds of learners in the ELSAC study thought that materials need not necessarily look nice. The second difference was that there were more learners who considered Item 8 was unlikely to be important among SALC learners than with ELSAC learners (Table   7.). Almost all learners in Reinders and Lewis' study thought that materials should definitely tell them how to learn best, while only 30% of SALC learners thought so.
Moreover, about 25% of the learners tended to think that it is not important. This result may imply that Reinders and Lewis' learners have more awareness in the process of "learning how to learn" than their SALC counterparts. According to the results of analysis above, it is concluded that two categories, "Look nice" (Item 3) and "Tell me how to learn best" (Item 8) need further investigation. The next section describes what follow-up interviews revealed in relation to those two points. give me information or important things. Because only long sentences are difficult to understand or I can't image what does the textbook mean". Interestingly, a few students noted that 'looking nice' could mean portability; many students at the university commute long distances and use their commuting time for studying and desire materials that are easy to handle.

Concerning
Regarding Item 8, whether or not a good material should tell how to learn best, interviewees struggled to provide examples/meanings of materials demonstrating the aspect of learning how to learn. After several turns of probing between the interviewees and interviewers, five features of such materials emerged. First, providing learning strategies was pointed out as important. For instance, one interviewee stated, taking an example of a TOEFL test book, "…so firstly I need to learn vocabulary but and secondly I have to practice a lot…just practicing and reviewing…like that…". Second, materials having an easy-to-navigate format was considered helpful. One mentioned, "…in my favorite books there are one picture and under the picture there are many important words are written in the list…". Third, showing rankings of essential contents was described preferred. One stated, "…it had ABC ranking like A is really important, you need to learn this stuff…". Forth, introduction of how to study with the material was found useful. One found the feature necessary, stating, "…on the first page it shows me how I am supposed to learn from the book". Lastly, navigating which chapter to start was claimed important. One stated, "…if they put like, 'first, I have to do this, then I need to do this', if a book directs me to study, that's good". The next section discusses what the results obtained from the quantitative and qualitative analysis will inform us with respect to the categories that should remain, be removed, or be modified in the checklist.

Discussion
The result that the majority of responses (six out of eight total items) showing similar distribution may suggest that there is a high probability that we can use Reinders and Lewis' checklist as a basis. It also suggests that the other criteria (except visual presentation and learning to learn) could remain, but the two points need to be reconsidered in developing our checklist. Meanwhile, the insights gained from the interviews as well as predicted gaps between the two sample groups (e.g. linguistic and cultural background, the time gap of ten years between the two studies) should be taken into consideration.
Pertaining to the look of the materials (Item 3), we contend that the criteria of visual presentation of materials should be added to our checklist (Table 8). The interviews revealed that "Look nice" means effective color and picture usage (including diagrams and charts), which, for the learners studied could be aligned with the idea that their culture reflects an acute sensitivity to the look of things (Inouye, 2016). Likewise, as Hill (2013) suggests, students today may prefer elements like color, pictures and text layouts because they are constantly exposed to visual imagery and that therefore may feel more comfortable seeing colorful images and graphics. Thus, a new category was incorporated as shown in Table 8. Sakai, Takagi, and Chu note that "educational and behavioral norms in Japan simply discourage their autonomy" (p. 12). If Japanese learners do indeed tend to have a lesser degree of autonomy, it explains why the interviewees struggled to provide their interpretation of "Learning how to learn". This could mean that more guided support for autonomous learning strategies for using materials are called for. The preferences that the interviewees expressed after long probing processes, such as showing the most essential content and learning strategies, were incorporated into our checklist. Thus, these new points were added as sub-categories (Table 9). Whereas ELSAC's checklist has a subcategory of "Shows how to set goals", we eliminated it because none of our interviewees mentioned this. Setting goals in learning English with materials might be too higher-order thinking for our target learners. That is to say, they may not have yet reached that stage in their journey of becoming autonomous learners. We hope that additional activities to facilitate engagement with learning materials should be incorporated more prominently into a SAC system. Tomlinson (2010) stresses that educators should mediate materials and learners so that self-access materials have learners experience the language in use. We are convinced that SALC facilitators should assist learners to foster their skills to use materials more autonomously. The new added features should certainly be repeatedly trialed and tested for inter-rater reliability and practicality. We expect that such a process will happen after this study. Such investigation will add more precise and reliable information about learners' preferences for learning materials.

Conclusions
The present study illustrated one attempt of adapting a SAC materials evaluation instrument originally developed more than a decade ago in New Zealand to a current Japanese EFL context. The results of analysis suggest that the instrument could be a sound basis for us to develop our context specific evaluation checklist. In other words, it was revealed that materials providing clear instructions, the language level, a lot of practice as well as examples, feedback (e.g., answers), easy navigation, and visual representation of materials (e.g., color and images) are considered to be valued features of good SAC materials for Japanese EFL learners. Also, it was hinted that they lack knowledge or training of how to use learning materials autonomously, and therefore an evaluative instrument with respect to learning how to learn may require different evaluation sub-categories, such as showing the most essential contents and learning strategies. Further investigation is also required regarding the point that they are likely to prefer easily carried materials.
Despite the limitations, we suppose that this study contributed to current scholarship in that it presented a procedure of adapting an evaluation checklist to another context. The present study also suggested preferred features for SAC materials that may be unique to Japanese EFL learners. These features will allow SAC staff in Japan to make better decisions about what materials are purchased and recommended to their learners. A further direction of this study will be the proposed checkpoints being further tested and negotiated so as to make the checklist more rigorous and reliable. It is hoped that the proposed procedure will encourage SAC facilitators elsewhere to develop their own context-specific evaluation checklists.

Notes on the Contributors
Atsumi Yamaguchi is currently a Learning Advisor at Meijo University and a senior lecturer at Kanda University of International Studies in Japan. Her research interests include learner autonomy, self-access, intercultural pragmatics, and intercultural communication.
Erin Okamoto is an Assistant Professor at Woosong University in Daejeon, South Korea and a former learning advisor/lecturer in the Self-Access Learning Centre at Kanda University of International Studies in Japan.
Neil Curry is a learning advisor in the Self-Access Learning Centre at Kanda University of International Studies in Japan. He has been teaching in Japan for 10 years and his primary interests are in Foreign Language Anxiety and language advising.
Katsuyuki Konno is a Lecturer at Ryukoku University, Japan. His research interests include individual difference variables in language learning, especially learner motivation, and their effects on learning behaviors. He is currently interested in motivational dynamics during language learning tasks and activities.