The Effects of Lexical Profiling Tools on Academic Writing Performance

ABSTRACT


Introduction
Many studies were done to investigate the factors that contribute to difficulties in learning English.(Misbah, Mohamad, Yunus, & Ya'acob, 2017) where their findings conclude that students' weakness in writing is due to lack of vocabulary (Usman & Abdullahi, 2018).Several recent studies on student lexicon also revealed that productive vocabulary is predictor of writing quality of second language learners of English (Usman & Abdullahi, 2018).These studies reported similar findings of earlier studies on the relationship between writing quality and students' lexicon (Astika 1993;eg. Engber 1995;Laufer & Nation 1995;Staehr 2008;Olinghouse & Wilson 2013).
Small lexicon is the biggest obstacle that hindered students from successfully acquiring language skills (Misbah et al., 2017;Usman & Abdullahi, 2018).Limited lexicon also prevents students from writing good academic work as their writings lack of lexical diversity and contain less lexical sophistication which are important elements that are judged in their writing.Ideas are not smoothly presented as students are struggling to look for accurate lexical items that can represent their ideas coherently and performing their academic tasks.As a result, their work is decorated with plagiarism.
Realizing the need to increase students' lexicon and enhance their academic performance, the current paper investigates the effectiveness of using lexical profiling tools to enhance students' academic performance in writing.

The objective of the study.
The purpose of the study is to investigate the effect of lexical profiling tools on academic writing performance and students' writing based on CEFR levels.The research questions that guide this study are, What are the effects of lexical profiling tools on the students' academic writing performance?and What are the effects of lexical profiling tools on the students' writing based on CEFR levels?

L2 academic writing challenges
Academic success is the ultimate goal for second language learners entering higher institutions.However, special register of academic language becomes a challenge to the learners as they face problems attempting to write academic tasks, a discourse/genre which is new to most novice university students (Fernsten & Reda, 2011).Rahmat et al. (2017) analysed 373 students' perceptions towards academic writing difficulties discovered that language use is among the factors perceived to be an obstacle for writing good academic work.Another study by Xiao & Chen (2015) conducted to gain insight into students' academic writing difficulties revealed that students' difficulties lie in three aspects which are language, content and structure.Language aspects were found to be the most difficult among these three aspects and students believed that small lexicon and word repetition are their main weaknesses in academic writing.Thus, control over academic language is crucial particularly when the students have to face higher-order thinking skills such as planning, organizing ideas, and revising as they advance further in their study (Srikrai, Lin, Lakaisone, & Sirinthorn, 2016).Therefore, learners must be proficient in academic language to perform academically.

Lexical richness as predictors to writing quality
Lexical richness often reflects writing quality produced by academic writers (Engber, 1995).Human judgement of the students' writing was also influenced by lexical richness found in the academic texts (Engber 1995;Olinghouse & Leaird 2009;Olinghouse & Wilson 2013;Auensen 2018;Ha 2019;Crossley 2020).Lexical diversity and lexical sophistication are the two strong constructs of lexical richness that predict the quality of writing quality found in several studies conducted over the years.One of the studies was done by Vögelin et. al. (2019) which analysed thirty-seven pre-service teachers in Switzerland rating four manipulated argumentative essays.Their findings showed that lexical diversity and lexical sophistication have a positive effect on writing quality as texts with greater lexical diversity and lexical sophistication scored higher by the teachers.
Although several studies have recently shifted to acknowledging the influence of ngram as predictor to writing quality (Kyle & Crossley ,2016), the importance of teaching single lexical items should not be neglected especially for 2L learners with poor lexicon as research findings also report the overlapping nature of lexical sophistication and ngram and these two constructs were the predictors for human judgement of L2 writing proficiency level (Kim, Crossley, & Kyle, 2018)

Lexical items in Academic Language
One of the special features of academic language is academic lexis (Coxhead 2000;Durrant 2016;Csomay & Prades 2018).Academic lexis is more frequently found in academic work as compared to general English texts and tend to contain more Latin and Greek lexis, Morphologically complex lexical items, Nouns, adjectives, and prepositions, informational density, Grammatical metaphor, including nominalizations and abstractness (Nagy, Townsend, Lesaux, & Schmitt, 2012).These features sometimes can become an obstacle for L2 learners to reach reasonable lexical threshold recommended in previous studies to excel in academic setting.
Previous studies have suggested the lexical threshold for reading comprehension Laufer & Ravenhorst-Kalovski (2010) and novice academic writers (Staehr 2008;Douglas 2013).Information on lexical threshold would help language instructors set the goal of teaching and learning lexical items.Staehr (2008) in his study investigating lexical threshold for reading, listening and writing discovered that learners' receptive lexicon is crucial to their writing as the size of lexicon accounted for 52% of writing quality.The result also showed that students with a receptive knowledge of 2000 word families performed better in all three skills.Thus, he suggested a knowledge of 2000 word families as lexical threshold for low-level language learners.Douglas's (2013) study to determine reasonable lexical knowledge required for novice university students revealed that students with a knowledge of 3000-word families could write a paper that covers 95% of their total lexical output while the access of 5000-word families would cover 98% of their lexical output.Thus, the automatization of 5000-word families knowledge could allow students to spend more time on planning and writing more critical academic work.However previous studies on the depth and breath of students' lexicon revealed that students' lexicon is limited, (Mokhtar, Mohd Rawian, & Singh, 2016;Usman & Abdullahi ,2018) and did not reached the lexical threshold recommended in the previous research (Staehr, 2008;Douglas, 2013).
Nor Ashikin, Noraziah, & Nur Fatima Wahida (2017) conducted a study on productive and receptive knowledge of 156 diploma students in a public university in Malaysia.Their findings revealed that the students have only knowledge of 2000-words families of productive lexicon which are below the threshold suggested previous studies.This finding also revealed that the students in the study were not ready to face the challenging demand of academic environment and the lack of lexicon would hinder the students to complete and perform academic tasks successfully.The poor lexicon would bring to the conclusion that the teaching and learning approach needs to be improved to enhance the students' lexical development (Mokhtar et al. 2016;Usman & Abdullahi 2018).Therefore, vocabulary intervention should be introduced to ensure the students are prepared with large lexicon to deal with academic writing tasks.

Teaching lexical items
Language instructors are sometimes unsure of what lexical items to teach (Schmitt ,2008;Nation ,2011;Sparks, 2013).They would rely on textbooks and reference books which mostly contain high-frequency lexical items (Koosha 2010;Oliveira, Martini, & White 2012).The students indirectly would have less exposure to the academic lexis and led to insufficient knowledge of academic lexis (Hsu 2009;Criado 2009).
The advancement in corpus linguistics has allowed language instructors to incorporate computer technology in teaching lexical items.The role of computer technology has provided the opportunity for language instructors to examine large authentic texts or also known as corpora and bring the information to the classroom.Corpus linguistics has led the path of collection of many corpora such as British National Corpus (BNC), The Corpus of Contemporary American English (COCA), The Cambridge English Corpus (CEC) and Cambridge Learner Corpus (CLC).These corpora have become the resource and reference for many wordlists as well as lexical profiling tools such as Lexical Profiling Tools, TAALES, Coh Metrics, English Vocabulary Profile, Textinspector.com and Text analyzer.Towns (2020) conducted a study on which wordlist would help university students in Thailand to learn lexical items.The result revealed that wordlist from CEFR levels are more useful for L2 students to use to improve their lexicon knowledge.The students in the study believe that lexis in B2 level is very important and useful to them.

Lexical Profiling Tools
Lexical profiling is an area of research that started with the breakthrough of computer software and the advancement of corpus linguistics.Nurmukhamedov & Webb (2019)  Advances in this area of research have also allowed the tools to be used in teaching and learning lexical items.
Thus, this study investigates the effects of lexical profiling tools on academic performance.

Method
This study employed quasi-experimental research design to answer the research questions in this study.The d ata were using quantitative method to investigate the effect of the intervention on students' academic writing.The independent variables in this study were Timed Informative Writing Test (TIWT), Lexical Profiling Tool s (EVP and texts analysers) and traditional methods language teaching using Communicative English 1 modu le.Timed Informative Writing Test (TIWT) was used to measure lexical diversity, lexical sophistication, stud ents' academic performance as well as students' CEFR levels in writing in pre and post-test.Web tools (EVP and text analysers) were used for the treatment group while traditional teaching method using Communicativ e English 1 Module was used for the control group.The corpus for this study came from Diploma in Electrical Engineering students of Polytechnic Sultan Abdul Halim Muazam Shah who registered for Communicative English 1 course during the study period of June-Oc tober 2018.They were from six intact groups between 21 to 34 students per group.Communicative English 1 is a requirement for all semester one students in Malaysian Polytechnics.

Research Instruments
This study employed a timed informative writing test (TIWT) to collect data for analysis.The topic chosen fo r the test was "The importance of education for the economy, environment, and health".The criteria used for constructing the topic for this test followed Douglas (2000) who suggested that participants' background kno wledge and subject matter knowledge are relevant in choosing the topic for the test.The students were requir ed to write an informative essay in two hundred words and complete the writing tasks in forty-five minutes.
The pre-test employing timed informative test was implemented in the first week of the semester.The same t opic for the post-test was given to treatment group on week fourteen of this study.This was decided based on Jacobs et al. (1981) who suggest that 'it is generally advisable for all students to write on the same topics beca use allowing a choice of topics introduces too much uncontrolled variance into the test', cited in Weir (2005) and Read (1990) provided evidence that different topic will produce different answers from the participants which will influence the choice of measure to be used for the study.This study applied three lexical profiling tools in the intervention to increase the students' productive lexicon which would help in improving their academic writing performance.The tools are EVP, a lexical profile base d on CEFR levels and two text analysers which are text inspector.comand Text Analyzer in roadtogrammar.com.
The experimental group was taught using the lexical profiling tools in their lessons while the control group w as taught using Communicative English 1 module.

Instructional Procedure
A fourteen-weeks plan of the activities was prepared for the experimental group.The control lesson plans we re similar to the experimental group excluding the use of lexical profiling in the activities.The students were required to use fifty percent words from A1, A2 and B1 level, another 20 percent words from B2 level and 20 percent for C1 level and finally 10 percent for words in C2 levels.Table 3.4 shows the schedule planned for t he experimental group.Fourteen Three hours 1. Use words in EVP to write a conclusion.

Data Collection
Pre AEW test was conducted during the second week while post AEW test was conducted during the fourtenth week of the study period.The answer scripts for both pre and post-AEW test were scored by two in dependent scorers who were not involved in the study.The marks awarded by both independent scorers were tabulated and averaged.The average scores were taken as the subjects" final pre and post AEW test scores.

0 Result and Discussion
4.1 Research Question 1: What are the effects of using EVP and text analysers on the subjects' academic writing performance?
To address the first research question, an independent sample t-test was conducted on the post-test scores of timed informative writing test for the experimental and the control groups.The effect size of using EVP and text analysers on the subjects' academic writing performance was subsequently determined.
An independent sample t-test was conducted on the Timed Informative Writing Test scores for the experimental and the control groups to determine the effect of using EVP and text analysers on the subjects' academic writing performance, since independent sample t-test rests on the assumptions of data independence and normal data distributions, descriptive statistics of the data are presented in the following   Independent sample t-test on TIW test scores.The assumption that the samples were drawn from the population with equal variances was established using Levene's test for equality of variances.The value for t is -3.798 with 66 degree of freedom (dƒ=66).The p (2-tailed) value is 0.00 which is smaller than the predetermined alpha value of 0.025 (α/2 =0.025)Thus, the null hypothesis which stated that 'there is no significant difference between the mean score of the experimental group and the mean score of the control group (Ho: µ1 = µ2),' was rejected.This suggests that there exists a significant difference between the means of the experimental and control groups.This conclusion was made at 95% confidence It can be inferred from the independent sample test result that the experimental group performed significantly better than the control group on Timed Informative Writing (TIW) test after going through the treatment (using Web tools, EVP and text analyser).This implies that using EVP and text analysers can significantly improve the subjects' academic writing performance.

Effect size of using EVP and text analysers on subjects' academic writing performance.
To quantify the strength of the difference between the means of the experimental and the control groups, the effect size of using EVP on subjects' academic writing performance was calculated.The effect size was calculated as the mean of the TIW test scores of the experimental group minus the mean of the TIW test scores of the control group divided by the pooled standard deviations.The result indicates that the effect size of using EVP on the subjects' academic writing test was 5.55.This can be interpreted as a large effect size according to Cohen (1992).

Conclusion.
Drawing from the inferential analysis results, the answer for the first research question "What are the effects of using EVP and text analysers on the subjects' academic writing performance?" is as follows; (a) EVP has a positive effect on the subjects' academic writing performance as the experimental group outperformed the control group significantly in the Timed Informative Writing (TIW) test and (b) The effect size of using EVP on the TIW scores is large.
4.2 Research question 2: What are the effects of EVP and text analysers on the students' writings based on CEFR levels?
To answer the second research question, the CEFR levels of the subjects' writings in the post-test were analysed.

Descriptive Statistics of the subjects' CEFR levels
Table 4.4 shows the descriptive statistics for the CEFR levels of the subjects' writing in the posttest.The mean and the standard deviation of experimental group for CEFR levels in the post-test are 3.36 and .898respectively while the mean for the control group is and its standard deviation is 2.92 and .753respectively

Conclusion
Based on the analysis of the CEFR levels received for the writings of both treatment and control group in the post-test, the answer to the fifth research question "What are the effect of using EVP and web tools on the CEFR levels of the subjects' writing in the post-test of Timed Informative writing test?" is that using EVP and text analysers encourage the treatment group to produce writings in a higher level of CEFR as compared to control group which was taught using traditional method.

Conclusion
Lexical Profiling Tools provide an alternative option for language learners to implement in their course besides the use of traditional method.These tools can arouse their interest in learning new lexical items.
From the analysis of this study, we can see that these tools have advantages in improving students' academic writing as well as enhancing their production higher level of CEFR lexis.Thus, this would give them the freedom to be critical in their writing and produce good quality writing.These tools could also allow students to work independently and become autonomous language learners.learner 6.0 Limitation.
Future studies should be conducted in other departments or other institutions.Future studies should utilize samples from a higher level of language skills and different semesters to see their development in their lexicon.Apart from that corpus from the students' assignments and other texts produced by the students throughout the semester could be used to examine their lexical development.
provide themes to this field and categorized them into four categories which are A. How does lexical coverage affect comprehension?B. How many words do you need to know to understand written or spoken discourse?C. To what extent do different text types provide opportunities for incidental vocabulary learning D. Which tools and methods can be used to investigate lexical coverage?

Figure 4 . 1
Figure 4.1 Normal Q-Q Plot for TIW Test

Figure 4 .
Figure 4.1 shows the normal Q-Q plot for the experimental group's TIW test scores and Figure 4.2 shows the normal Q-Q plot for the control group's TIW test scores.It can be seen from the graph that the distribution of both groups' test scores is normal since the observed test values are clustered along the straight line.

Table 3 .
Table 3.1 below shows the number of Electrical Engineering students registered for DUE 3012 course during the study period.Based on table 3.1, the sample size required for a 164 population of Electrical Engineering students in Sultan Muadzam Shah Polytechnic is 59 with the margin of error of .03.The sample size required for this study for each intact group is 30.However, for this study, 34 students are used for each experimental and control group since the number has been allocated for each intact group by the administrator.Two classes were assigned to the researcher to teach DUE 3012 Communicative English 1 during the study period.DET 1 C was assigned as the control group and DET 1 F as the experimental group. 2 shows the total number of students participated in the study is 68.This number constitutes 41.5% o f the total population.Table3.3illustrates the subjects' demography.The subjects' Malaysian Certificate of Examination (SPM) En glish results determined the subjects' general proficiency level in English.Those who scored A and A-were c onsidered advanced learners, those scored B+, B, and B-were considered intermediate learners and those sco red C+ and below were considered beginners level.

Table 4 .
1 shows the descriptive statistics for Timed Informative Writing test scores for the experimental and control groups.Based on table 4.4 the mean and standard deviation of Timed Informative Writing test scores for the experimental group are 7.26 and 0.23 respectively while the mean for the control group is 6.10 and its standard deviation is 0.19.The normality of data distribution was determined using normal Q-Q plot.

Table 4
Test for Equality of variances table reports the p-value of 0.306 which was larger than the predetermined alpha value of 0.05 (p > 0.05).This indicates that the null hypothesis which stated that the variances of the experimental group are equal to the variance of the control group (Ho: σ1¹ = σ2²)' failed to be rejected.In other words, there is no significant difference in the variance of test scores between the experimental and the control group.Thus, the assumption of equality of variance was met.Next table 4.3shows the result of the 't-test for Equality of Means'.

Table 4 .
5 shows the CEFR levels of the subjects' writings in the post-testTable 4.5: CEFR levels of the subjects' writings in the post-testBased on table 7.0, 32.4% of writings from the control group were in level A2 while only 20.6% of experimental group's writings were in level A2.As for level B1, the experimental group attained 41.2% while the control group percentage was slightly higher with 44.1%.The treatment group percentage for B2 level was 29.4% while the control group percentage 23.5%.Another 8.8% of the treatment group obtained C1 level for their writings however none of the subjects in the control group were able to obtain C1 level in the post-test.