1. Introduction

It is very important for students to ask questions on what they cannot understand while learning. For this reason, many universities have set up an environment to instantly respond to questions from students, deploying teaching assistant(TA)s or educational supporter (ES)s in classes [1]. For 98 sophomore students learning computer technologies in Ritsumeikan University in 2017, we conducted a questionnaire on what they would do when they encounter difficulties in learning. In the questionnaire result, less than half students gave many questions until they were convinced, while 36 students gave no question. The questionnaire also had an item to investigate whether students know how important the questioning is to overcome understanding failures. The investigation shows most of them knew the importance. The questionnaire has revealed they do not ask questions even though they know the importance of questioning.

Some people point out it comes from psychological factors, For example, they seem not to ask questions, because they mind asking stupid questions. The questionnaire result shows it is useless to persuade them not to hesitate asking questions. It seems they cannot ask questions, because they cannot identify the points they fail to understand. It is impossible for them to compose questions as long as they have no image of what makes them in trouble. The psychological treatment would not overcome the problem that students cannot ask questions on points they cannot understand. Matters different from psychological seem to prevent students from grasping what makes them stuck. To facilitate for students to ask questions in classes, we need a technical solution to let them grasp what makes them stuck.

In this paper, we propose a method for students to overcome the situation where students are in trouble because of their failure to know what makes them stuck. The method constructs learning materials to guide all students to follow learning strategies of students who are able to ask questions. Some students can construct questions to reveal latent faults in their understandings. The questions are good enough to help many other students overcome their difficulties. Who can give good questions are aware of what they do not understand. In this paper, we regard they have attained awareness in learning, which is one of metacognitive learning strategies.

We assume such students ask questions, because they cannot deduce knowledge to be achieved from knowledge they have already achieved. In other words, they find a discrepancy of new knowledge from what is deducted from vested knowledge. The discovery of this discrepancy is the awareness of the students, itself. When new knowledge cannot be deduced from the vested knowledge, they review the part where vested knowledge was explained in teaching materials. In this research, we focus on this point. We obtain a history of how good questioners refer to web teaching materials. We extract reviewed parts when they find a discrepancy. We develop learning materials which make other students to follow the review. It enables all students to attain the awareness same as the good questioners. It is expected that the ability to construct questions will be improved in all students.

In the remaining parts of the paper, Chapter 2 describes existing research works on metacognitive strategies in learning and the ability to compose questions. Chapter 3 proposes a method to develop learning materials which aim to attain the awareness to improve the ability to construct questions. In Chapter 4 describes our experiment and its results to verifying the effectiveness of the method. Chapter 5 discusses the implication from the experimental results.

2. Question Ability from Metacognition

2.1. Meta-cognition in Learning

To recognize points students do not understand, it is necessary for them to recognize the meta-level recognition which supports their own learning. Meta-cognitive learning strategies are strategies to adjust their cognitive activities and learning activities [2].

Existing research works on metacognitive learning strategies include planning to build learning plans [3], objective grasping of their own learning processes [4], and changing learning behaviors during learning depending on environments and contexts [5]. In addition, Umemoto says that each meta-cognitive learning strategy has its own influence on the learning process [6].

Students recognize points they fail to understand, when they recognize how the points are related to knowledge they have already obtained [7, 8]. To achieve it, it is necessary for them to master each of the cognitive strategies above. Meta-cognitive learning strategies seem to have great influence on abilities to construct questions in the learning process.

2.2. Questions Composition Ability

Strategies related to questioning have been evaluated as the help seeking in Motivated Strategies for Learning Questionnaire (MSLQ) [9]. In addition, help seeking has been studied in terms of typical cases [10], learning [11], counselling [12]. Phuong et al. [13] propose a method to clarify the relationship between multiple learning strategies and students' behavior, including the help seeking. They built personas of learners based on the relationships. According to the work, the learning behavior of students such as the learning time, pages students refer to, and their correct answer rate are used to quantify earning strategies. It divided students into 3 types, each of which corresponds to a persona. From this, it is the work suggested that a clear difference appears in the learning behavior due to differences in learning strategies including the help seeking.

Researches focusing on self-esteem [14] and social embarrassment [15] are carried out as a factor impeding help in learning. However, in the result of the questionnaire conducted for students of Ritsumeikan University revealed that many of the students avoiding questioning know the importance of questioning. Many of them also stated they do not know what to ask. It suggests the avoidance does not come from psychological factors, but from lack in abilities to construct questions.

Students who cannot organize their knowledge are difficult to construct questions. We should eliminate the situation where students have no image on points they cannot understand. Clear learning prospects would provide students with abilities to construct questions which lead to overcome understanding faults.

2.3. Questions from Awareness

Learning strategies to make prospects on learning are evaluated as meta recognition in [3]. In order for students to compose questions, they need the ability to make a prospect for learning.

We assume the meta recognition corresponds to awareness of students in their actual learning activities. If students get aware of a relationship of the knowledge to attain with the knowledge they have already obtained, they can build a prospect for their learning, which lead them to overcoming their understanding faults.

This paper regards the awareness of students as recognition of a strategy to make prospects on their learning. In order to construct questions to overcome their understanding faults, students need to get aware, while they are learning. Students can solve a problem they are inferior in the ability to construct questions by getting awareness.

3. Learning Materials to Enhance Question Composition Ability

3.1. Research Goal

The purpose of this research is to develop learning materials to improve the ability of students to construct questions. In order to develop the ability, it is necessary to acquire awareness with meta cognitive learning strategies.

In this research, we encourage students to acquire awareness in web teaching materials. It seems that students who acquire awareness in the learning process take learning behavior leading to it. Enforcement of the learning strategy extracted from such learning behavior to other students enables them to find clues to get awareness. Since there are various kinds of clues to get awareness, they vary with individual students. In order to provide awareness for more students, it is necessary to analyse the tendency of the clues so that we can extract their features. Web teaching materials that encourage students to get awareness lead to ability development of students to organize their learning, which improves their ability to construct questions.

Students have doubts on new contents to learn, when there are discrepancies of the new contents from what students deduce from the knowledge they have already acquired. They also feel empathy when other students express questions relating to the same doubts.

In order to compose a good question, students should have abilities to examine discrepancies of the contents they are studying from what is deduced from knowledge they have already attained. This research regards student awareness as discovery of the discrepancy.

It is necessary to encourage awareness in learning with teaching materials in order to improve student's ability to construct questions during classes. We propose a method to construct web teaching materials based on learning behavior of students who make good questions to encourage awareness of other students. The method promotes students who have not aware to follow the learning behavior of students who have attained awareness. It tries to identify factors which prompt awareness. It enhances Web teaching materials with a new mechanism, which guides the former students to follow the learning behavior of the latter students. The promotion of awareness with the Web teaching materials can develop the ability to construct questions, without imposing loads other than learning for instructors and students.

Figure 1. Method overview

3.2. Learning Material Development Cycle Based on Question

In this paper, we propose a method to develop web teaching materials that encourages students to be aware of the discrepancies from learning behavior of good questions. The teaching materials developed in this paper are supposed to be supplementary teaching materials to be studied before lectures. In addition, it is assumed that Web teaching materials are composed of explanation on contents to be learned, and question-and-answer functions with which the teacher responds to questions from students. Each of the explanation articles is assumed to be presented with a Web page, which is short enough for students to browse the whole of the page on the PC screen without scrolling. In this research, we refer to the record of page transition in browsing of explanation articles of a specific student as his learning history.

The first step of the method is to record the learning activity of students on the web teaching materials. The learning activity consists of two items: the time each student browsing of each student on a specific page and the transition from the page to another. Students take a lecture based on contents studied with web teaching materials. Students present questions during the lecture to the web teaching material. Using the question and answer function of the web teaching material, the teacher answers to the question. The function shows the exchange of the question and the answer to other students. When other students in the lecture feel sympathy with the question, they can vote the question as a good question. The method identifies the student presenting the question that gets many votes. It also extracts his learning history showing how he browses pages on the web teaching material before he issues the good question.

Analysis of the learning history of the good questioner identifies his learning strategy. The method incorporates a mechanism to make other students follow the learning strategy. Publish of the web teaching materials containing the mechanism is a feedback of the questions and answers as well as the voting. Repeating this cycle, we can develop teaching materials to promote awareness of students.

3.3. Evaluation of Good Questions

A good question is a question that helps many students solve doubts. A student who can construct such a question has the ability to be aware of a discrepancy of new knowledge to be attained from what many students can deduce from the knowledge they have already obtained.

Merely pointing out discrepancies is not a good question. Depending on the way of questioning, it is impossible to clearly indicate what the discrepancy is, even if the student can show that he has been aware of the discrepancy. A good question must reveal the discrepancy to the teacher to make him answer the question to solve the discrepancy.

In this research, questions which gets many sympathy from other students are regarded as good questions. In this method, the student gets a response from the teacher by posting questions on the web page in the lecture. This interaction can be seen by other students. In addition, participants in the lecture will vote on the question they feel sympathy on the web teaching materials. The question collecting many votes here is a question that notifies many other students of the discrepancy. Such a question gets a lot of votes because it reveals latent questions, which many students have, but few of them are aware of.

3.4. Feedback Method Using Links

Students who ask good questions recognize their doubts. They express the doubts as questions using words, after they organize their acquired knowledge and new knowledge to be acquired. It means the students must be aware of the discrepancy between deduction from the acquired knowledge and the new knowledge. The students with awareness would check explanation articles which gives them the acquired knowledge when they think the new knowledge is different from what is deduced from the acquired knowledge. It is a learning strategy of the students with awareness. Therefore, the learning strategies for acquiring awareness are considered to appear in the history of web learning.

This research pays attention to page transition and the time for students to stay in each page.

Suppose a student who has not been aware of discrepancies imitate the learning behavior of students with awareness. This corresponds to an unaware student following the learning strategies of students with awareness. Experience of learning strategies leading to the awareness would promote unaware students to check their discrepancies with pages they have studied in the web teaching material.

The behavior recorded in the learning history of a student with awareness implies his learning strategy to find the discrepancy. The student with awareness would look back to web pages containing explanation of a content he finds a discrepancy, which triggers his good question. Analysis of the learning history of the student with awareness identifies his learning strategy, which is represented with the page link corresponding to the page transition the good questioner takes to solve his discrepancy.

As the mechanism to make other students follow the learning strategy of the student with awareness, the method proposes to incorporate a page link into web teaching materials. The student with awareness would be good at creating an index indicating the location of explanation of the words and the items that brought the discrepancies. He uses the index to check the explanation. In order to allow students without awareness to trace this learning strategy, the proposed method adds a link to a page that explains the acquired knowledge to the web teaching material. By adding the links, other students have opportunity to visit an explanation part of the learned knowledge, which enables them to organize the difference between the deduction from the acquired knowledge and the new knowledge. It promotes them to constitute a question to solve the question.

3.5. Factors Providing Awareness

Given the same teaching material, a student who understands the contents well has an index showing which page explains the learned contents. If students have doubts to acquire new knowledge, it is considered they should refer to the same place. In other words, page transition probabilities of students who understand contents well are considered to be similar. For each student, we calculate the page transition probability for all pairs of pages.

Each of pages has various factors for the explanation. As the factors, this paper considers the keyword characteristics and the roles of pages. To know the factors, we calculate the TF-IDF value of each word in explanation articles on web teaching materials to extract keywords. Each keyword belongs to specific technical areas. On the other hand, each page has its role. Some pages describe a mathematical expression, while others illustrate new concepts using examples.

The understanding of leaners depends on the factors such as keyword characteristics and page roles. Let us classify pages based on the factors. The set of page groups created with the clustering differs depending on what factors are adopted. Due to the factors, the similarity of their page transition probabilities is different. Note that students who have awareness in the learning of given contents often look pages back to confirm the knowledge they have already attained in their learning. When pages are clustered with factors that are useful to promote the awareness, we could find high similarity in their page transition, while low similarity when pages are clustered with invalid factors. Examination of the similarity, we have a good chance to see which factor is useful to promote the awareness.

4. Experiment

4.1. Outline of Experiment

We conducted an experiment on 25 students studying computer technologies in Ritsumeikan University, in order to verify whether it is possible to construct teaching materials that encourages students to get awareness from learning behavior of good questioners for web teaching materials. In the experiment, students were asked to explain keywords of the following 3 field: concepts of linear algebra such as vector, norm and inner product, concepts of statistics such as variance, covariance, and, correlation coefficient. They were requested to submit a report summarizing the content of web teaching materials explaining collaborative filtering using those concepts.

We recorded learning history of students on web teaching materials in the process of creating reports. We prepared 2 kinds of web teaching materials. Material A has 20 links, which are created with the proposed method. Let us refer to them as links promoting awareness. Material B also has 20 links. To settle the links, we selected 20 keywords, to pick up pages where specific keywords appear in material B. We randomly select 2 pages from them, to settle a link from one page to the other. If the learning histories of many students browsing materials A get similar to that of students with awareness, it can be said that links promoting awareness play a vital role to bring the awareness to them. We compare the learning histories of students browsing material A and material B from this viewpoint. We also investigate the factors that promote awareness, comparing the quality of submitted reports with the learning histories. The quality of the reports should be good, if students succeed in organizing the content of a material. In addition, we examine what kind of behavior students take in their learning, if their report is evaluated to be high in its quality.

4.2. Link Settlement in Teaching Materials

To determine links in teaching material A, 10 students engaged in a group discussion, referring to the web teaching material. We picked up good questions in the discussion, by voting.

We analyse the learning history of students who asked good questions, to give links with awareness. Traversing links by mistake or misunderstanding causes noise in the transition information. To avoid it, we counted traverse of the links, only when students stayed for more than 5 seconds in the destination page of the link. We add 20 links extracted in this way to material A obtained.

To settle links in material B, the TF-IDF value was calculated for every word included in the web teaching material. The words of high TF-IDF values were extracted as keywords. Among pages containing a specific keyword, two pages are chosen to settle a link from one page to the other. Twenty of the links created in this way were assigned to material B.

4.3. Number of Link Usage in Each Web Teaching Material

We prepared a web teaching material combining the same number of sections selected randomly from material A and material B for each of 25 students in the experiment. Each student described a report, referring the web teaching material assigned to him. In order to investigate the influence the link has on the students in the experiment, we calculate the number of link usage times of each student as well as the link use ratio to the total number of transitions. The number of times of link use is shown in Table 1, and the link use ratio for all transition times is shown in Table 2. The rightmost column in Table 1 indicates the difference of the number of times of following the link in material A from that in material B. The value is positive when the former is larger than the latter. Similarly in the rightmost column of Table 2, the difference of material A from material B is shown for the use ratio of the link to the total number of transitions. 24 out of 25 students used more links in material A than those in material B, in case of the number of link usage times. In case of the link use ratio to the total number of transitions, 20 out of 25 students used more links in material A. Apparently, the students liked to use links in material A more than those in material B.

Table 1. Number of link usage times for learning materials A and B

Table 2. Link usage rate for all transition numbers of each student

4.4. Keyword Coverage in Reports

We requested the students to describe their reports up to 10 pages. We instructed the students to raise the uniqueness and the comprehensiveness to describe their reports. We explained the uniqueness means prohibition of plagiarism. The comprehensiveness of the reports submitted by the students in this experiment were evaluated by the coverage rate of the keywords used in the report.

Reports should cover as many keywords as possible within limited pages, to raise its comprehensiveness. We can regard students who grasp and compile important sections correctly have prospects in learning. In this experiment, TF-IDF analysis was performed on web teaching materials beforehand, which extracted 33 keywords. The comprehensiveness of their reports are evaluated by measuring how many keywords obtained here appear in their report. Comparison of the number of occurrences of keywords is shown in Table 3. Students whose link usage rate of material B was higher than that of web material A submitted reports containing 16.8 keywords in average. On the other hand, students whose link usage rate of material A exceeds that of material B submitted reports containing 20.0 in average. Let us focus on the top five students whose degree of link usage of material A is greater than that of material B. Their reports contain 20.8 keywords in average. Students using more links in material A are superior to students using more links in material B from the view point of the comprehensiveness.

Table 3. Average number of keyword appearances

5. Discussion

5.1. Effects of Links

From the number of times of link use and the usage ratio in each teaching material, we consider what kinds of influence is brought to students with links promoting awareness and those neglecting awareness.

Ÿ Link usage count

As Table 1 shows, 20 of 25 students used more frequently the links in material A than those in material B. The average of the number of times of link use in material A was 11.4, while that in material B was 3.5 times In addition, the difference of link use in materials A from that in materials B tends to increase, as students use more links. From this result, the links in material A help the students learn more than those in material B. The former are links promoting awareness, because they are created from the reference method of the students with ability to construct good questions. The latter are links neglecting awareness, because they mechanically associates keywords occurring in some pages with those in other pages.

Ÿ Link usage rate

Since the learning method differs depending on students, the number of page transition varies with students. Therefore, the significance of one page transition also varies with students. Since the learning contents are different in material A and B, the number of page transition cannot be compared fairly. For the fairness, we compare the influence of 2 kinds of links, with the ratio of the page transition using links to the total number of transitions in the web teaching material. As it is shown in Table 2, 20 out of 25 students use the linkage of material A at a high rate than material B. The average percentage of link usage was 8.7% in material A, while 5.7% in material B.

It implies the links promoting awareness help the students learn more than the links neglecting awareness, under various learning method.

The superiority of the links in material A in the 2 aspects above implies the advantage of setting links based on behavior of good students learning new items. It excels the keyword base setting of links in support of student learning.

5.2. Effects of Links

The student's learning degree was evaluated with the coverage rate of the keywords of their reports. In Table 3, 16.8 keywords appeared in average, in the report created by the students who use more links in material B than material A. On the other hand, 20.0 keywords appeared in average in reports created by students whose usage of links in material A is higher than that in material B. It reveals that students who prefer the links promoting awareness to the links neglecting awareness produce more comprehensive reports. Students can extract keywords comprehensively, because they learned web teaching material with balanced perspectives. Let us examine the keyword occurrences of reports submitted by the top five students whose degree of link utilization in material A exceeds that in material B. The average of the keyword occurrences of these five students was 20.8. The more the students value the links promoting awareness, the more comprehensive reports they would write. It suggests the links promoting awareness lead students to learning with perspectives.

5.3. Consideration on Page Transition

In this experiment, the page transition probability of the student was calculated for all pairs of pages in order to identify the factor that promotes awareness. In addition, clustering focusing on keyword features, page roles, and expressions used on pages was performed to calculate transition probabilities between clusters. In the page transition made by the students aware of their discrepancies, the similarity would be high when it is clustered with effective factors, and low when clustered with invalid factors.

However, as a result of the analysis, no significant similarity was found in either case of page transition probability, cluster transition probability of keywords, or cluster transition probability on page roles. It could be due to the experiment conducted in various environments. Students joined to the experiment in various environment, without specifying places, time, and consultation with other students. The learning environments might affect concentration of the students during the experiment. In order to find factors that promote awareness, it would be necessary to control the experimental environment so that all students engage in the task in the same condition. It would also be necessary to consider affecting factors from different aspects. To find aspects, we need to examine internal factors of students, interviewing what they do when they are stuck in learning of web teaching materials.

6. Conclusions

This paper proposed a method to develop web teaching materials that promote awareness. The method analyzes the learning history of students who can construct good questions during they receive lectures. The students look back pages explaining items on which they found discrepancies from their own knowledge, referring to previous pages of web teaching materials. The method adds links that allow other students to take similar reference. By the method proposed in this paper, it is possible to improve the teaching materials so that learning can be advanced efficiently. Experiments using web teaching materials proved that learning with links that promote awareness are more likely to lead students to comprehensive understanding than learning with links connecting keywords without cares for awareness. As future tasks, it is necessary to identify specific factors that encourage awareness. In this experiment, learners were allowed to study freely without restrictions. However, in order to identify the factors that encourage recognition, it is necessary to further limit the experimental environment. It is also necessary to investigate how each factor can be reinforced in learning.