The context in which this experiment took place is the Computer Science Department at the University of Calgary, which accepts 150 undergraduate students per year at the second-year level of a four-year program. Interest in innovative testing methods is motivated by the department's mandate to triple the number of students it accepts within four years. A worldwide shortage of, and very high demand for, qualified instructors in CPSC compounds the testing and grading problem and we anticipate increased pressure on course instructors with respect to expansion of computer science because of increased numbers of students, larger class sizes, and more sections of courses.

The course in which the experiment took place is a fourth-year elective introductory course in artificial intelligence (AI) that surveys a large number of topics. Student investigate particular topics (based on text chapters) in dyads or triads and present their findings to the larger class, and post their work on the web. The course outline, all course materials, and student web notes can be found at http://sern.ucalgary.ca/courses/cpsc/531/w99/. The course was divided into two simultaneous lecture groups where different groups presented the same material. All students had full access to all course materials, including students' web notes in the other lecture section of the course. There were approximately 130 students registered in the course, equally distributed between the two lecture sessions.

The instructors had previously used short answer and paragraph answer examinations due to the perceived difficulty of setting multiple choice questions for the type of content covered in the course. However, the burden of grading short answer and paragraph answer items for a increasingly large class motivated the trial of an on-line, multiple-choice exam tool. The instructors chose to investigate on-line examinations over using more traditional machine scored formats (i.e., scantron sheets) because on-line exams have the potential to blend multiple choice, matching, short answer, and essay items in one exam. The instructors viewed mixed scantron and essay answer sheets as confusing for the students and a potential administrative problem for large classes. On-line exam environments, such as WebCT's, allow more automation of the marking process and more timely results; marks can be automatically displayed to students immediately upon completing the exam.

The decision was made to construct and administer the midterm examination as a completely automated WebCT form using multiple choice and matching questions, and present the final examination, made up of short answer and essay type items, using traditional, in-class administration methods. The rationale for experimenting with a web-based examination methodology is to automate more of the multiple choice examination and grading process.

WebCTs Testing Tools

WebCT is a web server designed to support on-line, web-based instruction. Faculty with little or no prior experience with HTML will find the inherent structure and design of WebCT to be useful starting points for creating online documents. The tool sets offered by WebCT provide valuable entry-level supports and templates as one creates course materials and online examinations. An extensive summary of our evaluation of WebCTs features is available online (Jacobsen, Wijngaards, Kremer, Shaw and Gaines, 1999). In brief, WebCT offers encapsulation and paths through existing course content, student presentation areas for groups of students, individual student tools for note-making, and a chat facility for synchronous, online discussions. WebCT was chosen for experimentation for a number of reasons, including free testing of WebCT with full functionality, inexpensive licenses, a large customer base, proximity of developers, ease of access via common web-browsers, and ease of customization (if needed). In Spring 1998, WebCT was installed on a local server so we could experiment with many of its features. This led to enough confidence in WebCT for its use with a local and distance graduate course in Fall 1998, and a distance graduate course and testing for an undergraduate course in Winter 1999. The technical support person and a course instructor were well versed in HTML, scripting, and other computer-related activities which influenced the evaluation and usage of WebCT. Based on experiments using WebCT to construct and administer graduate courses (without examinations), the following observations have been made:

• Support by the developers is very good. Queries and requests were dealt with quickly and to our satisfaction.
• It is easy to (a) install WebCT (beta 1.3 on Windows NT took slightly more time than 1.2 on Unix), (b) create courses as an administrator, and (c) encapsulate existing course material.
• It is moderately difficult to get used to all of the WebCT concepts and develop a mental map of the course structures, and to keep information up-to-date in a course, because the tool imposes a certain approach to structuring on-line course materials. One must become familiar with the affordances and constraints inherent in WebCT's design in order to construct well-organized course materials.
• It was difficult to customize WebCT to add connections to the external list server; Perl scripts had to be modified to add a mailto: hyperlink (referring to the list server) to WebCT's common button bar

• WebCT question banks and examinations for multiple choice, matching, short answer, and essay questions are relatively easy, though somewhat tedious, to construct.
• Multiple choice and matching exam items can be accurately marked by the automatic grading system.

Although WebCT offers short and long question styles, the selected-response type items were chosen in order to take advantage of the automatic grading feature in WebCT. Exam questions were either multiple choice or matching items. We chose not to display the students' grade immediately upon completion of the midterm to discourage potential cheating. There was no penalty for incorrect answers, so if time was running out, it was in the student's best interest to guess at the remaining exam items. The exam represented a considerable amount of instructor preparation time, but, since grading was done automatically, the marking time was essentially reduced to zero. WebCT provides handy statistical and graphical summaries of group results, as well as a complete item analysis. Instructors will find this summative information useful in analyzing student results and the "discrimination" performance of each item on the examination. While the additional preparation time for this type of exam would probably not recommend it for class sizes of less than 40 students, the ease of marking and availability of detailed statistical results makes it a useful tool for large class sizes. If statistical results are used in conjunction with a question bank (also supported by WebCT) and selected questions are re-used in subsequent courses, then the adoption of WebCT may eventually result in time savings for instructors of large section courses, and improvements in test item construction. Support was provided during the entire exam period by the two course instructors on campus, who were available by phone, and a network manager in case there were WebCT server difficulties.

Exam Construction and Preparation

For this administration of this exam we constructed a test bank of 51 items from which 33 questions were randomly selected when the student logged in. Some questions were chosen from groups of unique items and others were variations of the same question. The instructors can easily add to this test bank for subsequent sections of the course.

A feature of WebCT is that exam items are randomized only once per student. In the event that a student experiences a technical problem, such as a broken network connection, they would be able to log in again and continue at the point they left off. A potential drawback is that the timer keeps ticking while the student is disconnected; however, this "security" feature prevents students from purposely experiencing "technical failures" during the exam so they can distribute copies of the questions, or work out answers and reconnect later.

It is critical to provide opportunities for students to practice with the exam interface prior to the actual exam. Thus, we invited students to complete a pre-exam survey that used a similar interface to the actual exam (i.e., selected response and open-ended response items). The survey gathered information about students' prior online exam experience, expectations, and concerns. The pre-exam survey also served as practice with logging in and changing passwords prior to the exam date. Some students reported feeling unduly pressured by the time constraints during the actual midterm. A few students mentioned that they were very aware of the timer during the exam (there where no time constraints on the survey). Thus, before a high-stake online midterm or final exam, it may be beneficial to expose students to some short, timed quizzes in WebCT to familiarize them with WebCT's examination timer.

Student Profile

The majority of students are registered in their fourth (36.6%) and fifth (33.8%) year of undergraduate studies, and over 75% of students are taking 4 or more courses per semester. The gender ratio is four male students to every one female student, and over 90% are between the ages of 18 and 29 years.

Perceptions of Online Testing Prior to the Web-based Midterm Examination -- Students were asked to indicate where they planned to complete the online midterm exam. Most planned to complete the exam using computer facilities on campus (56%); some planned to use their home computer (31%); a small number planned to complete the exam at their workplace or office (5.6%) or some other place (7%). For the majority of students (97%), completing a course-based online midterm examination was a novel experience. In response to pre-exam survey about their preferred method of testing, 49% of students indicated that they would choose on-line, web-based exams, while 45% said they would choose an in-class, pen and paper examination. Students used a five-point scale (i.e., 1 = Strongly Agree, 2 = Agree, 3 = Neutral, 4 = Disagree, 5 = Strongly Disagree) to indicate their level of agreement with three, first-person statements. Table 1 summarizes the responses to these statements.

Prior to the exam (Table 1), the majority agreed that they "liked" (63.3%) and were "comfortable" (60.4%) with the idea of an online midterm exam. While 33.3% disagreed that they were "worried" about the exam, there were 43.3% of the students who were worried. Prior to taking the exam on-line, students were asked to consider the potential benefits and drawbacks to web-based testing methods. Students expressed a range of opinions about the potential benefits of on-line testing, from "increased time and location flexibility", "faster grading and feedback", "convenience for both student and instructor", "access to resources (i.e., open book exam)", to "comfortable environment (i.e., home vs. campus)". Students also described concerns about completing the midterm using WebCT, such as "effects of cheating", "possible technological failures", and "students might work in groups".

Student Feedback After Taking the Web-based Exam -- Students were asked to indicate the actual location at which they completed the online exam. The proportion of students who actually completed the exam on campus rose from the planned 56% to 65.4%. Almost one-third actually completed the exam at home (29%), a small proportion at their workplace or office (3.6%) or some other place (1.8%). A majority of students indicated that taking the on-line midterm examination did not change their attitude about on-line testing in any way (61.8%). Of the students who did report a change in attitude, the majority indicated that the online examination was better than they thought it would be (i.e., easy to navigate, ease of participation, no technical problems). A small number of students indicated that taking the examination strengthened their opinion that this is an ineffective method for testing (i.e., no instructor present for questions, worry about cheating, felt pressured by timer).

User Interface of WebCT Examination -- Students used a five-point scale to indicate their level of agreement with four, first-person statements about the design of the user interface for the examination (Table 2). Overall, the majority of students expressed high levels of agreement with all four statements.

After taking the midterm examination using WebCT, students were asked to describe the most important strengths/benefits of the on-line testing method used in the course. The most cited benefit was "anytime, anywhere" access, closely followed by "increased flexibility" and "convenience". One student wrote, "It allows you to define the atmosphere in which you write the test. I like to listen to music while doing this, so had a comfortable area with my discman going". Other students also described their satisfaction with being able to choose the environment and conditions under which they wrote the exam. Students were also asked to describe the most important weaknesses of online testing. Along with concerns over cheating and limited access to the instructor for questions, a small number of students mentioned "no immediate feedback about results" as a weakness. This could stem from their knowledge that immediate feedback was possible using WebCT, however the instructor chose not to use this feature for security reasons. One student indicated a lack of trust in the interface: "I went through the whole exam to double check that I had an answer for each question because I couldn't trust the information window on the right".

Students were asked to describe how the on-line midterm examination methods made it easier or more difficult to complete the examination. A large number of students indicated that online exams were better because they could type faster, or click on radio buttons faster, than they could write or fill in answers on a paper and pencil exam. One student wrote, "It made it easier for me to write the exam from my parent's home in XYZ, Ontario, thus not interrupting my vacation plans". Some students indicated that the novelty of the exam environment made it harder for this first exam, but they expected that this effect would be less for the next online exam. One student who completed the exam in a campus lab commented, "it was harder because there were more distractions than in a classroom where everyone is quiet because they are writing a test". The following excerpt effectively summarizes the mixed blessing that the online exam environment may be for students who would prefer to demonstrate their learning in alternate ways: "The interface is excellent and easy to use, like any multiple choice exam. However, not being able to show my thought process before arriving at a definitive answer for a question bothers me. Although I may have a pretty good idea of what's going on, if I'm limited to choosing a number and being "right" or "wrong", I can't show what I know". It is critical, in our opinion, to offer more than one type of exam or assessment for a course so that students have a number of different ways to demonstrate their understanding of course concepts and topics.

Students were asked to describe any problems they had accessing the on-line midterm examination (i.e., location, technology, WebCT interface, and so on). The majority of students indicated that they had no problems accessing the web-based exam. Only one student reported a crashed browser, and that student was able to re-establish a connection and complete the exam with no further problems. Connectivity and bandwidth apparently affected the submission time for some students who reported that it took a number of minutes to save certain answers. However, this problem is unverified in our experiment, as the tests and retests that we conducted did not repeat this problem.

Students used a five-point scale to indicate their level of satisfaction with their performance on the midterm exam. The majority indicated that they were neutral (61.1%), which may be a measure of their uncertainty with the content of the exam or the method of participation. A modest proportion of students agreed that they were satisfied with their performance (16.6%), and a smaller proportion disagreed (12.9%). In response to a request to indicate their expected grade, however, students were very optimistic: 13.2% of students believed they performed in the top 10 percent of the class, 47.1% in the top 25 percent, and 39.6% in the top half of the class. No one believed they performed in the bottom half of the class! Overall student satisfaction with the on-line examination method used in the course was relatively high immediately after the exam: 49% of students were satisfied or very satisfied with the web-based midterm methods, 27% were neutral, and 22.9% were either dissatisfied or very dissatisfied. Students provided useful information on how participation in this type of on-line examination can be made easier and more convenient for future students, such as giving more practice exams prior to the graded exam, schedule designated campus areas for students to write the exam in (both for supervision and noise level reasons), and improving the wording of a few items that seemed ambiguous.

Student Performance Results

Almost all students completed the exam and most students achieved acceptable scores. An interesting result was that performance expectations were much higher than actual performance. A number of students were surprised by their grades, which were lower than they expected. There are a number of possible explanations for this result. The midterm exam was designed to be rather challenging to complete in the time given if students were not adequately prepared. Most items were designed to require application, analysis and synthesis of course content, rather than merely recalling or recognizing facts. Students who had attended lecture, participated in labs, and had read all assigned materials, had a greater chance of success in this exam. One reason for designing a more challenging exam was that it was open book. Therefore, we expected that students who studied for memorization or recall would not perform as well as those who studied and attempted to understand higher order concepts. If students tried to look up answers in the textbook, then they would not have had time to complete the entire exam. One item, worth 10%, required students to complete a lengthy calculation. It is unclear whether this question added "noise" or error to the exam results. Many students spent an inordinate amount of time on this question, and may have had to guess at other questions as a result. It may be better to divide the exam into two separately timed parts to mitigate the possible random effects of the two types of questions (long calculation vs. standard multiple choice). We believe that varied environmental conditions may have affected some students' exam performance. Students chose the time and location to write the exam Some students were concerned about slow access times from home, and therefore chose to write the exam at a public computer on campus. However, a number of students reported being frustrated by the noise and distractions in the campus computer labs. Therefore, students who did not have access to a private study area equipped with an internet-connected computer on campus may have been at a disadvantage.

Student Reaction to the Online Exam

Completing a midterm exam online was a novel experience for almost all of our students. A majority of students identified the flexibility of "anytime, anywhere" access as an important strength and benefit of the online exam. They liked being able to write the exam at a time most convenient to them. Students did not have to travel to campus for the exam; they could complete the exam using a web browser from home, from a work place setting, or on campus. One student completed the exam from another province! Students appreciated being able to define the atmosphere and set up their work space for the exam; for example, listening to music at home, spreading out books, resources and tools. Finally, students liked the open-book nature of the exam; some reported increased confidence because they could access their textbook and notes during the exam.

On the other hand, students disliked the idea of unsupervised exams and the potentially unethical conduct of their colleagues. The two main concerns with online exams were cheating and being able to access the instructor during the exam. Although we published contact information for the instructors (home and campus numbers), students were concerned that they would not have immediate access to the instructor during the exam for questions, such as explaining the wording of an item, providing missing information, clarifying ambiguous items, and so on. A good number of students were very concerned about the potential for cheating, or group writing of the exam. They described the many (ingenious!) ways in which students could cheat, which they believed would penalize the more ethical students. Individuals reported some concern with the differences between a web interface and paper-based exams: 1) cannot write or do calculations in the margins, 2) little sense of how long the exam was or how much they had completed, and 3) not sure how to set up the workspace for the exam. Others commented on the relative benefits of submitting answers using point-and-click radio buttons versus color-in-the-bubbles scantron sheets.

Instructors who wish to experiment with WebCT's online testing features are well advised to post the exam on a dedicated server (we used a dedicated NT server) in order to reduce potential access and network load problems. One student prepared to answer the test at home early in the evening (~6pm) only to discover that his Internet connection was down (i.e., Calgary-based internet service provider over which we had no control). The student had to drive to the University and complete the exam in a public area. A feature of WebCTs testing environment that concerned students was that the timer only updated posted "time remaining" when they submitted an answer. Ultimately, it is the student's responsibility to keep track of time. However, students felt the WebCT timer provided misleading information. Finally, all the questions were selected-response, with one best answer. Some students were disappointed there was no opportunity for partial marks.

Next Steps

We believe this experiment was a success, and will continue to refine our online testing and grading procedures. Moving forward, we believe it is crucial to use exemplary test construction procedures and principles for an online exam. For example, to alleviate student concerns about content and the exam structure, the instructor should provide a clear table of specifications that outlines the number of items and relative weighting of content areas, and provide practice items and opportunities prior to the graded online examination. In order to address student concerns about cheating, a supervised, online exam situation could be arranged whereby students complete the midterm during a scheduled time in a supervised lab. This would require booking sufficient computer workstations for the class, assigning students to a particular lab, and employing several laboratory assistants who could check student identity, monitor the exam, and answer student questions during the exam. However, a scheduled supervised exam would sacrifice the benefit of "anytime, anywhere" access.

Acknowledgments

Financial assistance for this work has been made available by the Industrial Research Chair in Software Engineering, and the Government of Alberta's Learning Enhancement Envelope. Thanks are also due to the undergradaute class of 1999 for their participation in the experiments and the comments used to illustrate this article.

References

Goldberg, M.W., Salari, S. and Swoboda, P. (1996). World Wide Web Course Tool: An Environment for Building WWW-Based Courses. Computer Networks and ISDN Systems, 28. [On-line].http://www.webct.com/webct/papers/p29/index.html

Jacobsen, D. M., Wijngaards, N. J. E., Kremer, R., Shaw, M. L. G., & Gaines, B. (1999). The comparative evaluation of classroom and distance sections of an industrial software engineering graduate course. Proceedings of M/SET 99: International Conference on Mathematics/Science Education & Technology, San Antonio, Texas, March1-4, 1999. [On-line]. Available: http://www.acs.ucalgary.ca/~dmjacobs/seng/mset99_paper.html

Kristapiazzi, G. (1998). Compare Web Tools for Course Authoring. [On-line]. Available: http://www.geocities.com/Eureka/Gold/6012/compare_web_tools.htm