Jacobsen, D. M., Wijngaards, N. J. E., Kremer, R., Shaw, M. L. G., & Gaines, B. (1999). The comparative evaluation of classroom and distance sections of an industrial software engineering graduate course. Proceedings of M/SET 99: International Conference on Mathematics/Science Education & Technology, San Antonio, Texas, March 1-4, 1999.

The Comparative Evaluation of Classroom and Distance Sections of a Industrial Software Engineering Graduate Course

D. Michele Jacobsen
Niek J.E. Wijngaards
Rob Kremer
Mildred L.G. Shaw
Brian Gaines

Software Engineering Research Network (SERN)
Department of Computer Science
University of Calgary, Calgary, Alberta, Canada
{jacobsen, niek, kremer, mildred, gaines} @cpsc.ucalgary.ca

Abstract: This paper describes the extent to which we achieved our original objectives with a distance-based learning model for a graduate course in software engineering. The approach taken to compare concurrent local and distance learning environments includes evaluations from three perspectives: 1) the teachers, 2) the learners, and 3) technical support. As a result of the current experiment, our next experiment will be focussed on a distance-only offering of a graduate course. Introduction

The Software Engineering Research Network (SERN) is funded by industry and administered by the Industrial Software Engineering Chair at the University of Calgary to support the dissemination of good practice in software engineering. One component of SERN's activities is a research-based masters program with a specialization in software engineering targeted at students with industrial experience in full-time employment (Shaw & Gaines, 1998). This program has dual objectives: 1) to develop highly qualified personnel, and 2) to encourage industry-based software engineering research with a focus on good practice. The coursework component of the M.Sc. includes required courses that have been developed in consultation with the industry partners. The learning environment for these required courses is unconventional and reflects the industrial experience of the students. After the first background lecture, students take over much of the responsibility for presentations on various topics in each course. The instructors' role is that of facilitator managing a process of debate and exploration rather than attempting to be an authority in the domain. The overall aim of the M.Sc. program is to provide a supportive and nurturing learning environment in which experience and knowledge can be constructed and shared, ignorance displayed and errors made without censure and with ready access to diagnostic help. It becomes clear to students that, while there are no easy answers to the core questions of industrial practice in software engineering, there are many useful perspectives and that simplistic, surface level answers generally have very limited applicability. Students make their coursework accessible to others by posting assignments on the World Wide Web. An e-mail list server is used for continuing discussion of the course topics outside the class environment. These two features have made it possible for students whose companies relocate them, or who change jobs to companies in other locations world wide, to continue to participate in the courses. Remote students who have already come to know their colleagues can continue to present and share material through the web and participate in discussion through the list server.

Towards A Distance Model

We are currently addressing the challenge of making the M.Sc. program formally available to remote students while maintaining effective ongoing participation. A first step has been to develop and offer parallel sections of SENG611, a team-taught, six-week Requirements Engineering graduate course. One section of SENG611 was delivered on campus, and the other delivered at a distance using facilities provided by WebCT (Goldberg, Salari, and Swoboda, 1996). The option of participating in the course using a distance model was offered to students during the first of six lectures. Of the 21 students in the course, 7 participated in the distance section, and 14 on campus in a conventional face-to-face learning environment. In addition to individual assignments, all students collaborated on a major group project that they presented during the final lecture. This paper describes the extent to which we achieved our original objectives with a distance-based learning model. The approach taken to compare and analyze the two learning environments includes evaluations from three perspectives: 1) the teachers, 2) the learners, and 3) technical support. First, an analysis of the instructors' experience with the two sections of the course includes a description of pedagogical requirements before and throughout the course, and a retrospective consideration of lessons learned. Second, an exploratory analysis of information collected from students, both throughout and at the completion of the course, is presented. The third perspective taken is that of technical support. WebCT was customized in order to meet the requirements of SENG611; a report is provided on the efficacy of this tool for a graduate course in software engineering. The results of the current experiment have implications for the design and delivery of subsequent software engineering courses offered at a distance.

Pedagogical Requirements

The original pedagogical requirements for the local and distance sections of SENG611 included the following:

  1. The existing course structure of SENG611 was to be maintained during the experiment, which includes publicly available student work and collaboration on group projects.
  2. Existing web-based course materials were to be re-purposed for use in WebCT.
  3. Both local and distance students were to be actively engaged in the course.
  4. Differences between local and distance student participation in the course were to be minimized by providing: a) the same course content, assignments and deadlines, b) access to in-class interactions (instructor & students), and c) insight into the personalities of the two instructors.
  5. Neither group was to be disadvantaged by the experiment with concurrent sections of SENG611.
Subsequent discussion focuses on the impact, implementation, and an assessment of each of these requirements from each of the three perspectives.

Instructor’s Experience

Prior research has found that faculty spend more time preparing materials and resources for instruction when integrating computers into their teaching and learning on campus (Jacobsen, 1998). Instructor workload increases, at least initially, for teaching and learning online (Harasim, Hiltz, Teles, & Turoff, 1997). Our results are consistent with these findings. To meet student needs in concurrent local and distance sections of the course, one instructor spent additional time on the following distance-related tasks: 1) "repurposing" existing course web pages for use within WebCT, 2) creating detailed web materials for the distance group, and 3) locating and organizing online references and links to additional materials. In addition, both instructors felt they spent more time fielding e-mail and telephone inquiries from distance students than would have been spent for these questions in an exclusively face-to-face offering of the course. Time spent on these tasks was over and above the time required for conventional classroom instruction. The instructors were participant observers in the list server, rather than active and directive moderators (except when students appear to be involved in a non-productive debate or require expert advice). The instructors assessed list server participation during the course, and found the sheer volume of discussion was greater than in previous, classroom-only offerings of the course. There are a number of possible explanations for this trend. First, a large proportion of the local students speak and write English as their second language, and seemed to prefer to participate more in the list server than in the lectures. It may be that these students are more confident when they have time to reflect and respond to a written discussion, rather than speaking out in class. Second, distance students actively participated in list server discussions, possibly because they were not present for the classroom discussions. A third possibility is that students actively participated because this was a graded expectation for the course, although this expectation was unchanged from previous classroom-only offerings of the course.

Pedagogical Lessons Learned

We discovered that it is important to carefully structure the entire course before it starts and to be faithful to that structure throughout the course to provide distance students with a stable context in which to learn. While this finding may seem rather obvious from an instructional design standpoint, it is not the typical approach in SERN's graduate course offerings. Instead, flexibility to change course material and follow "excursions", as dictated by student interests, experiences, and current situations, is favored over rigid structure. This constructivist approach may be at odds with the need for more structure in a distance model. This is worrisome because this approach is highly valued by both students and instructors. We are currently investigating instructional strategies that provide a balance between carefully structured and defined course design, and responsive, flexible learning environments. It may be that our distance students felt they needed more structure because the distance model of learning was new to them. The distance students monitored what the local students were doing in the course, and were fearful of falling behind or missing important information. We will continue to evaluate student learning experiences and concerns with subsequent distance course offerings.

We learned that although the course has always included extensive web-based notes, considerable effort had to be invested to reconsider and repurpose the online materials to make them useful for the distance students. For example, overhead slides that presented point-form information were insufficient instructional supports for distance students who did not have the benefit of the lecture context, presentation, and related discussion and interaction. We also found that descriptions and instructions for assignments and "in-class" exercises had to be made more explicit for distance students.

The instructors believed that the quality of assignments and amount of effort invested in participatory activities varied much more in the distance than in the local groups. One possible explanation is that the face-to-face interaction among and between students and instructors in the classroom allowed them to arrive at a common understanding of expectations. This finding suggests to us that instructors must find a way to be very explicit about discussing and arriving at a common understanding of expectations with their distance students. It is our hope that we find an innovative way to achieve this goal without stifling the creativity and individual style of student's knowledge construction. A related finding, which may be an artifact of this comparative study, was the instructors' belief that distance students' contributions to the list server varied more than discussion by local students. Again, a possible explanation for this difference is that distance students were participating in the course without the benefit of the shared context created in collective face-to-face discussion and interaction. We attempted to capture this context by appointing local student groups to record and post summaries of class discussions. However, the posting of summaries was often not timely enough to be of significant benefit.

Comparison of Two Learning Environments

Two unconventional data collection procedures, along with a conventional end-of-course evaluation, were used to explore and evaluate the two learning environments offered in SENG 611. Students e-mailed weekly learning logs and a final Biography of Learning to a distance education researcher (who took no part in the grading of student achievement in the course). Students recorded three types of information for their learning log:

1) reflections on the outcomes of lecture/sessions (main objectives, what was important new learning, what was confirmed, what changed for the student, value of topics discussed to student's work life or future career goals),
2) what worked particularly well, and what did not (according to preferred learning methods, individual learning styles or personality types), and
3) time spent on various course-related tasks, such as reading the textbook, writing responses, participating in project work and group meetings, reading and contributing to the course list server, and constructing and uploading HTML documents.
The learning logs proved to be a rich data source from which students could conduct a comprehensive review and analysis of their personal learning experience in SENG611. The Biography of Learning was a retrospective narrative account through which the student questioned and reflected upon their own educational biographies and learning experience throughout the course.

What Worked Well, and What Did Not

Both local and distance students regarded the list server discussions as an important extension of classroom discussions, and all students agreed that group work was a valuable component of the course. Students in the local section of the course were assigned the task of summarizing the in-class discussion and interaction, and posting this information on the course web page for the benefit of the distance students. Unfortunately, this did not work well for distance students because of the variable quality of the transcripts and the delayed posting times. Also, according to local students, summarizing the lecture reduced the quality of their participation because their attention was focused on "catching every word" rather than on contributing to discussion. Students criticized some technical aspects of the current experiment. In particular, unreliable accessibility to the course material, which was related to network access issues that were beyond our immediate control. Students also experienced difficulties generating and transferring their assignments as HTML documents; some lacked prior experience with HTML, and others encountered problems accessing the server.

Time Spent On Course-Related Tasks

Weekly time estimates for course-related tasks were collated for each of the five weekly learning logs. An aggregate summary of the total and average number of hours per week per task, and the average number of hours spent on each task per student is presented in Figure 1. Surprisingly, local and distance students spent a similar number of hours per week on average for course-related tasks. This finding contradicts the distance students' perception that they spent more time overall than local students on course-related requirements. An explanation for this difference in perception may be found in how local and distance students spent their time on various tasks (Figure 2).


Figure 1. Average Hours Per Task Per Student, Local versus Distance Cohorts

Figure 2. Percent of Total Time Local and Distance Students Spent on Various Tasks (including class time).

Distance students did spend more time on certain tasks; participating through email and the list server, creating and transferring HTML documents, and collaborating on group projects. Distance students reported they spent a great deal of time coordinating group work over the network, creating, exchanging and editing HTML files, and meeting for group work. Local students were able to meet before and after class for group work tasks. Distance students attempted to mediate some of these activities using the network; perhaps their initial unfamiliarity with this type of electronic exchange cost them additional time. Conversely, local students appeared to spend more time reading course materials. Given the additional amount of time required for the distance group exercises, the distance students reported that they read the absolute minimum to complete their work.

Improvements to Evaluation Procedures

The Learning Logs and Biography of Learning proved to be a successful data collection technique. Students provided rich and extensive qualitative and quantitative information about their participation in and perceptions of the course, types of tasks and time spent completing them, and reflections on their learning styles and preferences. Students were guaranteed that the learning logs and biographies of learning submitted to the researcher would not be made available to the instructors during the course, and that only group reports would be made available after the course. This step was taken to preserve the anonymity of students, and to promote an open and honest evaluation of the strengths and weaknesses of our distance model. This data collection procedure yielded 5 learning logs and 1 biography of learning per student, and with 21 students, the researcher's task included organizing, sorting, and recording 126 documents. Use of this data collection procedure in future courses will be greatly facilitated by employing a web-based, fill-in form that would automatically save student information to one file which can then be imported seamlessly into a statistical application for analysis. A web-based form will also provide a standardized format for incoming data, and enable the researcher to provide group responses to the instructor as an on-going, formative evaluation of the course. The course evaluation survey was administered using conventional paper-and-pencil forms. This data collection procedure has been automated by constructing a web-based form that students can use to submit their course evaluations on-line.

Technical Support of WebCT

The technical support requirements for SENG611 were concerned with two main issues: 1) supporting the organization and delivery of the course, and 2) minimizing the differences in student engagement. Although the course material already existed as web pages, WebCT was chosen as the web-based software tool to support the dissemination of course related information in this experiment. WebCT offered encapsulation and paths through the existing course content, student presentation areas for groups of students, individual student tools for note-making, and a chat facility for synchronous, online discussions. It was the preferred choice for a number of reasons, including free testing of WebCT with full functionality, inexpensive licenses, a large customer base, proximity of developers, ease of access via common web-browsers, and ease of customization (if needed).

In Spring 1998, WebCT was installed on a local server so we could experiment with many of its features. This led to enough confidence in WebCT for its use with the graduate course in Fall 1998. The technical support person and a course instructor were well versed in HTML, scripting, and other computer-related activities which influenced the evaluation and usage of WebCT. Based on the current experiment using WebCT, we have made the following observations:

  1. Support by developers of WebCT is very good. Queries and requests were dealt with quickly and to our satisfaction.
  2. It is easy to (a) install WebCT (beta 1.3 on Windows NT took slightly more time than 1.2 on Unix), (b) create courses as an administrator, and (c) encapsulate existing course material.
  3. It is moderately difficult to get used to all of the WebCT concepts and develop a mental map of the course structures, and to keep information up-to-date in a course, because the tool imposes a certain approach to structuring on-line course materials. One must become familiar with the affordances and constraints inherent in WebCT's design in order to construct well-organized course materials.
  4. It was difficult to customize WebCT to add connections to the external list server; Perl scripts had to be modified to add a mailto: hyperlink (referring to the list server) to WebCT's common button bar.
WebCT is geared to non-technical course designers. It was our experience that the WebCT designer interface often hampered the HTML-knowledgeable instructor's efficiency in building and maintaining the course material. For example, sequencing course materials requires the designer to set up a "path", which involves a number of intermediate steps that would not be required if the designer was using only HTML. However, for an instructor who is new to creating web-based documents, the structure and tool set offered by WebCT provide valuable supports as they construct course materials. We found that WebCT was not completely applicable to the current offering of the SENG611 course. However, based upon research that compared WebCT to other web-based course tools (e.g., Kristapiazzi, 1998), we expect that other web-based course tools are also not completely applicable and would require customization for SENG courses. Courses in the research-based M.Sc. program have been developed within an open architecture philosophy and the belief that there should be public access to past and present SENG course materials and student work. Inherent to the design of WebCT is a closed architecture philosophy -- courses are password protected, and information in courses is not easily accessible to the general public. WebCT's student tools, in particular note making and group presentation areas, offer useful tools for online courses. The e-mail and chat facilities in WebCT were not used for the current experiment; an external, publicly accessible list server remained in use. WebCT's quizzes, which employ selected response and open-ended questions, and so on, were not useful for this graduate-level course.

Future Investigations

As a result of what we have learned in this experiment, we plan to improve the design of our distance model in a number of ways. In this experiment we offered concurrent sections of the Requirements Engineering course for the purpose of directly comparing the two learning environments. In future, we plan to offer courses either at a distance or locally, but not concurrently. First, it is a conservative estimate to suggest that instructor workload is doubled because they are, in effect, teaching two courses to two different classes of students with different needs. Second, if the "distance" students are not truly at a distance (students in our graduate program tend to be located in Calgary), they tend to gravitate towards campus to meet anyway. It was difficult to keep the distance students at a distance to increase the authenticity of the experiment. We discovered that students in both sections wanted to meet face-to-face to work on assignments, and usually met on campus prior to and after the lecture. Third, our experience was that students felt somehow disadvantaged no matter which group they were in -- the local groups felt that allocating attention to creating a lecture summary reduced their opportunity to fully participate in the classroom discussion. The distance group felt they were missing vital information by not being present for classroom discussions and the face-to-face contact with instructors. Fourth, both local and distance students perceived that they spent more time per week on course-related activities, although the average time spent by each group per week was not significantly different. Our second experiment with a distance learning model will be with a graduate course in Design Patterns, and we will offer only a distance section for all students. The first and last lecture will be face-to-face, and all other classes will be at a distance. The course will be project-based; students will "present" their projects (as a web document), then lead web-based discussions about their work for the subsequent week. We will also experiment with real-time, synchronous class discussions at designated "lecture times" employing a computer-based conferencing tool with video, audio, and application sharing features. The final lecture of the course will be a face-to-face seminar, where students will present and discuss their work in a conference-like setting.

References

Goldberg, M.W., Salari, S. and Swoboda, P. (1996). World Wide Web Course Tool: An Environment for Building WWW-Based Courses. Computer Networks and ISDN Systems, 28. [On-line]. http://www.webct.com/webct/papers/p29/index.html

Harasim, L., Hiltz, S.R., Teles, L., & Turoff, M. (1997). Learning Networks: A Field Guide to Teaching and Learning Online. Cambridge, MA: MIT Press.

Jacobsen, D. M. (1998). Adoption patterns of faculty who integrate computer technology for teaching and learning in higher education. Proceedings of the ED-MEDIA AND ED-TELECOM 98: World Conference on Educational Multimedia and Hypermedia & World Conference on Educational Telecommunications, Freiburg, Germany, June 20-25. [On-line]. Available: http://www.acs.ucalgary.ca/~dmjacobs/phd/phd-results.html

Kristapiazzi, G. (1998). Compare Web Tools for Course Authoring. [On-line]. Available: http://www.geocities.com/Eureka/Gold/6012/compare_web_tools.htm
Shaw, M. L. G., & Gaines, B. (1998). A research-based masters program in the workplace. Proceedings of WCCCE'98: 3rd Western Canadian Conference on Computing Education. Addison Wesley Longman Publishing Company.[On-line]. Available: http://panache.cs.ubc.ca/wccce/program98/mildred/mildred.html



Last modified - June, 1999