Paper presented at WCCCE'99, Hosted by British Columbia Computers in Education Committee & Computer Science Department, Okanagan University College, Kelowna, BC, May 6 & 7, 1999. 

An Examination of Distance Education Learning Environments in Two Software Engineering Graduate Courses

Michele Jacobsen, Rob Kremer, and Mildred Shaw

Software Engineering Research Network (SERN)
247 MS, Department of Computer Science
2500 University Drive, N.W.
University of Calgary
Calgary, Alberta, Canada T2N 2N4

ABSTRACT

The Software Engineering Research Network has experimented with methods for delivering graduate software engineering courses at a distance. This paper describes our experiences with instruction, evaluating student learning environments, and technology tools and requirements. The methods used to prepare the courses for distance delivery are described. Several technological tools to facilitate computer mediated communication were used: WebCT, electronic mail, a list server, and NetMeeting.  Observations about workload requirements for teaching, support, instructional design, and maintenance are described for the benefit of faculty interested in moving towards distance delivery methods. The  methods used to evaluate the distance learning environments experienced by graduate students are outlined and explained. An analysis and report on the data and information collected from students and instructors is presented, and web-based data collection methods for educational and psychological research on distance learning are discussed. Finally, recommendations are made for future research and practice with distance learning environments.
Keywords: Distance Education, Graduate Software Engineering, Web-based Instruction


1 Introduction

The Software Engineering Research Network (SERN), created in 1996, is funded by industry and administered by the Industrial Research Chair in Software Engineering at the University of Calgary to support the dissemination of good practice in software engineering. One component of SERN's activities is a research-based masters program with a specialization in software engineering targeted at students with industrial experience in full-time employment (Shaw & Gaines, 1998). The MSc degree with a specialization in Software Engineering is offered jointly through the Department of Computer Science and the Department of Electrical and Computer Engineering, with funding provided by the Province of Alberta through the Access Fund. The software engineering program has dual objectives: to develop highly qualified personnel, and to encourage industry-based software engineering research with a focus on good practice. There are currently more than 30 graduate students working on SERN projects, and 6 completed research-based masters' degrees. A wide range of graduate courses in software engineering has been developed and taught, and a number of significant industry and university-based research projects have been initiated.

The program philosophy is based on experience with constructivist approaches to undergraduate courses over the last five years where all course material and student assignments have been made publicly available on the web. This has involved the restructuring of existing pedagogical approaches, de-emphasizing received wisdom and authority of the instructor, and emphasizing collaborative learning and meta-reflection on the learning process. We have seen evidence of the positive feedback processes that are the basis of major innovations in the way that our students are using resources provided by other students world-wide and are increasingly seeing themselves as contributing to those resources rather than merely 'taking a course.' Students who have graduated continue to return electronically to use the list server associated with a course, either because they have come across an interesting idea or technology to contribute, or because they wish to raise a question that might be answered within the class.  One of the goals of the MSc program is to make material available to software engineers and small companies who cannot afford the time or cost of formal graduate education. The industry sponsors agreed from the start that putting everything in the public domain, that is on the web, might help to achieve this end.

The required courses emphasize collaboration and cooperation with students working in groups to research a topic and present their findings to the others. They are continually changing groups for each assignment, and get to be familiar with many working styles and others' widely varying background experience. They are encouraged to take advantage of work done by others, acknowledging help where appropriate, and to submit their work on the web then revise it according to feedback in class and on the list until they feel that they have done as well as they can. This puts the emphasis on mastery of concepts and skills rather than "getting it right first time" which is the experience many have had in their earlier programs.

1.1 Increased Access to Post-Degree Continuous Learning in Information Technology

The University of Calgary supports the provision of increased opportunities for post-degree continuous learning at both the undergraduate and graduate levels to address the needs of increasing numbers of students who are in full-time employment while taking academic programs. A needs assessment conducted by the Calgary Research and Development Authority (CRDA) identified the need for higher education to develop strategies for increasing access to information technology and software engineering programs. Industry has also identified the need to increase opportunities for Alberta students to develop high technology skills in software engineering to meet current human resource demands. One of SERN's goals is to increase access to programs for students by expanding to include a distance model. SERN's graduate program is in high demand; therefore, a distance model will enable us to provide increased access to students who are also non-traditional, work full time, have industry experience, and often work internationally.  In support of the Campus Alberta initiative (Advanced Education and Career Development, 1998), a system-wide initiative that will build upon all Alberta institutions’ strengths and foster collaboration, a second goal of this distance project is to expand opportunities for all Alberta graduate students to access courses in masters programs. Our approach offers an effective alternative to classroom-based teaching and learning methods because the pedagogical requirements that guide the design of our courses are: 1) participation will be engaging and interactive for local and distance students in courses, and 2) differences between local and distance student participation in the course are minimized by providing a) the same course content, assignments and deadlines, b) access to in-class interactions (instructor & students).

Tom Russell, from North Carolina State University, has compiled a large bibliography of research that demonstrates that there is no significant difference related to the media used for distance learning. His well-known "no significant difference" study (i.e., a 50+ year retrospective analysis of the difference between distance and local education, available online at http://teleeducation.nb.ca/nosignificantdifference/) suggests that student learning is not disadvantaged by distance education models. We believe that computer mediated distance methods offer several learning advantages to students that go beyond "anytime, anywhere" (Daniel, 1997) access: 1) learning about networks and computers, 2) interaction and contact with experts and others outside of their immediate community or workplace setting, 3) increased self-reliance, and 4) developing independent learning approaches. The use of technology in teaching and learning has the potential to produce gains in productivity and efficiency. Although there is an initial increase in resources needed to research, develop and implement a distance model, which will be discussed in detail later in this paper, a distance education model can be cost-effective in the long run (Daniel, 1997; Bates, 1995; Rumble, 1997).

1.2 Distance Education in Software Engineering

Computer conferencing systems were first used for course activity and delivery in the early 1980s, and interest in educational networking has increased with the introduction of bulletin boards, sophisticated media for group communication, and interactive, web-based applications (Harasim, Hiltz, Teles, & Turoff, 1997). Over the last two decades, three basic approaches have emerged for educational applications of computer networks:
  1. course enhancements,
  2. on-line course delivery (which includes distance education and open learning), and
  3. knowledge networking using the Internet (Harasim, Hiltz, Teles, & Turoff, 1997).
Computer networks have always been used to enhance and extend the learning environment in the SERN graduate program.  All supporting course materials are already made available to the students through the web. The e-mail list server provides a forum for students to communicate with instructors and other students outside normal classroom or office hours, and students are required to submit assignments on the web. Instructors use e-mail and the web for distributing class outlines, supplementary notes, handouts, and instructions.

We are currently addressing the challenge of making the MSc program formally available to students using the second approach, on-line course delivery, while maintaining current levels of interaction and ongoing participation. First steps have included developing and offering parallel sections of SENG 611, a compulsory course in Requirements Engineering. One section of Requirements Engineering was delivered on campus, and the other delivered at a distance using facilities provided by WebCT. A second step was to offer SENG 609.04, an optional course in Software Design Patterns, as a distance education course that was supported with WebCT and weekly chats using NetMeeting.

Trialability is the degree to which an innovation may be experimented with on a limited basis (Rogers, 1995), and represents less uncertainty to individuals who are considering it for adoption. We chose to conduct our first distance education experiments with graduate rather than undergraduate students because class sizes are smaller, and graduate students are less "grade-driven" and more independent in their approach to learning, and as a result, tend to be more forgiving of the fits and starts that tend to accompany changed learning environments. Thus, although on-line course delivery approaches and software tools were novel, we deliberately chose a group that would have lower levels of uncertainty with this type of experiment. It is important to note that all of our graduate students were local, and had convenient access to the instructor and other students at all times.

This paper reports on our experience to date with these two distance education graduate courses in software engineering.  Described first are the methods used to prepare the courses for distance delivery. This work experimented with several technological tools to facilitate computer mediated communication: WebCT, electronic mail, a list server, and NetMeeting. Second, the paper outlines and explains the methods used to evaluate the distance learning environments experienced by the graduate students. An analysis and report on the information collected is presented. Web-based data collection methods for educational and psychological research on distance learning are discussed. Finally, recommendations for future research and educational practice in distance learning environments are summarized.

2 Instructional Design and Delivery of Two Graduate Courses

2.1 Overview of Instructional Environment

Both courses that were offered at a distance are part of the MSc program in Software Engineering and students are normally required to have at least one year industrial experience before enrolling in the program. Courses in the program have always been unconventional in that they promote collaboration, rather than competition, which is intended to reflect and leverage students' industrial experience. After initial background lectures, students take over much of the responsibility for presentations on various topics in each course. The instructors' role is that of facilitator managing a process of debate and exploration rather than attempting to be an authority in the domain. Students work on individual assignments, but there is always a requirement that students complete group projects as well. Group work requires that students develop and refine teamwork and negotiation skills as they plan, develop and complete projects.

The overall aim of the graduate program is to provide a collaborative learning environment in which experience and knowledge can be constructed and shared, ignorance displayed and errors made without censure and with ready access to diagnostic help. It becomes clear to students that, while there are no easy answers to the core questions of industrial practice in software engineering, there are many useful perspectives and that simplistic, surface level answers and approaches generally have very limited applicability. Several resources distinguish the program from more traditional programs:

The latter two features have made it possible for students whose companies relocate them, or who change jobs to companies in other locations world wide, to continue to participate in the courses. Remote students who have already come to know their colleagues can continue to present and share material through the web and participate in discussion through the list server.

2.2 Instructional Design and Delivery of Two Courses at a Distance

The pedagogical requirements for the redesign of both courses included the following: Prior research has found that faculty spend more time preparing materials and resources for instruction when integrating computers into their teaching and learning on campus (Jacobsen, 1998). Instructor workload increases, at least initially, for teaching and learning on-line (Harasim, Hiltz, Teles, & Turoff, 1997) but tends to approach more normal levels of workload as the instructor gains experience in the online environment. The results from these experiments are certainly consistent with these findings. Although it seemed these courses lent themselves to on-line delivery (i.e., extensive web-based notes and links, information about assignments, requirements to publish on the web), several components required modification. To meet student needs in concurrent local and distance sections of the first course, instructor workload increased. The learning environment in the Requirements Engineering course has always included two instructors.  In past offerings of the course, the instructors shared equal teaching roles in the classroom. However, to offer both a local and distance section of the course, the instructor's roles had to be modified slightly to allow one instructor to coordinate the distance half of the course, and one to coordinate the local offering on campus.  The instructors spent additional time on the following distance-related tasks: More time and energy was spent fielding email and telephone inquiries from distance students than would have been spent for these questions in an exclusively face-to-face offerings of the courses. Time spent on these tasks was over and above the time required for conventional classroom instruction. Complete course notes already existed as web documents, but these had to be modified and expanded considerably to be useful for the distance students. The web based documents consisted mainly of outlined text and visuals that would support in-class lectures and discussion.

All of these extensions to the courses took a considerable amount of time, and would not have been possible without additional help with these projects, given that one instructor was also teaching other courses and involved in other research activities, while the other is a full-time Software Engineering Manager in local industry.  A post-doctoral research fellow provided assistance and support with the repurposing of course content for distance delivery. The post-doctoral assistant also provided technical assistance in investigating features of WebCT, adapting the WebCT scripts as needed, and coordinating with the WebCT developers. Evaluation of the learning environments for both courses was done by a post-doctoral researcher (with a background in Educational Psychology) who was not involved in student evaluation, and protected student anonymity with respect to information gathered throughout the project. A graduate student repurposed the original web notes for the second course and posted these in the WebCT environment, and also participated in the course as an informal peer-advisor. A network manager was available at all times to address technical concerns, install and implement new software, and react to technological breakdowns and failures. Although these first experiments with distance offerings of two courses demanded a large infusion of time and additional support, from the instructors and other assistants, future offerings of the courses will require less time and effort. Learned lessons and hands-on experience with distance education has provided insight into refining and improving the processes.

As a result of outcomes from the first experiment, we improved the design of our distance model in a number of ways. Concurrent sections of the Requirements Engineering course were offered for the purpose of directly comparing the two learning environments. The Design Patterns course was offered only as a distance section rather than offering concurrent distance and local sections.  This was done for two reasons: 1) it is a conservative estimate to suggest that instructor workload was doubled because they were, in effect, teaching two courses to two different classes of students with different needs, and 2) if the "distance" students are not truly at a distance (students in our graduate program tend to be located in Calgary), they tend to gravitate towards campus to meet anyway. It was difficult to keep the distance students in the first course at a distance to increase the authenticity of the experiment. It was discovered that students in both sections wanted to meet face-to-face to work on assignments, and usually met on campus prior to and after the lecture.

A third reason for offering only a distance section of the Design Patterns course was that students in the first experiment felt somehow disadvantaged no matter which group they were in -- the local groups felt that allocating attention to creating a lecture summary reduced their opportunity to fully participate in the classroom discussion. The distance group felt they were missing vital information by not being present for classroom discussions and the face-to-face contact with instructors. Students in the local section of the course were assigned the task of summarizing the in-class discussion and interaction, and posting this information on the course web page for the benefit of the distance students. Unfortunately, this did not work well for distance students because of the variable quality of the transcripts and the delayed posting times. Also, according to local students, summarizing the lecture reduced the quality of their participation because their attention was focused on "catching every word" rather than on contributing to discussion. Finally, both local and distance students perceived that they spent more time per week on course-related activities, although the average time spent by each group per week was not significantly different (see Figure 5). Thus, it was decided to minimize perceived equity problems by offering only a distance section of the second course.

For the Design Patterns course, the first and last lecture were face-to-face, and other classes were at a distance. The course was project-based; students "presented" their projects on the web, then led web-based discussions about their work for the subsequent week. NetMeeting, a computer-based conferencing tool with video, audio, and application sharing features, was used to  experiment with real-time, synchronous class discussions at designated "lecture times" employing . The final lecture of the course was a face-to-face seminar, where students presented and discussed their work in a conference-like setting.

2.3 Technology Tools

The experiments employed several technological tools to facilitate on-line course delivery: WebCT, electronic mail, a list server, and Microsoft NetMeeting. WebCT is a web server designed to support on-line, web-based instruction (Goldberg, Salari, and Swoboda, 1996). Faculty with little or no prior experience with HTML will find the inherent structure and design of WebCT to be useful starting points for creating online documents. The tool sets offered by WebCT provide valuable entry-level supports and templates as one creates course materials and online examinations. The technical support requirements for the first course were concerned with two main issues: 1) supporting the organization and delivery of the course, and 2) minimizing the differences in student engagement. Although the course material already existed as web pages, WebCT was chosen as the web-based software tool to support the dissemination of course related information in this experiment. WebCT offeres encapsulation and paths through the existing course content, student presentation areas for groups of students, individual student tools for note-making, and a chat facility for synchronous, online discussions. It was the preferred choice for a number of reasons, including free testing of WebCT with full functionality, inexpensive licenses, a large customer base, proximity of developers, ease of access via common web-browsers, and ease of customization (if needed).

An extensive summary of our evaluation of WebCTs features is available online (Jacobsen, Wijngaards, Kremer, Shaw and Gaines, 1999). In brief, WebCT is designed for non-technical faculty who are new to creating web-based documents. We found that the WebCT designer interface often hampered the HTML-knowledgeable instructor's efficiency in building and maintaining the course material. For example, sequencing course materials requires the designer to set up a "path", which involves a number of intermediate steps that would not be required if the designer was using only HTML. However, for an instructor who is new to creating web-based documents, the structure and tool set offered by WebCT provide valuable supports as they construct course materials.

It was found that WebCT was not completely applicable to the current offerings of the two graduate courses. However, based upon research that compared WebCT to other web-based course tools (e.g., Kristapiazzi, 1998), we expect that other web-based course tools are also not completely applicable and would also require customization for these courses. Courses in the research-based M.Sc. program have been developed within an open architecture philosophy and the belief that there should be public access to past and present course materials and student work. Inherent to the design of WebCT (and most web-based distance education tools) is a closed architecture philosophy -- courses are password protected, and information in courses is not easily accessible to the general public. WebCT's student tools, in particular note-making and group presentation areas, offer useful tools for online courses.  The email and chat facilities in WebCT were not used for either course; an external, publicly accessible list server remained in use. The Software Engineering courses share a common list server that allows the students to enjoy input from students in other courses, former students and industry partners who subscribe to the list. An advantage of the list server is that each message is delivered to subscribers (push technology), as opposed to the student having to seek out the postings of others by accessing a separate web page "discussion list", threaded discussion or newsgroup, or bulletin board (pull technology). WebCT's quizzes, which employ selected response and open-ended questions, and so on, were not useful for graduate-level courses. For a brief review of the efficacy of WebCT's online testing tools for undergraduate computer science, see Jacobsen, Kremer, and Flores (1999).

Microsoft NetMeeting was used as a substitute for face-to-face class discussions. The instructor and students participated in weekly discussions about course topics using a version of NetMeeting running on a local server. The potential of this on-line chat environment was not fully realized in this course. For example, at the end of a sixty or ninety minute session, the instructor observed that very little material was actually covered. The level of interaction and discussion productivity did improve during the course as students adapted to the limits of the technology.  Although it offers a multimedia discussion environment using audio and video, NetMeeting offers little more than a standard text-based chat tool as it allows only point-to-point audio and video and its shared drawing tool was found to be inadequate. The one thing that NetMeeting does add over a text-based chat tool is shared application viewing, which some students used to great advantage in making network "presentations" using PowerPoint. Other factors that seemed to limit the efficacy of NetMeeting chats were:

It seems possible to refine and improve the use of NetMeeting for interactive, real-time discussions for an on-line course by correcting the identified deficiencies (i.e., establish discussion rules, appoint weekly moderators, and upgrading software).

The list server proved to be an important communications element in the first course, but appeared to be under-used in the second course.  For both courses, the instructors were participant observers on the list server, rather than active and directive moderators (except when students appeared to be involved in a non-productive debate or required expert advice). The instructors assessed list server participation during the first course, and found the sheer volume of discussion was greater than in previous, classroom-only offerings of the course. There are a number of possible explanations for this trend. First, a large proportion of the local students speak and write English as their second language, and seemed to prefer to participate more in the list server than in the lectures. It may be that these students are more confident when they have time to reflect and respond to a written discussion, rather than speaking out in class. Second, distance students actively participated in list server discussions, possibly because they were not present for the classroom discussions. A third possibility is that students actively participated because this was a graded expectation for the course, although this expectation was unchanged from previous classroom-only offerings of the course.

Participation in list server discussions was not as active during the second course. A possible explanation for this result is that attention may have been diverted from the list server as a result of the emphasis on real-time weekly interaction using NetMeeting. Students may have saved their questions for the weekly chat sessions rather than posting them to the list server. As a result of the experiments, we have concluded that the list server offers several advantages over the weekly NetMeetings including:

There are some drawbacks to the current implementation of the list server when comparing it to computer-mediated conferencing software. For example, the messages are not currently archived for review and revisiting. The transcripts of each NetMeeting discussion were saved as an archival record. The list server discussion is not threaded, and there can be a large number of messages per day which can be time consuming because it is difficult to follow and participate in every thread/topic. Future refinements will include upgrading the list server software to facilitate threading and archiving of messages.

2.4  Pedagogical Outcomes

The instructors believe that the quality of assignments and amount of effort invested in participatory activities varied much more in the distance than in the local groups. One possible explanation is that the face-to-face interaction among and between students and instructors in the classroom allowed them to arrive at a common understanding of expectations. This finding suggests to us that instructors must find a way to be more explicit about discussing and arriving at a common understanding of expectations with their distance students. It is our hope that we find an innovative way to achieve this goal without stifling the creativity and individual style of a student's knowledge construction. A related finding, which may be an artifact of this comparative study, was the instructors' belief that distance students' contributions to the list server varied more than discussion by local students. Again, a possible explanation for this difference is that distance students were participating in the course without the benefit of the shared context created in collective face-to-face discussion and interaction. We attempted to capture this context by appointing local student groups to record and post summaries of class discussions. However, the posting of summaries was often not timely enough to be of significant benefit.


Figure 1. Example of One Student's Web Page and Links to Coursework


 






By the second course, students were accustomed to the expectation that they post all of their assignments and coursework on the web, and reported very few difficulties with achieving this requirement. An example of one student's course web page with links to their coursework is presented in Figure 1. An example from a student's work for their textual and graphical presentation on a design pattern is presented in Figure 2. Finally, Figure 3 provides an example of a different student's presentation materials posted on the web.
 
 


Figure 2. Example of Student Assignment on Design Patterns

Figure 3. Example of Student Presentation Materials on the Web


 



3 Evaluation of Distance Learning Environments

3.1 Methodology

The evaluation goals in this project were to describe the nature of the educational experiences and outcomes in the distance learning environments, in the first instance to compare the distance learning environment to the traditional classroom, and determine those conditions associated with what worked well and what did not. A quantitative and qualitative analysis of information collected from students, both throughout and at the completion of each course, is presented and discussed.

Two unconventional data collection procedures, along with a conventional end-of-course evaluation, were used to explore and evaluate the distance learning environments offered to students in the two courses. Students emailed weekly learning logs and a final biography of learning to a distance education researcher who took no part in the grading of student achievement in the course. Students recorded three types of information for their learning log:

The Biography of Learning is a retrospective narrative account through which the student questions and reflects upon their educational biographies and learning experience throughout the course. The learning logs proved to be a rich data source from which students could conduct a comprehensive review and analysis of their personal learning experience in the course.

The use of open-ended questions allowed students provide feedback in their own words on the nature of their learning experience. The questions provided an opportunity for students to highlight any ideas, concerns, problems, or suggestions which they considered of relevance to teaching and learning at a distance. Student responses were condensed to reflect emergent categories and themes, and subsequent reporting includes verbatim selections from the source data.

3.2 Findings

The student workload cost to complete the learning log task was at least 0.5 hours per week, with some students reporting they spent 1.0 hours over the week writing their learning log. The information collected through use of the Learning Log is a defensible and credible account of the student learning experience because instead of inferring complex learning processes and tasks from server log files (i.e., logged time = work time, or learning time), this technique asks the learner to estimate and record the time it takes to complete various learning activities, tasks and activities to do with the course, reflect and report on what worked and what did not, and summarize weekly learning outcomes. The students define the tasks and the times and the experiences that are for some reason memorable or notable to them.

When using this data collection technique, it is important to keep this task open-ended rather than deriving categories or imposing structure on student's time reporting (e.g., by setting up form-fill-in web pages with defined categories for response). With preset categories, one risks that students will only attend to the amount of time it takes for the identified tasks, rather than constructing a genuine account of time spent on tasks (many of which may not have been anticipated by the researchers).

3.2.1  Reflections on Weekly Learning Outcomes

Students were prompted to reflect on the main objectives of each session, the important new learning, what information or concepts were confirmed for them, what changed for them as a result of the session, and the value of topics discussed to the student's work life or future career goals. The following student, new to the program, gives us insight into how the graduate program is viewed at first glance:
"This is the first time that I am taking a class specialized in professional computer industry.  You can understand that there are lots of excitement and yet also lots of strange terminology as well.  I was surprised that two chapters of the textbook must be summarized due next week.  This is going to be a fresh experience for me since I have been working as a mechanical engineer and most of the time I spent has been on design and calculations, rarely I have spent time on book summarization. I will have to cultivate my fast reading skill in order not to fall out the class".
Students would often use the forum offered by the weekly learning log to reflect upon their learning discoveries and comment on the value of topics investigated throughout the course.
"During the reading assignments and writing the paper, I found that classifying design patterns depending on their relationships with other patterns can help designers to select patterns for solving problems. This is interesting because I think learning how to use patterns is more important than just knowing what patterns are. Writing this paper makes me think differently and I start to wonder about why people are using different schemes to classify the design patterns".
It is a rare opportunity to get insight into the thinking and discovering that accompanies a reading and writing assignment in graduate school. We will discuss later how students viewed the Learning Logs and the Biography of Learning as valuable opportunities both for increased self-awareness as a learner, and to reflect upon their own learning processes as they worked through the course.
"The main objective in the week #4 was implementation of the selected pattern and a class presentation. Since my pattern is hard to implement, I wrote a paper on it. It required lots of reading, and from reading references, I learned more. I also spent a lot of time preparing for the presentation. The presentation went well, except that I had some problems with presentation skills. I need more practice to improve that skill. I also learned something from other students' course work. Since I didn't do programming, it is good to see patterns in code from other people's work"
This student was not required to write a paper on the design pattern but made an independent learning decision to actively seek out more information, and organize this into a form that was appropriate for their peers. The student identifies an individual learning goal related to presentations, and outlines the benefits of participating in other students' presentations.

Students were given the option to participate in either the local or distance offering of the first course. This student was motivated in their choice of the local section, not because of their philosophical beliefs about learning environments, but as a result of not having the necessary computer equipment for participation.

"The decision to take the course by attending the class rather than distance group was motivated by the fact that I don't have computing facilities at home".
Planners must take into consideration the requirements, both hidden and visible, for students to be able to access a distance program. Though this is a graduate program in software engineering, we cannot assume that all students have adequate computing facilities at home that will enable them to consider online, or web-based, instruction. Although many students work at home, the distance model was originally designed to be useful to students who are employed full time, to increase accessibility for students working in industry. One way to describe the interaction that we hoped to facilitate is the "coffee-break model", whereby students access course materials during natural or scheduled breaks in their work day. Instead of trying to schedule large blocks of time for coursework, the student learns to accomplish certain tasks and goals in shorter periods that contribute to larger goals.

3.2.2  What Worked Well, and What Did Not

Students were asked to comment on aspects of the weekly session that worked particularly well for them as a learner, and what did not work well. Because the students had completed the Myers-Briggs personality inventory as part of group building exercise in the Requirements Engineering course, students were also asked to consider whether there were aspects of the session that appealed to them because of their preferred learning methods, individual learning styles or personality types.

Both local and distance students regarded the list server discussions as an important extension of classroom discussions, and all students agreed that group work was a valuable component of the course. Students criticized some technical aspects of the current experiment. In particular, unreliable accessibility to the course material, which was related to network access issues that were beyond the instructors' immediate control. Students also experienced difficulties generating and translating their assignments as HTML documents; some lacked prior experience with HTML, and others encountered problems accessing the server.

"Writing summaries forces me to read and understand the material. Course participation forces me to think deeply about the material and come up with questions and answers. I had problems posting my summaries on WebCT. A peer helped me. It was not clear where to post my stuff. This was not clearly stated in the course pages".
The weekly learning logs were a valuable formative evaluation of the course that enabled us to learn about problems, omissions, concerns, and student needs as they occurred, rather than waiting to find out from a summative, end of course evaluation. As a result of the weekly feedback from each student, we were able to refine and improve the course materials on WebCT, respond to technical difficulties, and address concerns immediately, which enabled us to better meet student needs.

Using NetMeeting in the second course for weekly discussions was not always successful. The instructor was online for each session, and at times it proved to be a challenge to sustain interest and participation in certain topics. On the other hand, some topics really stimulated active and profuse discussion, which led to rapid fire comments appearing on the screen. The active discussions in NetMeeting were hard to follow as several responses to a question would appear at once, which in turn set off another round of responses and new questions.

"Based on the online chat we had on Tuesday afternoon, I find that the discussion through email is more fruitful than the chat room (i.e., NetMeeting). Basically, I found that during chat session, there were many topics discussed at the same time, and focusing (or following) the topics at the same time were rather difficult. I think we have to find a way to moderate the discussions. When a topic starts I think everybody should focus on that until it concludes, then the next topic starts".

"This week the discussion was not very productive. First of there were fewer people than there normally are. Secondly, the discussion was not on topic. I prefer that the discussion remains focussed on one topic before moving on to the next one. Hopefully, this will work better next time".

Students made a number of suggestions about how to improve the organization of discussions using NetMeeting. First, generate a list of discussion topics before the meeting using the list server to elicit suggestions during the week. An student-constructed agenda may reduce the time lags in participation in the meeting. Second, assign a student moderator each week who will pose questions during lags, attempt to focus the group discussion, and guide participants through the agenda items. A student moderator may help to reduce the number of items being discussed at any point in time, and because each student would have an opportunity to participate in this role, the whole group may develop useful strategies to facilitate discussions using this medium.

In both courses, students were required to post all of their coursework online using HTML and WebCT.

"Writing web notes is a good learning experience. Because I was not familiar with web page posting and editing, I spent a lot of time on web page construction. I also had some problem in uploading the notes. I lost all of my images. But I finally got it fixed".
Although students were keen to experiment with the distance model, a number of them were relieved to gather together at the end of the course for the group presentations and class discussions.
"It was nice being back in the classroom and being able to interact normally".

"I enjoyed being back in a traditional classroom setting.  I still feel that that is the best learning environment for me".

It is becoming increasingly difficult to identify what is a "normal" learning environment given the growing diversity of the student body. We no longer have traditional students who start post-secondary programs in their late teens right after high school.  Students are starting or returning to post-secondary institutions who are all ages, have all types of background experiences, and have different expectations and needs. Institutions need to be flexible in the programs they offer, when they offer these, and the methods they use to construct learning environments, both local and distance. All learners are different. Some students work full time, and others are at home with young children, and can only attend classes in the evening. Others prefer to be on campus full time and taking classes during the day. Some people enjoy coming to class, and feel that they benefit most from discussions with other students. Other students enjoy or prefer working on their own, in short intervals, and rely less on the groupings and class gatherings for their learning and interaction. It may be that some of our students have only experienced the traditional classroom-based campus learning experience, and are tentative and unsure about the suitability of other learning environments based upon this learned preference.

3.2.3  Time Spent On Course-Related Tasks

In the first course, student estimates of time, and descriptions of tasks, varied. As such, several categories were created to organize and summarize results. Tasks included in the “Writing” category include local groups’ summaries of the content of lecture interaction and discussion, writing the weekly learning log, completing group reports, and writing individual assignments (excluding the chapter summaries). The “Chapter Summaries” were a graded assignment, and time spent on this task was given its own category. Tasks included in “Reading Course Material” included reading the course textbook, class notes, and downloaded materials. On-line research tasks included accessing materials via WebCT and searching for materials related to course content on the World Wide Web. Time estimates for both HTML and FTP tasks were summarized by one category. All time estimates for group work, regardless of whether this was a face-to-face meeting or virtual meeting, were recorded in one “group work” category.

Weekly time estimates for course-related tasks were collated for each of the five weekly learning logs. An aggregate summary of the total and average number of hours per week per task, and the average number of hours spent on each task per student is presented in Figure 4.
 
 



Figure 4. SENG 611 - Percentage of Time Spent Per Task Per Week Per Student (Local vs. Distance Cohorts)


 






Local students reported spending a larger proportion of their time in the classroom, reading course material, and conducting on-line research and reading documents online. Distance students report spending slightly more time on chapter summaries, and a large proportion of their time interacting using email and the listserver, constructing and transfering HTML documents, and meeting with their group to work on projects.

Surprisingly, local and distance students spent a similar number of hours per week on average for all course-related tasks (Figure 5). This finding contradicts the distance students' perception that they spent more time overall than local students on course-related requirements. An explanation for this difference in perception may be found in how local and distance students spent their time on various tasks (Figure 5).
 
 



Figure 5. SENG 611 Percent of Total Time Local and Distance Students Spent on Various Tasks (including class time).


 








Because we only offered a distance section of the Design Patterns course, Figure 6 presents a summary of the average number of hours spent on the course each week by each student. The average number of hours remains fairly stable for the first three weeks of the course, with students reporting an average of 12.9 to 14.1 hours spent each week. As the course moves into the fourth and fifth week, students report spending much more time, from 19.6 to 22.5 hours on average per week. A ready explanation for this result is the additional time allocated to preparing their final project for public presentation during the sixth week. Students report spending an average of 13.1 hours on the course during the final week of classes.
 



Figure 6. SENG 609.04 Average Number of Hours Spent On Course Per Student Per Week (including class time)


 


In ways that are similar to the first experiment, student estimates of time spent and descriptions of tasks varied. As such, several categories were created to organize and summarize results from the second course (Figure 7). Reading the text and WWW documents represents an average of all reported hours spend on required and elective reading time reported for both media. The average of 5.3 hours per week spent browsing and creating web pages includes the construction of HTML documents, reported hours spend searching and browsing for research documents relevant to their final projects or weekly readings, and time spent accessing course materials on WebCT. The average 1.1 hours spent on writing tasks include preparing their learning log and writing summaries of required readings. Because students were required to post all of their coursework on the web, the time spent for writing each week is probably better captured by the "creating web pages" task than the conventional writing task. Most students created their original documents as HTML rather than word processing first and converting later. "Prototyping/code/implentation pattern" arose as a category to record the average 6.4 hours of time students reported spending per week on their design patterns and prototyping and coding a variety of programs. The task that students reported spending the most time on was presentation preparation, which includes research, writing, and preparing visuals for their public presentation. The average reported time was 9.2 hours per week, but this figure is best understood using individual numbers to get a perspective on how students allocated their time. Students had to prepare presentations for different weeks throughout the course, and again for the seminar at the end of the course. One student spent 12 hours on presentation preparation in the second week, and another 8 hours in week 5. Another student spent 10 hours in week 3 and 7.5 hours in week 5. A third student reported spending 27 hours in the first week.
 
 


Figure 7. SENG 609.04 Average Time Per Task Per Student Per Week (including class time)


 






3.2.4  Biography of Learning

The Biography of Learning serves as a retrospective analysis and narrative account of each student's learning experience throughout the course. As such, the Biography of Learning proved to be a valuable data collection technique, both for research purposes and to the student, in providing insight into the individual's overall learning experience. The very act of reflecting on one's learning processes and strategies in a regular way, then recording and analyzing this information in a learning log, provided an opportunity for students to learn more about themselves as learners. The Biography of Learning required students to revisit their entire learning experience, and draw some conclusions about their experience in the course.

A number of students described how writing weekly Learning Logs and a final Biography of Learning raised their awareness about self as learner because of ongoing attention to how much time they were spending on various tasks, what was working and what was not, how many tasks, type of engagement in course, and so on. Students commented on the benefits from time invested in completing the Learning Logs and the Biography of Learning. Many students observed that the requirement to take time to reflect upon their learning processes as they worked through the course created a valuable opportunity to become more self-aware as a learner and direct attention to habits and patterns of tasks and time spent on learning activities.

"Forcing myself to look back on my impressions of the lectures and the time I spent on the different activities was revealing.  Normally, I attend lectures and absorb what I can. By looking back in a more structured way, I have probably absorbed a little extra.  If I were a more organized person, I would have also been better able to manage my time.  It may just help me become more organized in the future".
Some students commented on specific aspects of the course that were beneficial, such as using the Myers-Briggs as a technique to better understand self as learner and potential contributions and roles in group work. Others submitted positive descriptions about the value of group work to their learning.
"Since there was a lot of group work, I was fortunate to be in a group that worked well together.  Each person was more than willing to do their share which helps create a feeling of good will between the group members. For the most part, I thought the text was very useful and I'm sure that I'll refer to it many times in my career.  I enjoyed critiquing the text and found that the process helped a great deal in my understanding".
Some students used the Biography of Learning as a forum to reflect on their fluctuating satisfaction with the technologies used in the course, and provided summative views on the efficacy of certain types of interaction. The following two comments provide examples of the range of reactions to the distance technologies.
"As I signed up for SENG609.04 I was very excited to begin the course.  However, through the course my excitement has withered.  I personally found the NetMeetings were an inappropriate medium for discussing complex issues.  I am still open minded towards distance learning through tools similar to videoconferencing.  But NetMeeting’s awkward audio/video interface and comparably slow typing versus speaking made the NetMeeting’s largely unproductive when compared to a real meeting".
"I learned a lot by the new method of distance learning that was employed in this course. At times the immaturity of this new technology (NetMeeting) and its users (us) showed up. As the course proceeded, we could make better use of the tools we had at our disposal, and the course was even more fun... There was a lot of research involved and a lot of material had to be read in a relatively short period of time... the instructor was able to devote more time to individual students. Overall I spent more time on this course than any other course this semester. I spent a total of around 110 hours during the five weeks of this course".
As instructors interested in refining and improving the learning environments for students, it was instructive to have access to student reflections on what they believed to be important to their experience in a particular course. The following student provides insight into how time gets prioritized and allocated for projects that will be presented to peers.
"Web notes preparation was an important new learning in this course. Because I thought I had to know more to be a class expert on my pattern and to answer other peoples' questions, I spent a lot of time on that, and I found I learned more from reading references and writing web notes than I did from just writing summaries. I also learned some other patterns that are related to my pattern, on which I don't think I would spend time if I just wanted to write a summary".
Breaking with traditional post-secondary models, in which student assignments and projects are submitted to and read by only one person, is one way to stimulate and promote exemplary work. When students know that their assignments and projects will be examined by their colleagues and peers, they invest more time and effort to do their best work. The Biography of Learning provided insight into student thinking and planning for public presentations and displays of their work.

3.3  Web-based Data Collection Methods for Educational and Psychological Research on Distance Learning

The two data collection methods used to gather information from students proved to be very successful. Students provided rich and extensive qualitative and quantitative information about their learning experiences in the two courses. In the first course, students submitted their learning logs and biography of learning using e-mail. This data collection procedure yielded 5 learning logs and 1 biography of learning per student, and with 21 students, the researcher's task included organizing, sorting, and recording 126 documents. Use of this data collection procedure in the second course was greatly facilitated by employing a web-based, fill-in form that automatically saved student information to individual files which were imported seamlessly into a statistical application for analysis. The 7 students in the second course submitted 42 learning logs and 7 biographies of learning which took a fraction of the time to sort, organize and analyze.

Many areas of inquiry can benefit by extending their data collection activities to take advantage of an Internet connection (Mueller, 1997). The use of a web-based interface to collect information for educational research offers several advantages over traditional methods. First, web-based forms save the expense and delay of regular postage and paper-based distribution methods. If sampling error is a concern, researchers have the flexibility to collect data using a local area network configuration (LAN), rather than making the instrument available to "the world" on the Internet. In the present experiment, the learning log form and exit survey were published on the World Wide Web and students were provided with links to the relevant documents along with due dates. Second, instead of using manual or scantron methods for data entry, the data collected using a web-based interface are entered by students and are stored in a cumulative data file that is ready for analysis with SPSS or SAS software. The web-based form reduced the time spent sorting and organizing the student-provided data sets by more than half, which in turn meant the instructor got the anonymous formative evaluations very quickly each week. Third, because the Internet is available to anyone with a computer and a connection, at anytime and from anywhere, students can submit information 24 hours per day without the researcher having to be physically or temporally present and without having to procure laboratory space or photocopy questionnaires.

Anonymous participation was important and was guaranteed to students. In general, when results are submitted using a web-based, HTML-generated response form, the cumulative data file lists the date and time the results were submitted and the IP address of the server from which the survey was completed in a header. For each learning log and the biography of learning, students were asked to include their name so the researcher could keep track. This information was used only to confirm that students had submitted weekly logs, and the instructor was only informed whether or not students were up-to-date with their assignments.

4. Recommendations Based Upon Lessons Learned

We will continue to refine and improve our methods for offering courses at a distance. In the form of recommendations for research methods and educational practice, we present some lessons learned from the experiments with a distance education model.

Mechanisms and support systems need to be put in place that allow for cyclical and iterative development and assessment of distance learning environments. Tolerance has to be build into the system for time lags, unexpected barriers, and longer course development periods. Distance education environments and new technological tools constitute a major change in some people’s lives. Such change does not happen quickly or easily. Even in the best of circumstances, teachers and students need high levels of support, training, and access to technology. The instructors for both courses spent a significant amount of additional time on the distance courses than would have been required for classroom-based courses. In addition to the instructors, several individuals provided on-going technical support, instructional design and development, and evaluation and assessment for the distance courses. Therefore, it is recommended that more instructor/design time be allocated for initial distance course offerings. Technical support and help with the technology will be needed if the instructor is not completely familiar with the tools used to support on-line course delivery.

While computer networking can open up possibilities for teaching and learning at a distance, care should be taken to make sure that the chosen technology fits the core values and goals of the program, and not the other way around. The technology tools used in the present experiment were evaluated based upon the fit between core learning objectives and program requirements, rather than designing courses based upon what affordances and constraints were inherent in the technology. For example, a number of WebCT's features and built-in tools were not applicable to graduate courses, and therefore were not used.

Distance students require very explicit, well-organized, and well indexed web materials and resources. It is important to structure carefully the entire course before it starts and to be faithful to that structure throughout the course to provide distance students with a stable context in which to learn. While this finding may seem rather obvious from an instructional design standpoint, it is not the typical approach in SERN's graduate course offerings. Instead, flexibility to change course material and follow "excursions", as dictated by student interests, experiences, and current situations, is favored over rigid structure. This constructivist approach that puts the subject and the student at the center may be at odds with the need for more structure in a distance model. This is worrisome because the more open-ended and responsive approach is highly valued by both students and instructors. We are currently investigating instructional strategies that provide a balance between carefully structured and defined course design, and responsive, flexible learning environments. It may be that our distance students felt they needed more structure because the distance model of learning was new to them. The distance students monitored what the local students were doing in the course, and were fearful of falling behind or missing important information. We will continue to evaluate student learning experiences and concerns with subsequent distance course offerings.

Acknowledgments

Financial assistance for this work has been made available by the Industrial Research Chair in Software Engineering, and the Government of Alberta's Learning Enhancement Envelope. Thanks are also due to the classes of 1998/99 for their participation in the experiments and the examples used to illustrate this article.

Internet Access

The authors are available at dmjacobs@ucalgary.ca | kremer@cpsc.ucalgary.ca and mildred@cpsc.ucalgary.ca

Further information about SERN can be found at its web site - http://sern.ucalgary.ca/

Additional information about these courses, including examples of student coursework, is available from the course pages at SERN:

References

  1. Advanced Education and Career Development (AECD). (1998). 1997/98 Annual Report. [On-line]. Available: http://www.aecd.gov.ab.ca/department/publications/annual_reports/1997/
  2. Bates, A. W. (1995). Technology, Open Learning and Distance Education. Routledge Studies in Distance Educaton. London, UK: Routledge Press.
  3. Daniel, J. S. (1997). Why universities need technology strategies. Change, 29 (4), 11-17.
  4. Goldberg, M.W., Salari, S. and Swoboda, P. (1996). World Wide Web Course Tool: An Environment for Building WWW-Based Courses. Computer Networks and ISDN Systems, 28. [On-line]. http://www.webct.com/webct/papers/p29/index.html
  5. Harasim, L., Hiltz, S.R., Teles, L., & Turoff, M. (1997). Learning Networks: A Field Guide to Teaching and Learning Online. Cambridge, MA: MIT Press.
  6. Jacobsen, M., Kremer, R., & Flores, R. (1999). Online Testing and Grading Using WebCT in Computer Science. New Currents in Teaching and Technology, 6(3). [On-line]. Available: http://www.acs.ucalgary.ca/pubs/Newsletters/Currents/Vol6.3/ToC.html
  7. Jacobsen, D. M., Wijngaards, N. J. E., Kremer, R., Shaw, M. L. G., & Gaines, B. (1999). The comparative evaluation of classroom and distance sections of an industrial software engineering graduate course.  Proceedings of M/SET 99: International Conference on Mathematics/Science Education & Technology, San Antonio, Texas, March 1-4, 1999. [On-line]. Available: http://www.acs.ucalgary.ca/~dmjacobs/seng/mset99_paper.html
  8. Jacobsen, D. M. (1998). Adoption patterns of faculty who integrate computer technology for teaching and learning in higher education. Proceedings of the ED-MEDIA AND ED-TELECOM 98: World Conference on Educational Multimedia and Hypermedia & World Conference on Educational Telecommunications, Freiburg, Germany, June 20-25. [On-line]. Available: http://www.acs.ucalgary.ca/~dmjacobs/phd/phd-results.html
  9. Kristapiazzi, G. (1998). Compare Web Tools for Course Authoring. [On-line]. Available: http://www.geocities.com/Eureka/Gold/6012/compare_web_tools.htm
  10. Mueller, J. H. (1997). Research on-line: Human participants ethics issues. [On-line] Available:  http://www.psych.ucalgary.ca/CourseNotes/PSYC413/StudyTools/ethics/online.html
  11. Rogers, E. M. (1995). Diffusion of Innovations. (4th ed.). New York: Free Press.
  12. Rumble, G. (1997). Costs and Economies of Open and Distance Learning. London, UK: Kogan Page.
  13. Shaw, M.L.G. & Gaines, B.R. (1998). A research-based masters program in the workplace. Proceedings of WCCCE '98: Western Canadian Conference on Computing Education, Vancouver, B.C.[On-line]. Available: http://panache.cs.ubc.ca/wccce/program98/mildred/mildred.html
Software Engineering Research Network (SERN)
247 MS, Department of Computer Science
2500 University Drive, N.W.
University of Calgary
Calgary, Alberta, Canada T2N 2N4

© Jacobsen, Kremer, & Shaw 1999