Document View

During the last decade, the use of computer-based technology in education has dramatically increased. Higher education has spent some $20 billion in the 15 years through 1995. In 1994, $2.4 billion was spent on educational technology in kindergarten through 12th grade and $6.0 billion in higher education. Computer sales have risen by 25% in each of the past two years up through 1996, placing 50 million new computers into American homes and offices. More than 90,000 miles of fiber are already in place in the United States, and fiber deployment spreads at a pace of 1,300 miles per day (Katz, Tate, & Weimer,1995).

Given that educational technologies are currently receiving significant attention, questions are now being raised regarding the research and assessment results that support the adoption and inclusion of technology in all levels of the educational system, particularly because the investments have been and remain so high. Today, many objective observers on this issue are beginning to realize that the research supporting the massive adoption of technology simply does not exist to the extent that these widespread trends are justifiable. Katz et al. (1995) claim that, in spite of significant increases in per-student expenditures, the use of technology in education has had little effect on productivity (especially when compared to similar sectors such as health care). A more recent editorial suggests that:

With all the studies and documentation available, research on why and how the use of technology is effective in education remains minimal . . . challenges remain for accurate and meaningful research to ensure the proper use of technology in education (Charp, 1998, p. 6).

McKenzie (1995) has indicated that there may be any number of hypotheses and rationales explaining why valid assessments of technology application and massive expenditures have not been sufficiently evaluated, including inability of program participants to conduct appropriate studies, vested interests in protecting new programs, little respect for educational research by the educational community, and unwillingness to set program goals, which are required by valid research.

A recent empirical study performed by Jones and Paolucci (1997) estimates that since 1993, less than 5% of published research was sufficiently empirical, quantitative, and valid to support conclusions with respect to the effectiveness of technology in educational learning outcomes. They argued that claims for the influence of technology, although substantial, were largely unfounded, and serious consequences may result if acceptance of technology in educational delivery continues without considering the appropriate application. They questioned the untested educational quality resulting from relatively unproven paradigms involving technology and the questionable cost benefit associated with this continuance. In addition to the fact that very few empirical educational technology evaluation studies have been performed, those that are available often provide unclear conclusions and lack generalization possibilities because a clearly defined research model or framework is often missing (Jones & Paolucci).

We suggest here that the approach to evaluating educational technology should consider variables that are common to recognized pedagogical models. Furthermore, we suggest that there must be a research framework into which specific studies can be placed and from which practitioners can draw unified, high-level conclusions regarding the appropriate application of technology to various domains and teaching and learning requirements. This matrix, by its nature would allow the identification of relevant technologies to be applied to appropriate content. In this article, we suggest such a framework. We identify a matrix of factors to allow researchers to identify and target specific work and ultimately contribute to more comprehensive results.

RESEARCH FRAMEWORK AND DIMENSIONS

What is the most appropriate technology to use? In what educational context is a particular technology most effective? How effective is the chosen technology? The answers to these questions and others are critical in establishing the value added to the students learning and achievement with the use of educational technology. However, with too many evaluation research studies, these questions are rarely addressed. And, if they are, the results are often unclear, because of the lack of a clear evaluation framework. In this section, we will present and briefly describe a general framework for evaluating the effectiveness of educational technology in enhancing student learning and achievement.

Before progressing to the presentation and discussion of the research framework, it is worthwhile to clarify certain terms and criteria that are central to our theme. It is important to remember that we are dealing here with the issue of educational technology-that is, the use of technology to enhance the teaching and learning process. This should be distinguished from the term technology education, which involves teaching the use of technology. Technology education focuses on the learning and instruction of technology while educational technology involves the use and application of technology-based tools in the educational process. We focus only on the latter, because it from this perspective that we, as educators, have the opportunity to most significantly add value to the teaching and learning process.

Second, we emphasize that we are concerned with educational learning outcomes. Certainly technology is integrated into the teaching component of pedagogy, but we suggest and take the approach that ultimately it is the contribution of technology to learning outcomes that should be of interest to educators and that is currently not adequately assessed. Often in discussion of the use of technology in education, the terms teaching, education, and learning are synonymous. We strive to emphasize the difference among these. When we define learning, we define it in terms of cognitive outcomes and not simply in the delivery paradigm that may be used at any particular time. It is the implicit ability (or inability) of technology to affect learning outcomes that we believe should be of utmost interest.

To address these questions, we believe that the use of an instructional system design (ISD) approach to educational technology evaluation is critical. In its most general form, an instructional system can be viewed to consist of three major components: instructional objectives (input), delivery system (process), and learning outcomes (output). Although there are many ISD models that include system evaluation as a major component (Dick & Carey, 1996; Gagne, Briggs, & Wager, 1992; Kemp, Morrison, & Ross 1994; Reiser & Dick,1996; Seels & Glasgow, 1997; Willis,1995), with some even providing guidelines for selecting media and delivery systems (Kemp er al.), none of them adequately addresses the complexities and dimensions of the technology and how these may relate to learning outcomes. In response to this shortcoming, this study will focus on the dimensions of the delivery system and technology. It will present a general framework that it is hoped will guide researchers in establishing a clearer relationship among instructional objectives, the choice of technology-based delivery system, and learning outcomes. These components and their relationships are detailed in the following sections.

Instructional Objectives

Learning is achieved when a permanent change in thinking, attitude, or behavior occurs. Thus, the overarching objective of any instructional system should be to facilitate this process. (It should be noted, however, that although instructional systems are intended to provide an environment for facilitating learning, learning is an internal process that can be done only by the student.) For many formal learning situations, an instructional system does not happen by serendipity. It requires significant planning, design, and a sophisticated decision-making process. This process begins by clearly identifying a set of instructional objectives and goals. All instructional objectives can and should be based on one or more of the following factors: learning domain, learner profile, task characteristics, and grouping. Descriptions of these dimensions follow.

Learning Domain

The instructional objectives should correspond to one or more learning domains. There are three basic domains of learning: cognitive, affective, and psychomotor. Taxonomies have been widely used to define learning within each of these domains. The most popular ones are by Bloom (1956) for the cognitive domain; Krathwohl, Bloom, and Masia (1964) for the affective domain; and Harrow (1972) for the psychomotor domain. Although in practice it is very difficult to separate these, it is often possible to clearly emphasize one over the others.

Learner Profile

The instructional objectives should be appropriate for the learner's level of ability. Many learners may need prerequisite skills and knowledge for success with any delivery system. Key information about the learner can be used to develop a profile. It is recommended that, as a minimum, such a profile include information on cognitive style, aptitude and ability, relevant experience, educational level, level of achievement with subject domain, motivation, attitude, age, and gender (Seels & Glasgow, 1998).

Task Characteristics

The instructional objectives should be appropriate for the tasks associated with the subject matter that is to be learned. A clear description of the topic to be learned and the steps necessary to achieve this can lead to clear instructional objectives. Task analysis is frequently employed to achieve this goal. Its purpose is to define the operational components of a skill or subject matter (Jonassen & Hannum, 1995).

Grouping

The instructional objectives should be appropriate for the grouping arrangement and learning situation. This can be simply determined by deciding whether the instruction will be with a large number of students, a small group, or just one student. Additionally, whether the instructional objectives require independent or group study and the physical requirements (e.g., location) for student need to be identified. Many technologies can provide support opportunities for all of the above options. However, research shows that the inherent properties of some technologies have more affinity for supporting some factors over others (Seels & Glasgow, 1998).

Delivery System

The delivery system is the method used by the instructional system to transfer information and knowledge from the subject matter expert (human, machine, or both) to the learner or vice versa. The choice of the "optimal" delivery system should be made only after the instructional objectives have been clearly identified and specified. In general, the delivery system can either employ older and more traditional technologies (e.g., print, audiovisual, etc.) newer technologies (e.g., computers, telecommunications, etc.), or a combination of both (Seels & Glasgow, 1998). Although both the older and the newer technologies coexist as possible alternatives, the focus of this study (and of much of the research on this topic) is on the newer electronic or digital technologies. Furthermore, older technologies are increasingly being incorporated in newer technologies (e.g., video and audio information can be delivered by cassettes or by multimedia computer software such as CD-ROMs).

Today, most of the delivery systems are highly integrated digital information technologies, with the computer playing a pivotal role. They incorporate two major classes of technologies: telecommunications (e.g., teleconferencing, Internet, etc.) and multimedia and hypermedia computing (e.g., CD-ROM, World Wide Web, etc.). These technologies (whether used together or separately) are often used for distance learning and can provide individualized, interactive, and multisensory learning experiences. They have the advantage of learner control features, which can allow interactivity. They can be used to access information and people linearly and/or randomly, as well as locally and/or remotely. Moreover, they are capable of integrating media of many different types from many different sources, all within the control of the system software, the learner, or some combination of both Seels dc Glasgow, 1998). We believe that these digital delivery systems, when used as components of an instructional system, have major characteristics and dimensions that effect the educational experience. These characteristics include control, presence, media, and connectivity.

Control

The locus of control is a measure of the amount of control both the learner and instructor have over the learning activities and the information sources. With many instructional systems, this interaction is complicated by the inclusion of digital technology. That is to say, with the use of a technology-based instructional system, one needs to determine its relationship (and the information accessed through its use) to the traditional instructor-learner dyad (Branson, 1997). These instructor-technology-learner relationships can be categorized in terms of the configurations described on the next page. (Figure 1 pictorially presents these, with the solid lines representing direct control and the dashed lines representing weak or indirect control.)

Instructor-Centered

Enlarge 200% Enlarge 400%

Instructor as Lecturer: The instructor has direct control over the learning activities, technology, and content, while the learner has very little control or none at all (e.g., audio-visual technology or programmed instructional software).

Instructor as Facilitator: The instructor has direct control over the learning activities and indirect control of the technology and content, and the student has direct control of the technology and content (e.g., hypermedia or Web).

Technology-Centered

Technology as Mediator: The instructor-learner dyad is mediated by the technology, with the instructor having direct control over the learning activities, and both the instructor and learner having direct control of the technology and content (e.g., teleconferencing or computer conferencing).

Technology as Tutor: The instructor has direct control over the learning activities and no control of the technology. The learner has complete control of the technology and content (e.g., intelligent tutoring systems or simulations).

Learner-Centered

Learner as Constructor: The instructor and learner collaborate in the construction of knowledge, with the instructor having indirect control over the technology and learning activities, and the learner having direct control of the technology and content (e.g., hypermedia or computer conferencing).

Learner as Explorer: The instructor has very little or no control over the learning activities and no control of the technology, the learner has complete control of the learning activities, the technology, and content as he or she independently explores the knowledge space (e.g., intelligent tutoring systems, simulations, or the Web).

Presence

Digital technology systems provide the means for the instructor and learner to come together physically or virtually, synchronously or asynchronously. These modes can be characterized along two dimensions: place and time (see Figure 2). The possible combinations of these two dimensions yield the following possible spatiotemporal configurations (Hedberg, Brown, & Arrighi, 1997).

Same Place/Same Time: Instructional and learning activities are synchronous, and instructor and learner are located at the same place (e.g., lecture, computer lab activity, or seminar). Typical technologies available for this configuration may include all computer-assisted and programmed instruction and presentation software.

Enlarge 200% Enlarge 400%

Different Place/Same Time: Instructional and learning activities are synchronous, and instructor and learner are remotely located (e.g., distance learning and teleconferencing). Typical technologies available for this configuration may include MUDs (multiuser dungeons), online chat systems, and many types of computer-conferencing systems.

Different Place/Different Place/Different Time: Instructional and learning activities are asynchronous and instructor and learner are remotely located (e.g., distance learning or asynchronous learning). Typical technologies available for this configuration may include electronic mail, and Internet newsgroups.

Same Place/Different Time: Instructional and learning activities are asynchronous, and instructor and learner are located at the same place (e.g., self-- paced and individualized learning). Typical technologies available for this configuration may include tutoring systems and simulation software.

Given this, in meeting the educational objectives, it would seem that the choice of any of the above modes of deliveries requires the choice of an appropriate technology. It is a commonly held view within the research community that some technologies are more appropriate than others. However, the level of technology effectiveness in enhancing the instructional and learning processes remains to be seen, especially with distance learning (Recker, 1997).

Media

Digital technology systems allow the dynamic access and processing of individual media types, such as text (one medium) or, more commonly, a multitude of media types (multimedia). These multimedia types commonly include text, audio, graphics, and video, and are usually interactive (e.g., CD-ROM or DVD). Furthermore, when these media are dynamically linked or hyperlinked (e.g., hypermedia), they allow significant learner control over the information (e.g., the Web). Finally, increasingly, these media are becoming immersive, where the information can be presented in three-dimensional space, allowing the learner to interact with a virtual environment (e.g., virtual reality or simulation). In general, the choice of media is highly dependent on the specific learning context (Kozma, 1991). However, much research is still required, especially with hypermedia and virtual reality.

Connectivity

Digital technology systems allow people to connect with one another and with information resources. More specifically, these systems can be designed to facilitate and support information access and exchange, communication, and collaboration among individuals and groups in accomplishing their tasks and in their activities. These technologies are often referred to as computer-mediated communication and groupware. Currently, the Internet is the network of choice with educators to connect instructors, learners, and information on a global basis. Furthermore, with the popularity of the Web, this information can be hypermediated, highly unstructured, and readily available. Consequently, we are currently experiencing an explosion of Web-based instructional systems. The Web has suddenly become the de facto global technology platform for instruction and learning. Although Web-based instruction is the fastest growing area of educational technology research, we know little about how to effectively design and implement these systems for educational applications (Romiszowski, 1997).

Learning Outcomes

The assessment of learning outcomes provides the major feedback mechanism within the instructional design process. It is critical in evaluating the instructional system and its effectiveness. The information that is collected as evidence of learning achievement will depend on the nature of the competency being measured. Usually, these consist of cognitive tests (measurements of intellectual skills), performance tests (measurements of capability), and attitudinal test (measurements of disposition and perspective). Additionally, the instrument and technique used to assess these outcomes will also depend on the learning domain and objective-written tests for cognitive objectives, portfolios for performance objectives, and interviews for attitudinal objectives (Seels & Glasgow, 1998).

The use of Bloom's Taxonomy of Educational Objectives (Bloom, 1956), as an example, provides a widely accepted and researched framework for evaluating cognitive abilities. These include knowledge, comprehension, application, analysis, synthesis, and evaluation. Furthermore these cognitive skills often are classified as lower order (e.g., knowledge, comprehension, and application) and higher order (e.g., analysis, synthesis, and evaluation). Assessment tests based on Blooms Taxonomy have been effectively employed to measure the effectiveness of educational technology on cognitive learning (Paolucci, 1998).

Finally, it has been our observation that too many educational technology evaluation studies minimize the cognitive domain (Jones & Paolucci, 1997). In general, we believe that all learning objectives have a cognitive component associated with them, whether these are primarily behavior.al or affective in nature. In general, we think that cognitive tests are particularly important in the assessment of learning and technology, and, therefore, they should receive much more research attention than they are presently given.

CONCLUSIONS

In this article, we have attempted to define a framework that brings together the multiple dimensions of integrating technology with the learning process. Our goal is to establish the relationships among the various dimensions of instructional objectives, delivery system, and learning outcomes (see Figure 3). This is done with the aim of identifying the need for and laying a foundation for controlled studies that contribute meaningful inputs to the open question on the effectiveness of technology on learning outcomes. Only through systematic research and assessment will we identify the appropriate technologies to deliver specific learning objectives and materials. This framework was developed to provide some guidance in the development of this research agenda.

We identify that this is but a first stage in providing meaningful answers to these questions. We submit that additional work must be completed in reviewing the previous and current research that contributes to the matrix identified and, consequently, which areas of the matrix remain open for future assessment. Jones and Paolucci (1997) identify that less than 5% of research completed to date may contribute in this respect; this finding leaves substantial work for the future. In fact we argue that it is this lack of research addressing the specific mix of the dimensions discussed that warrants this type of academic discourse. Additionally, we agree that although this framework addresses the teaching-- learning process and the potential for technology to contribute to the system, it ignores what may be a major consideration that at some point must be considered in a full multidimensional analysis: cost-effectiveness. Given the investment in technology, we must at some level consider the incremental cost of adding technology to the process and the value added to the learning outcomes by the expenditures made. This may indeed be the most difficult of all dimensions to assess.

Enlarge 200% Enlarge 400%