Document View

Most performance technologists spent many hours in college or graduate school conducting literature reviews, but this method of data collection is often abandoned at the threshold of the workplace. Although literature reviews can be an efficient approach to gathering a large volume of information quickly and inexpensively, to many they carry strong negative connotations of academia. If you work in the corporate world, try suggesting a literature review to your colleagues and watch them groan, roll their eyes, or exclaim in an exasperated tone, "That's so academic!" This article explains the practical value of literature reviews and how they can be a worthwhile component of the performance technologist's toolkit. A literature review is the identification, reading, summarization, and evaluation of previously published articles, books, reports, or Internet entries on a particular topic. Sometimes the review may also encompass unpublished documents such as dissertations, manuals, or personal correspondence.

Enlarge 200% Enlarge 400%

Most performance technologists spent many hours in college or graduate school conducting literature reviews, but this method of data collection is often abandoned at the threshold of the workplace. Although literature reviews can be an efficient approach to gathering a large volume of information quickly and inexpensively, to many they carry strong negative connotations of academia. If you work in the corporate world, try suggesting a literature review to your colleagues and watch them groan, roll their eyes, or exclaim in an exasperated tone, "That's so academic!" This article explains the practical value of literature reviews and how they can be a worthwhile component of the performance technologist's toolkit.

A literature review is the identification, reading, summarization, and evaluation of previously published articles, books, reports, or Internet entries on a particular topic. Sometimes the review may also encompass unpublished documents such as dissertations, manuals, or personal correspondence. A field's literature is the record of earlier work in that field (Barzun & Graff, 1977), which forms the foundation on which all future work in the field is based (Borg & Gall, 1979).

Dimensions of a Literature Review

There are four dimensions to consider in planning a literature review: the genre of literature, the types of sources, the timeframe, and the extent of coverage. The objective of the literature review will determine the choices made for each dimension.

Genre

The literature relevant to performance technology can be classified into four groups: academic, professional, business, and government. The academic literature includes journal articles and books primarily written by and for university professors, other scholars, highly experienced professionals, and graduate students in the field. These documents are usually heavily focused on basic and applied research and theory. Examples of academic articles are those published in the International Society for Performance Improvement's Performance Improvement Quarterly and the American Psychological Society's Psychological Science: Research, Theory, & Application in Psychology and Related Sciences. Doctoral dissertations are another example of the academic literature. Many dissertations and some theses are available for download from ProQuest.com. The academic literature reports the basic and applied science that underlies our work in performance technology and allows us to ensure that the practices we employ have an empirical basis for efficacy.

The professional literature is primarily written by and for practitioners in the field, with some contributions from university professors and graduate students. These articles and books are grounded in theory and research, but their primary focus is on the practice of performance technology. Typical content includes case studies, discussions of methods, interventions, or best practices and how to apply them (this article is an example), lessons learned, suggestions for improving practice in the field, and explanations or synopses of theory and research and their potential applications. This journal and the Harvard Business Review are examples of professional literature. The professional literature offers practitioners the opportunity to learn from experienced colleagues about key concepts, theories, and practices that have direct applicability to the workplace.

Business literature includes books and articles published by professional and trade associations and for-profit organizations. The content tends to be less grounded in research and theory, and issues are usually not addressed in depth. In most cases, references are not cited. The content typically includes descriptions of current events and trends, opinion pieces, broad overviews, profiles of organizational leaders, and recommendations for how to improve practice. The American Society for Training and Development's T&O and the business magazine Fortune are examples of business literature. Business literature is helpful in acquiring a basic understanding of a topic and understanding current trends.

Government literature relevant to performance technology includes the guides, white papers, and studies commissioned, written, and contracted by the federal government. These can be located through published indexes and bibliographies (Horowitz, 1984). Some state and local governments also publish guides and studies. In the federal government, most of the documents of interest to performance technologists are published by the Office of Personnel Management, the Government Accountability Office, the Department of Education, the Department of Health and Human Services, and the Office of Management and Budget. Government documents are typically focused on practice or applied research. They provide an excellent source of information about common practices. They also often provide recommendations for improvement and suggest future possibilities.

Type of Source

There are two types of sources for literature that can be included in a review: primary and secondary. Primary sources are direct descriptions of research studies or other events written by an individual who actually conducted the study or witnessed the event (Borg & Gall, 1979). For example, academic journal articles that report research studies are typically written by the scientist or team who conducted the research. Another example of a primary source is a book explaining the author's philosophy or theoretical models.

Secondary sources are documents written by an author who did not directly observe or participate in the events described or who was not the originator of the concepts outlined. Textbooks, encyclopedias, and reviews of research are examples of secondary sources. They are reports and interpretations of the work of others (Borg & Gall, 1979).

Both primary and secondary sources can be valuable parts of a literature review. Primary sources offer a direct view of the research or thinking of the authors without the intermediate interpretation of others. Primary sources offer the depth and veracity required for an in-depth study. secondary sources are useful because they integrate information from many different sources into one document and provide an easy method of developing an overall understanding of the topic (Borg & Gall, 1979).

Timeframe

A literature review may include all documents published on the target topic from the current day back through the first publication on the topic. Alternatively, and much more commonly in performance technology, the review will only examine sources within a defined period. For example, a practitioner may decide to review documents published within the past 10 or 15 years. For a topic that is relatively new, the time horizon may be further limited.

However, a common error made in literature reviews, especially by those new to a field, is to assume that a current topic emerged recently, when in fact, it has long roots in the past that continue to have a major impact on current practice. For example, although e-learning is a relatively recent trend, it has roots that extend more than 40 years into the past. I remember that when I wrote my doctoral dissertation I was required to include all literature published in the past 10 years. Then, 10 years seemed like a very long time. Now that I have worked in performance technology for many years, I see more clearly how the work of today builds directly on the work of the past. If I included only the past 10 years in a literature review today, I would feel that I was missing a large amount of important information on my topic. Two types of literature that are especially valuable in developing a long-term view of a topic are journal articles that are themselves reviews of the literature in a specific area and meta-analyses, which synthesize many research studies on a single topic conducted over a period of several years. Both of these can be excellent sources of information.

Level of Coverage

A literature review may consist of a comprehensive examination of all sources published on a topic within the time-frame, or it may include only selected sources. A comprehensive literature review would include publications in related disciplines as well as in the reviewer's own discipline. In some cases, however, given the purpose of the literature review, it may be reasonable to search only selected sources. For example, for a benchmarking study, one may wish to limit the sources to professional literature and business magazines within a single field.

Often, however, much valuable information is missed if one checks only sources in one's own discipline. For example, I have often been surprised by how infrequently practitioners in psychology review the literature in related disciplines such as performance technology, education, sociology, anthropology, and management. Expansion of a literature review to multiple disciplines provides a much wider perspective. I learned this lesson many years ago. Shortly after I completed graduate school, my husband accompanied me to a comparative psychology session at a major psychology conference. The presenting scholars were discussing the songs of birds as a primitive form of language and the possibility that bird song may provide clues to the development of human language. They referred to a troublesome point that they could not resolve despite much effort. My husband, who was then a graduate student in music, turned to me, whispered, "That's easy!" and answered the question in a few words. If those professors had checked the music literature, they easily could have found the answer to the question they found so perplexing.

Applications of Literature Reviews in Performance Technology

There are many applications for literature reviews in the field of performance technology. Below are discussions of several.

* Professional Grounding: An important element of Human Performance Technology (HPT) is its grounding in theory and research. It is important for all HPT practitioners to build a strong knowledge base before they begin work in a new area. It is also important to understand underlying concepts, principles, and theories and to be familiar with previous thought and work. If the practitioner does not develop a strong knowledge base, work tends to be shallow and naïve. Building on the foundation of previous work allows HPT professionals to enrich and expand the HPT knowledge base rather than duplicate work done by others or repeat detours and mistakes (Borg & Gall, 1979).

* Performance and Cause Analysis: A review of the liteature can assist in the determination of which datacollection methods are appropriate for the situation while also alerting one to potential pitfalls. Reviewing relevant studies of organizational behavior, motivation, interpersonal relationships, and other features of the workplace can also lead to the insights needed to identify possible root causes of problems.

* Identifying and Developing Performance Interventions: A literature review can be an efficient approach to identifying potential performance interventions. The sources reviewed can provide ideas for interventions as well as guidance in the factors to consider in evaluating the potential effectiveness of an intervention in a specific context. Past work can also offer models and instruction in the development of interventions.

* Competency Modeling and Job Analysis: A review of past studies for the target job can be an efficient and effective first step in job analysis and competency modeling. Learning about the target job in other organizations and the competencies that have been identified as important provides a time saving baseline for your work by providing the background you need to ask the right questions. When I conduct a competency modeling study, I routinely begin with a review of published work focused on my target job. These studies help me quickly grasp the general content of the job, its key responsibilities and issues, and the kinds of competencies that are commonly seen as important for success. I am then able to talk intelligently with job incumbents and their managers about the relevant competencies. They appreciate the fact that I have tried to learn about their jobs and that I use the terminology they use. I can also plan an effective methodology for my study because I have a conceptual framework of the constraints and issues involved in the target occupation.

* Benchmarking: A literature review can be a quick approach for preliminary benchmarking. Articles in professional and business publications often describe the practice of an intervention in an organization, thus providing both a way of identifying organizations that apply the practice of interest as well as an overview of the practice in action. Market research studies provide company profiles as well as SWOT (Strengths, Weaknesses, Opportunities, and Threats) analyses of various companies. The performance technologist can then contact the organizations named for follow-up or simply use the collected information as a summary of practices. For example, for a benchmarking study I led regarding performance management practices within large organizations, our first step was to conduct a literature review to identify companies with performance management systems of interest. We then contacted these organizations for participation in a telephone survey.

* Measurement and Evaluation: A literature review can be an effective approach for identifying potential methods of measuring or evaluating interventions, processes, or individual and organizational performance, especially for difficult-to-measure areas. Test Link is a database of 20,000 tests and other measurement instruments dating from the early 1900s. For example, when I developed a performance measurement framework for a large government agency's workforce planning process, I conducted a review of the academic, professional, business, and government literature to learn about existing measurement models and performance indicators. A valuable source of information on measurement methods is the Test Link database available from Educational Testing Service.

Advantages and Disadvantages of Literature Reviews

As with other methods of data collection, literature reviews have advantages and disadvantages.

Advantages

* Literature reviews are versatile. They can be conducted for almost any topic and can provide information either at the overview level or in-depth.

* Literature reviews are relatively inexpensive and efficient. A large amount of data can be collected quickly at minimal cost.

* No scheduling or coordination is involved. The cooperation of others is not required.

* The only resources needed are a good library or online database and a competent reviewer.

* Literature reviews can be an excellent first step in a project or study because they provide a conceptual framework for further planning and study.

Disadvantages

* An effective literature review requires a high level of skill in identifying resources, analyzing the sources to identify relevant information, and writing a meaningful summary.

* Literature reviews are limited to collecting information about what has happened in the past, and usually within organizations other than the researcher's own workplace. They cannot provide data about current actual behavior.

Guidelines for Literature Reviews

Plan the Review

* Concisely define the topic in a few words or a sentence or two. It is important to limit your search to a clearly defined area so time is not wasted on peripheral topics.

* List the key words that you -will use for searching indexes, databases, and bibliographies to identify relevant publications. Begin with as many relevant words as you can think of and be prepared to amend your list after you begin your search to include only those that prove fruitful and to add additional terms you discover as you search (Borg & Gall, 1979). Descriptor thesauruses, such as the Educational Resources Information Center (ERIC) Thesaurus of Descriptors (available via www.eric.ed.gov), and the Thesaurus of Psychological Index Terms (www.apa.org) are very helpful in identifying key search words.

* Consider the four dimensions of genre of literature, type (primary or secondary sources), timeframe, and level of coverage as you plan your search.

Identify and Locate Sources

* Use your key words to identify potential publications by checking indexes, abstracts, and bibliographies in your areas of interest. These can be searched in hard copy or online. Examples of indexes that are valuable in performance technology are the Education Index, Business Index, the Social Science Index, and the Monthly Catalog of United States Government Publications. Helpful abstracts include Psychological Abstracts, Sociological Abstracts, Dissertations Abstracts International, and Resources in Education. Librarians can help you locate bibliographies. The Bibliographic Index is also a handy resource for identifying specialized bibliographies.

* Most literature reviews will require use of a large university library or access to several online databases.

* Many online databases offer both a key word index for identifying possible sources as well as abstracts and the full text of articles. These databases are a great boon in literature reviews because they can identify hundreds of possible sources in a few minutes. However, it's important to remember that these databases typically include only a portion of all the journals in the field. One usually must search several different databases for comprehensive coverage. Access to the databases also can be quite costly.

* Consulting the online Books in Print and conducting key word searches of your library's online catalog are good ways to locate books on your topic.

* An excellent approach to identifying further sources is to review the list of references included in the books and articles you find. These will typically lead you to many other valuable publications.

* Plan ahead for retrieving the sources you locate. Even large university libraries will not carry all the books, journals, and other sources you identify, and not all journal articles are available online, so you may need to request interlibrary loans, travel to several libraries, or request reprints.

Read and Summarize the Sources

* Once you have located an article or book, quickly skim the abstract and the conclusions to see if it includes the information you need.

* Identify the sources that seem most closely related and important to your topic, then begin reading the most recent of these sources. Recent sources are most likely to be valuable because they have the benefit of earlier work and research as a foundation. Reading the most important articles and books helps one quickly build an understanding of the topic and then use this knowledge to fit the less important publications into an overall picture (Borg Su Gall, 1979).

* As you read, take notes on the key information, putting it into your own words. Using your own words helps you understand and remember the information better than if you simply copy the author's exact words.

* Taking notes on index cards works well because you can easily shuffle the cards into categories when you have finished reading all your sources. Some people prefer to photocopy articles and portions of books rather than taking notes, but this approach does not generally allow for internal cognitive processing and tends to result in a more superficial understanding of the material. A useful alternative to handwriting notes on index cards is the use of bibliographic software such as Endnote (www.endnote.com), which serves as a relational database of bibliographic information that can be input by hand or downloaded from the Internet.

* Be sure to record the bibliographic information. This does not need to be done on every card. Assign each source a number, then note that number on each card on which you record information from that source. Also note the page number for the information on each card.

* As you read, evaluate the soundness of the ideas and information presented. Consider whether they are well supported. Include your personal comments in your notes, but be sure to indicate which are your opinions and ideas and which are the author's.

* When you have read all your sources, you can efficiently summarize the information by organizing your note cards into categories of information, then creating a synopsis of the information on the cards in each category.

* There are several excellent books that offer detailed information on preparing formal literature reviews. These are listed in the Related Readings section below.

Conclusion

Literature reviews can offer an efficient method of building a professional knowledge base, understanding performance issues, identifying potential interventions and measurement methods, providing a foundation for asking the right questions in a project, and defining common practices in organizations. They are an important component of the performance technologist's tool kit.