Week Eleven Formative Evaluation (1 on 1 testing)

This week begins our journey into formative evaluation -- learning how to evaluate our instruction while it is in the formative stage, before we implement it within our organization for final use. We'll begin with evaluating our instructional materials on a one-to-one basis. Note that we have almost completed the instructional design process.

Learning Objectives

1. Describe the purposes for and various stages of formative evaluation of instructor-developed materials, instructor-selected materials, and instructor-presented instruction.
2. Describe the instruments used in a formative evaluation.
3. Develop an appropriate formative evaluation plan and construct instruments for a set of instructional materials or an instructor presentation.
4. Collect data according to a formative evaluation plan for a given set of instructional materials or instructor presentation.
5. Describe various methods for summarizing data obtained from formative evaluation studies.
6. Summarize data from formative evaluation studies.

Overview

Formative evaluation is simply evaluation that occurs while an instructional product is being formed.

The purpose of formative evaluation is to collect data and information while instruction is being developed in order to improve the effectiveness of the instruction. It is quite likely that you have been doing formative evaluation all along, although you may not have known what to call it. Essentially, think of it as a reality check—are you on target with the design and development of the instruction? Every time you ask someone else to comment on what you are doing in a particular phase of the ID process, you are performing formative evaluation. Now, we are going to formalize the process (and this will look very familiar to those of you with usability testing experience).

We will conduct two types of formative evaluation: one-to-one and small group evaluation. Note that there is a third type, field trial, but you will not be responsible for conducting a field trial for this course. The ultimate goal of formative evaluation is to "try out" the instruction with a set of learners who are representative of the target population to see if it works, then make the appropriate revisions before you "go live" with the intended learners.

You do have the option of substituting an expert review for the small group evaluation, but we will talk about this later in the course.

The results of your one-to-one formative evaluation efforts constitute section 4.1 of Assignment Four, while the results of your small group evaluation should be included in section 4.2.

Please note that we will spend the next four weeks performing formative evaluation tasks and revising or planning the revision of the instruction (Weeks 11-14).

Advice on Chapter 10-11

Although Chapter 10 discusses formative evaluation and Chapter 11 discusses revising the instructional materials, I think this division is arbitrary, and suggest you read both chapters before starting the formative evaluation process. Here are my suggestions on how to proceed with formative evaluation of your instructional materials.

This week I suggest you concentrate on one-to-one formative evaluation. After you have skimmed both Chapter 10 and 11, to give you the 'big' picture, go back and reread carefully p. 261-266 on how to conduct one-to-ones. Then read p. 277-279, which essentially summarize formative evaluation activities.

One-to-One Evaluation Factors

These who-what-where-when-how questions will guide you as you complete the one-to-one evaluations.

• Who are the sample learners? You should choose a minimum of three, according to the guidelines in Dick and Carey.
• Where will I conduct the evaluation?
• What will I say to the learners to ensure that they understand the purpose of the evaluation and to put them at ease with the process?
• What kind of data do I want to collect? See Table 10.1 on p. 260 for ideas.
• What will I do with the results of the one-to-ones?
  • if you have time, you should correct any errors (that can be easily fixed) before proceeding to small group evaluation
• How will I report the results?
  • In section 4.1, describe how you selected the learners and the tools, methods and procedures (including setting) for the evaluation
  • Collate learner feedback on a master copy of the instruction
  • Summarize data on pre- and/or posttests
  • Describe changes, you made, if any, to the instructional materials (or include a revised version of the materials with your changes color coded)

Required Reading

1. Chapter 10-11 (skimming)

Assignment

1. As you finish Assignment 3, think about how you will begin to develop a rough draft of your instructional materials and how you will organize and complete a one-to-one evaluation, eventually summarizing the results in assignment 4.1. Prepare to discuss at a very high level on Monday night.

Online Class Discussion

Monday night, 8-9:30 p.m., Eastern time via Webex.

• 8:00-8:10 Introductory discussion -- Varad
• 8:10-8:45 Discussion of formative evaluation techniques.
• 8:45-:920 Discussion of your plans for formative evaluation.
• 9:20-9:30 Preview of next week

Please email me with any questions.