Formative and summative evaluations are designed and conducted to evaluate the instructional materials themselves. Formative evaluations occur during the development of the instructional materials; summative evaluations occur after the instruction has been implemented. Essentially, we can view formative evaluations as type of usability testing - a concept that should be very familiar to technical communicators. Summative evaluations are often used to determine if the instruction meets organizational needs and should be continued or terminated. In this assignment, you will create a draft of your instructional materials and conduct two types of formative evaluation.
The deliverable for this assignment is a copy of your draft instructional materials and a report with three sections.
4.1 One-to-One Formative Evaluation
4.2 Small Group Formative Evaluation
4.3 Plan for Revising Instructional Materials
The goal of this part of the assignment is to create the instructional materials, based on the instructional strategies you have specified in Assignment 3. This will the the fun part for many of you (finally!) as you will get to use many of the skills you have learned in other MSTCO courses or elsewhere. I suggest you follow the advice of Dick and Carey and develop self-paced instructional materials rather than instructor-led materials. Self-paced materials will allow you to focus on the instruction itself and will be easier to assess during the formative evaluation process.
However, if you choose to develop instructor-led materials, use this example of a previous student assignment to produce the instructor's narrative.
Follow the procedure on p. 236 to create a rough draft of the instructional materials that can be used in formative evaluation. Do not invest an inordinate amount of time in creating elaborate instructional materials at this point. I guarantee that once you have done a formative evaluation, you will need to revise the materials and you will have wasted lots of time, energy, and money if you have produced more than a rough draft to be evaluated. Save a copy of these materials and submit them as an attachment to your report.
Good designers evaluate their instructional materials during the design process, then revise the materials based on feedback from members of the target population,SME's, outside reviewers, etc. You probably have done some informal formative evaluations as you were developing the instructional analysis diagram or deciding on instructional strategies. Now, you are ready to test your instructional materials on 'real' students.
One-to-one evaluation is exactly what it implies: you, the designer, sit down with one learner from the target population as s/he goes through the instructional materials. Obviously, to get the most information from this interaction, you will need to do some planning. I suggest you follow the procedures listed on p. 277-279. Also look at Table 10.1 on p. 260 for a suggested framework. Perform one-to-one evaluations with at least three learners.
Now, you must make a decision: how much revision to make to the instructional materials before proceeding to small-group evaluation? This is entirely up to you, but I suggest that if you have time to make improvements to the instructional materials you do so—you will get much more relevant feedback during small group evaluation if you have corrected some or all of the problems identifed during the one-to-ones. If you choose to revise the materials, include a copy of them as an attachment to your report.
For this part of the assignment, ideally you would select at least 8 (and preferably more) learners from your target population as subjects for the small group formative evaluation.
In reality, select the group that is logistically possible in your organization.
Follow the procedures on p. 279-280 to conduct this phase of the evaluation process. (Note: you are not required to conduct a field trial.) This section of the report should describe the learners, the procedures, and the materials and instruments used in your small group formative evaluation.
Consider how best to present the data collected during the evaluation. See Table 11.1 (p. 298), 11.2 (p. 299) and 11.3 (p. 300) for examples. Graphs (Figure 11.1 and 11.2, p. 301) are also an option. Your goals here are two-fold: first, to identify weaknesses or flaws in the instruction that prevent the learner from achieving the terminal objective by actually conducting the evaluation, and second, to describe to others (perhaps your manager, co-workers, other designers, etc.), how you conducted the evaluation and what you found.
Option Two: Expert Review (adapted from http://classweb.gmu.edu/ndabbagh/Resources/IDKB/eval_techniques.htm)
As an alternative to small group evaluation, you may utilize expert review as a formative evaluation technique.Typically, expert reviews are conducted by an expert with or without the evaluator presents. Experts may be content experts, instructional design experts, content-specific education specialists, or experts on the learners, such as teachers (Smith & Ragan, 1999). Experts can review the instruction to provide feedback on accuracy, completeness, instructional integrity, and time required for completion (although completion time may not be representative of the target audience).
Content experts are helpful because they can validate the congruence of content with current educational theory in the specific subject area.
Smith and Ragan (1999) suggest dividing "context experts' comments into three categories:
Additionally, experts can provide verification and validation of both content and instructional strategies.
To conduct an expert review, share your instructional materials with a designated expert reviewer, along with guidelines for requested feedback based on the above criteria.
Now that you know where the problems are with the instruction, the next step is to develop a plan for revising the instructional materials. This section of the report should contain a description of how the instruction will be revised. See Table 11.6 (p. 313) as a potential format for describing such a plan. You are not required to actually make these revisions to your instructional materials based on your plan, but you may choose to do so for work-related or other purposes.
codone_s@mercer.edu
Office: 478-301-4185
(Example 1 represents a formative evaluation)
(Example 2 represents a draft of materials)
(Example 5 is long because it has a detailed appendix and extensive instructor directions)