Wednesday, June 03, 2009

#ASTD09 Web-Based Instruction - Design and Technical Issues

Presentation: Web-Based Instruction - Design and Technical Issues that Influence Training Effectiveness
Presenters: Traci Sitzmann, Katherine Ely, ADL

Wow! Pretty good crowd for this one.

Methods for improving the effectiveness of eLearning (summary of the handout)
- Trainees get control over content, sequence and pace
- Practice and feedback throughout the course
- Require them to be active - assignments, practice quizzes, utilizing interactive activities
- Incorporate variety of instrutional methods
- Incorporate synchronous human interaction (chat rooms, virtual classrooms) rather than asynchronous
- Offer computer and Internet skills courses
- Provide access to online lecture notes


Research Projects
- Examine relative effectiveness fo WEb-based (WBI) vs. blended (BL) vs. classroom (CI)
- Recommendations for WBI
- Investigate effect of intervention to increase learning in WBI
- Study effect of tech glitches on learning and attrition

Meta-Analysis - comparing WBI, BL to CI in terms of effectiveness for teaching declarative and procedureal knowledge
- Examine training design characteristics which influence effectiveness
- Gathered every study could find Classroom to WBI or Classroom to BL
- Avg age 25, 208 training courses (143 undergrad courses, 34 grad courses, 31 corporate) 26,460 trainees
- Focus on comparison
- Examined results differ. Results apply to all settings.
- Slides and research available on ADL website.
- Web Based - anything delivered online.

HEdges and Olkin procedure (1985)
- d>0 WBI more effective
- d<0 CI more effective
- d=0 WBI and CI equal

Effectiveness - we were comparing declarative and procedural score.
- It is just aggregate. Of various quality. Types. Styles.
- This is just blanket

[very aggressive group in the room]

Web based instruction 4% more effective than CI for teaching declarative knowledge
Web based instruction equally effective for teaching procedural knowledge

We are only measuring level 2. Did look at reaction. Most studies did not look at level 3 or 4.

Blended learning more effective than classroom for teaching declarative and procedural knowledge.
- Repetition in blended.
- Get more learning styles
- Time between the different types for training for processing
- Variety in presentation

Do instructional methods or delivery media have a larger effect on learning outcomes?
- Same instructional methods used - equally effective both Web Based and CI
- If different instructional method used in web based vs. classroom - 11% was more effective

In the courses that used different instructional methods - the web-based use more VARIETY of instructional methods than the classroom.
- They could go from 1 method to the next and could continue reviewing.

Instructional methods rather than delivery media predict learning outcomes.

What characteristics of most effective Web-Based courses
- Trainees control / customize learning experience.
+ More control = more responsibility
+ Spend as much time as want / need
+ More learner control = more learning
+ pace and content.
+ Applies across synchronous and asynchronous

- Provide practice and feedback
+ WBI allows you to incorporate more immediate opportunities, more immediate feedback
+ More practice opportunities period.

- Require trainees to be active
+ Questions, collaborate, discuss content, complete exercises.
+ Inactive when listentin learning or reading
+ Instructors should look for ways to incorporate active learning into the course by including tutorials and collaborative activities

- Incorporate variety of instructional methods
+ Lectures, tutorials, discussion boards, online readings
+ Can more easily self-tailor course to learner style using WBI

- Incorporate Synchronous Communication (Real-Time)
+ IM, Voice Chat, Online office hours.
+ Reduces frustration from lengthy time delays between asking a question and receiving a response

- Providing an internet skills course
+ Some trainees may not have the computer and internet skills [no kidding]
+ This may become less important as we move forward

Examining Time to Train
- 111 trainees going through occupational trainig consisting 33 self-paced modules
- Classroom instruction lasted 86 days before conversion
- TO advance, web-based had to pass an exam
- Goal of study - to examine the amount of time trainees spent in a self-paced course
- Found in web-based instruction - complete in 45 days for average learner.
+ AS few as 21 days
+ AS many as 77 days
+ Account for things like prior knowledge. The course was for electricians.
- 48% reduction in time
- Did not reduce knowledge levels
- Result: With online self-paced instruction, students can train at their own pace and STILL MASTER THE MATERIAL
- No motivational variables in the study
+ Some thought the time difference was a result of prior experience. Idea that people don't come to training with blank slate. Could test out - not forced to go through all of the stuff they already know.
- If had to go back and retake - also included in the study time
- Service in previous employment not a factor, but this did not take into account prior technical knowledge.
- Did not have transfer knowledge data - did not study what happened after the training.
- Training was done at a computer lab - for specific job. Physical validation of student in this example.

Sometimes, we just need reminders

Online and classroom instruction - very different learning environments
- Classroom instruction - schedule.
- Online - onus for learning on the STUDENT.

How learners self-regulate
- Learner has to set aside time
- Has to choose appropriate study environment

Self-regulation a process that enables individual to guide goal-directed activities over time and across circumstances
- Iterative process with a gradual effect on learning over time.
- Also want them to evaluate their progress.

Intervention to encourage students to self-regulate during the course
- ASk questions to remind to self regulate - experiment
+ AM I concentrating on learning the training material?
+ Do I need to continue to review before taking the final exam?

- Continous self-regulation
+ Prompted throughout entire course
- Delayed self-regulation
+ Prompted only in 2nd 1/2
- Control
+ No prompts to self-regulate

- Study 1 - Using Blackboard LMS, 93 trainees, Avg age = 44 Mult choice + performance
- Study 2 - TANDEM, PC based radar-tracking, 171 undergrads, Avg age = 19 Performance
- Study 3 - MS Excel avg age = 42 Exams

- Results across all studies.
+ Change in performance across 9 training modules
+ Control group - performance slightly below average to start, declines across the modules.
+ Continuous self-regulation - Performance start above average, peaks at 5 modules, stay at high level
+ Delayed self-reg - Performance below average. Once prompts start at module 5 - get improvement and level off at high level (but not as high as continuous, though not statistically significant).

- By using self-regulation - results improve between 10% - 20%

The intervention - 10 sets of powerpoint slides - likert scale.
- AM I concentrating? 1-5
- Gotta answer before you can continue content.

Self-regulation increases the amount of time spent in training

They had a firm presence across ALL. But no reminders for control.

Predicting attrition
- When trainees prompted to self-regulate - 17% reduction in attrition relative to control.
- Trainees less likely to drop out following poor performance when prompted to self-regulate

In these studies - asynchronous. Would probably see less of an effect with the prompt.

3rd study - full age range - effect did not differ by age.
- if you remove the prompts mid-training, a bit of drop-off.
- It was every 20ish minutes worth of material. Same marker in the course.

- Adults capable managing their own learning when receive reminders to self-regulate
- Prompting self-regulation is a NO COST INTERVENTION
- Can be incoorporated in any Web-based training course.
- Spend half an hour MORE when in continuous self-regulation

Sample prompt questions (PPT) Click Cancel when get the password prompt

Interruptions in Online Training
- In classroom - taking people to a quiet, distraction-free environment.
- Online (puported) - anytime anywhere

- Interruption - externally generated, randomly occurring, discrete event that breaks continuity of cognitive focus.
+ 77% could not complete the course in 1 attempt
+ Starting series on interruptions and how they impact online learning

- Technical difficulties studied
+ Low bandwidth, computer config, JavaSCript and other error messages
+ Current study examines effect of tech problems on learning and attrition
+ 530 adult trainees, 75% emplloyeed full or part time, 52% had bachelors +, 69% female, avg 41 years. MS Excel training - 5 hours. Divided into 4 online training modules.
+ At end of each module,
+ Conditions differed on how many modules contained tech problems, where in module
- The error designed to interrupt. See if they click through or not.
- Found only 19% completed online training course
+ Results consistent with previous research
- Social pressure may convince people to stay in classroom course. Online courses - no such social pressure. Easier to leave.

Predicting attrition
- Tech difficulties in module 1 - 10% increase in attrition
- Tech difficulties did not predict attrition in later modules.
- Wound up not being how many modules had errors - but where the errors were that predicted attrition. All modules with errors had 6 errors.
- TEST SCORES predicted attrition in modules 2-4.
- Attrition was 18% higher for trainees with low test scores in the previous module.
- Test scores 7% lower in modules with technical difficulties.
+ We can influence by an entire letter grade as a result of technical difficulties
- The people who eventually dropped out were substantially impaired with the tech difficulties. Ones who completed the course had a buffer - not impared by the technical difficulties.

Negative thoughts on learning
- When trainees encounter tech difficulties - level of neg thoughts increase
- Negative thoughts impair learning
+ for every 1 point increase in neg thoughts, test scores 7% lower
- Test scores predicted negative thoughts in subsequent module
- Big spiral effect!!!!
- Negative thoughts - self-reported. Questions like "I feel anxious"

Among those who complete the course - neg thoughts not affected by test scores.
- Low test scores result in higher levels negative thoughts.
- There was a buffering among the 19% completed.

TEchnical difficulties impair learning, increase negative thoughts
- Cyclical relationships low test scores, hight
- Attrition - Module 1 tech difficulties, Module 2-4 test scores

- Internet skills courses and tech support
- Trainees must have reliable internet connection
- Complete training in quiet environment, free from distractions [maybe the distraction-free computer lab is a good idea.

Prompting was an interruption also. Found very different effects.

Paying for a course, or have some extrinsic motivator - determine drop-out rate.

1 comment:

Anonymous said...

It won’t be wrong to say that the IT sector has made the world stand up and take notice of the countries like India. India’s IT poweress is one reason why such massive deals like the Tata Chorus deal and the Hindalco Novelis deal could get shape and turn into India’s favor. Had it not been for the IT sector, there were chances that these multi million dollar deals would not have matured the way they have.