Client: International Health Organization

Virtual Reality Training Effectiveness Study

Client
World Health Organization
Date
May 21, 2022
Services
VR Design, Emergency Triage Training
Website

Working with the WHO Academy team on VR training was an exciting experience. The need to train hundreds on emergency triage protocols is a top priority at the United Nations, and the World Health Organization’s Academy supports that effort. Being able to scale 2-3 doctors to train hundreds around the globe was the challenge. Virtual Reality looked like a viable option. But despite the depth of doctors, no one on the team had a background in learning science.

And yet working across 7 time zones and four languages can be challenging as well. Ultimately, I was able to discuss and discover the learning obstacles in utilizing and building from scratch a VR triage application. Surprisingly, it was my work with with the WHO that lead me to see the benefit in merging backward design with spiral software development.

Compelling Event: Global Demand and limited Resources to Scale

The brief section provides a concise summary of the learning design project, including the objectives, target audience, and key deliverables. It sets the stage for the rest of the portfolio by outlining the scope and purpose of the project.

UN-WHO learning design project plan
UN-WHO learning design project plan

Target: The WHO Academy VR team included emergency care doctors, software developers, academics, and third-party software engineers and designers. A global team that needed to be integrated with the WHO Academy LMS.

Objectives: Develop a VR Triage training application to solve the logistics of sending UN-WHO trained doctors to across the globe and to regions with limited training facilities. Provide a universal virtual reality platform for support personnel to help in emergency situations.

Deliverables: Despite the advanced experience of doctors and academics. No one on the team had a degree in education or learning science.Therefore, the team was looking for guidance to ensure the highest standards of training in a virtual training platform would be achieved.

This section delves into the theoretical foundations and principles of learning science that inform the design. It includes:

  • Theories and Models: Explanation of the learning theories and models applied (e.g., Embodied Cognition, Survival Advantage, Situated Learning).
  • Research and Evidence: Summary of relevant research and evidence that supports the chosen approach.
  • Learner Analysis: Insights into the characteristics, needs, and preferences of the target audience.
Theoretical foundations to Learning Science approaches
Theoretical foundations to learning science of design

Research and Evidence: In working with the WHO VR team, my learning science approach was to look at stress levels, technology constraints, and what cognitive information processing models might help increase retention and transfer of skills.

Theories and Models: Due to the immense pressure combined with the life and death consequences. It was clear that dual processing, cognitive load, and issues of survival processing would be at play. Additionally, due to the importance of motion and presence embodied cognition as well as social learning theories such as situated learning must be considered.

Learning Analysis: Doctors in mass casualty incidents (MCI) need to use instincts and react to various stimuli and attempt to make the best decisions. In conversations with both emergency doctors and software developers, it was clear there were gaps in understanding the needs of both to bridge the transfer of skills via virtual reality training application.

Here, the focus is on the practical steps taken to design and develop the learning materials. This includes:

  • Instructional Design: Detailed description of the instructional strategies and methods used.
  • Content Development: Process of creating and organizing the content.
  • Technology Integration: Tools and platforms used to deliver the learning experience.
  • Prototyping and Testing: Iterative process of creating prototypes and testing them with users.
Dr. Kaufman's learning materials development platform
Learning materials development platform

Instructional Design: The key component to looking at how to improve the learning and development of the VR training application was to focus on both the affective and psychomotor domains of learning. The trainees would be highly trained doctors, but the skills needed to triage many patients in a mass casualty incident required to look at information processing standards as well as emotional stability before and after training in the field.

Technology Integration: The project had many influences and sponsors. This included both Samsung and Facebook’s Meta. That combined with the effort of the developers to keep the software open source created tensions in how the technology would be used and what systems including the WHO LMS API’s to integrate with.

Content Development: The content was non-standard, as the methods of triage where unique to the subject matter experts (SME). This unique system required considerable re-tooling of the game engine to score correct triage decisions. Additionally, graphics and models were supplied by a third-party vendor donated by a Fortune 100 company. Demands on consistency were considerable and influenced the learning priorities and features of the learning to collaborate with facilitators.

Prototyping and Testing: Due to the fact that the SME’s were also the key stakeholders, iterative design was a constant. This was both a benefit but also a drawback. A benefit in that it delivered constant feedback to design decisions. A drawback because constant scrutiny slowed down minor details that were not yet finished product.

This section evaluates the effectiveness of the learning design and its impact on the learners. It includes:

  • Assessment Methods: Description of the assessment tools and techniques used to measure learning outcomes.
  • Data Collection and Analysis: Methods for collecting and analyzing data to evaluate the impact.
  • Results and Findings: Summary of the evaluation results and key findings.
  • Impact on Learners: Discussion of how the learning design has affected the learners’ knowledge, skills, and attitudes.
Learning evaluations through instructional design
Learning evaluations through instructional design

Learning Impact: To measure impact of development progress and effectiveness, I interviewed a dozen VR boutique firms. This included Medical VR, Emergency VR, Surgical VR as well as general academic and industrial VR development firms. I compared these with the short polls with internal team members as well as progress reports and testing. By using both internal and external experts, the final reports and advisement had immensely more impact.

Assessment Tools: Through conversational analysis using Jeffersonian notation and AI mind mapping of keywords. I studied over 5,000 words in various contexts and discussions. Additionally, progress with development was compared to external sources anonymously to benchmark the effort and output of the WHO team.

Data Analysis: The analysis showed that obstacles were created by not having a clear understanding of learning objectives and in the field skill attainment. This was compounded by various stakeholders, different cultures and incompatible technology stacks and deployment methodologies.

Evaluation Results: My engagement with the WHO only lasted 1/2 a year. But it became clear that the team dysfunction as well as the limited resources were a drag on the team. Despite several successful prototypes, the project was scrubbed after the submission of my report.

Reflecting on the project, this section highlights the key lessons learned throughout the process. It includes:

  • Challenges and Solutions: Description of the challenges faced and how they were addressed.
  • Successes and Best Practices: Identification of what worked well and best practices for future projects.
  • Recommendations for Improvement: Suggestions for improving the design and development process in future projects.
Key lessons learned during the entire process
Key lessons learned during the entire process

Challenges: Working with a global health organization with an immense bureaucracy, across multiple time zones, and several languages provides immense obstacles. This combined with limited resources for a pilot VR project, limited the acceleration of the project goals. And no learning science perspective.

Best Practices: The key is to align all stakeholders to the core input, activities, outputs and outcomes needed to succeed. With global projects on cutting edge technology, transparency is essential. Setting performance standards for all parties and their output is necessary to eliminate team drag.

Recommendations: Set performance standards on key deliverables. Provide transparent monitoring or progress with Key Progress Indicators. The project generated a quantitive radar map to aid in deciding whether training should be placed in Virtual Reality or not. (see Artifacts).

I am text block. Click edit button to change this text. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.

The final section showcases the tangible artifacts and elements that make up the portfolio. This includes:

  • Sample Materials: Examples of instructional materials, such as lesson plans, multimedia content, and assessments.
  • Design Documents: Documentation of the design process, including storyboards, wireframes, and design specifications.
  • Evaluation Reports: Reports summarizing the evaluation findings and impact assessments.
  • Learner Feedback: Testimonials and feedback from learners who participated in the project.
Tangible artifacts and elements that make up the portfolio
Tangible artifacts and elements that make up the portfolio

Links:

VR Report on Learning Evaluation
Tools: Unity, Word, Gimp Photo…
Micro-course on Dual Coding for Interactive and VR training.
Tools: Articulate, Final Cut, FileZilla, HeyGen-AI…
Process Report for VR training
Tools: Scrivener, Grammarly, Wizard.AI…

Looking for a First-Class Learning Science Consultant?