Metacognitive Tutoring Framework For Complex Decision Making Tasks
Developing a Metacognitive Tutoring Framework
In this project, we developed a metacognitive tutoring framework within the GIFT (Generalized Intelligent Framework for Tutoring) architecture to address US Army challenges in providing computer-based training environments that adapt to the learner’s competence and state. In this framework, our focus is primarily on enhancing adaptive support during learning and problem-solving that identifies and remediates deficiencies in both cognitive skills and strategies. Our goal is to create more detailed user models in the Generalized Intelligent Framework for Tutoring (GIFT) system, to support analysis of users’ cognitive, strategic, and metacognitive processes as they work on learning or training scenarios.
We implemented the Metacognitive Tutoring Framework in GIFT system and validated its performance using UrbanSim training environment [1]. UrbanSim is a PC-based virtual training application for practicing the art of mission command in complex counterinsurgency and stabilization operations.
Research Work Done:
We proposed a comprehensive Metacognitive tutoring framework to detect and provide support on learner’s cognitive skills and strategies. Our framework consists of two primary models a) multi-level hierarchical learner model to detect learner’s proficiency in cognitive skills, and cognitive strategies and b) instructions strategies algorithm for multi-level learner model to provide feedback.
Multi-level Hierarchical learner Models
We developed a three-level learner modeling framework for analyzing and representing learners’ proficiency in cognitive skills and strategies. The analysis in the learner model involves making a set of inferences based on learners’ behaviors observed while using the learning environment and their performance on learning goals in the learning environment. The bottom layer of our user model derives information from logs of user activities collected from the training environment (e.g., UrbanSim). The middle layer of our user-modeling framework captures learners’ proficiency on problem solving strategies, which represent meaningful sequences (combinations) of cognitive skills, and provide the link to higher level tasks and goals that the user is trying to accomplish e.g., clear the area of insurgents to help establish local governance. The top layer captures learner’ proficiency on metacognitive processes that consider the learners’ intent in using a skill or strategy, as well as their monitoring, evaluation, and reflection behaviors as they work on the system.
Instructional Strategies for Multi-level Learner model
Our instructional strategy algorithm builds on hierarchical user model structure and the performance metrics that we derive for each level. The algorithm analyzes the performance metrics top-down on the hierarchical structure, starting from the metacognitive level. It checks the competence and trend values of its child nodes (i.e.), the strategy nodes and picks the child node that has the least competence and a negative or flat trend. Then the algorithm repeats the analysis on its child nodes (cognitive skills). If the user is deficient in cognitive skills (i.e., their performance values for cognitive skills are below a threshold), the algorithm will pick one of the cognitive skills for feedback, otherwise, it focuses on the higher-level selected strategy for feedback. If the user has shown sufficient proficiency at the strategy level, then the algorithm selects a metacognitive process node for feedback.
Research Studies
We conducted our first research study (2015) with Initial Prototype of Metacognitive Tutor at the ROTC program, Vanderbilt University. 14 RTOC program seniors worked in pairs on GIFT and UrbanSim. We collected pre- and post-survey data, log files from UrbanSim and GIft and data form screen-capturing software Camtasia.
In the second year, 2016, we conducted research studies with learner model to detect domain dependent cognitive strategies at ROTC Program, Vanderbilt University. The data similar to year 1 studies were collected and analyzed.
In the third year, 2017, we validated our metacognitive tutoring framework using two research study. The first study was conducted using students from “War in Iraq” class. The second study was conducted at ROTC Program, Vanderbilt University.
Researchers involved:
Vanderbilt University: Prof Gautam Biswas, Ramkumar Rajendran, and Naveed Mohammed
Army Research Lab, Orlando: Benjamin S. Goldberg, Keith Brawner, and Robert Sottilare
Former Researchers:
Vanderbilt University: John Kinnebrew, James R. Segedy, and Michael Tscholl
References:
Funding agency: Army Research Laboratory
Grant period: May 2014 to Sep 2017