In the proceedings of the International Conference

 on Artificial Intelligence in Education, July 19-23, 1999

 Le Mans - France.

 

 

A Methodology for Building Intelligent Educational Agents

 

 

Harry N. Keeling

 

 

Department of Systems and Computer Science

Howard University

2300 6th Street, NW, Washington, DC 20059

Phone: (806) 806-4830, hkeeling@scs.howard.edu

 

Abstract. The acquisition of domain knowledge has traditionally been recognized as a major impediment to the development of knowledge-based systems. In particular, this “knowledge acquisition bottleneck” has significantly slowed the development of intelligent educational software where the domain experts are teachers with little or no experience with knowledge engineering.  However, recently there has been a growing recognition of the potential benefits that can be derived from the application of advances in the AI-related fields of machine learning and knowledge acquisition. In this paper, we discuss a full life cycle methodology for building educational agents that can be used in classrooms.  This innovative approach synergistically combines methods from machine learning and knowledge acquisition, concepts from intelligent tutoring research and advances in skill assessment from educational research. The result has been a demonstrated methodology for building educational agents that utilizes a multistrategy apprenticeship learning approach, a multi-dimensional evaluation phase and a customized software toolkit that facilitates its application.  This methodology has been successfully applied to develop three educational agents that act as indirect communicational channels between the educator and the student. Applied in the areas of American history and statistics, these agents generate test questions and provide tutoring through hints and explanations. One agent has been used to enhance the functionality of an existing educational software package, while the other two have been used in a stand-alone mode.  This research demonstrates solutions to the problems involved in building intelligent educational software and prescribes a new approach that draws from the fields of artificial intelligence and educational research.

 

 

1        introduction

 

            This paper presents a methodology for building intelligent educational agents through the integration of machine learning and knowledge acquisition. Based on a multistrategy machine learning approach called Disciple [9], this methodology reduces the involvement of the knowledge engineer as well as the time required from the domain expert. A claim of this research is that this methodology can be naturally employed to (1) build agents that are capable of adding intelligent features to educational software and (2) build stand alone agents with their own interfaces that are capable of assisting both teachers and students.  The following pages discuss this methodology by first describing the roles of each participant in the effort and then outlining the agent development process.

 

 

 

 

 

 

2        OVERVIEW OF METHODOLOGY

 

            An overview of the Disciple full life cycle agent building methodology is presented in Figure 1. This figure illustrates the stages of the agent as it is developed and the roles of the three “stakeholders” in the agent development process: the domain expert/teacher; the knowledge engineer/developer; and the user(s) of the agents.

User
 

Developer/

Knowledge Engineer
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


                                          Figure 1:  Overview of the Disciple agent building methodology

 

            Depending upon the complexity of the application domain, the expert may require the assistance of a separate developer (a software/knowledge engineer). The expert and the developer work closely to determine requirements and to build the agent. Broken lines indicate where the agent development process may require some iterative development efforts, due to changes in the application domain or in the requirements for the agent after the agent has been developed. These iterative development efforts may include the refinement of the agent’s knowledge or the further development of the agent’s interfaces and/or problem solving algorithms.

            To begin the agent building process, the agent developer in cooperation with the domain expert must customize the Disciple Learning Agent Shell [10]. This shell contains components for basic knowledge acquisition and learning, problems solving and knowledge base management. The customization of this shell requires the development of several domain dependent software modules.  These modules may include specific interfaces that operate on top of the shell’s domain independent graphical user interface.  These domain specific interfaces allow the domain expert to express his/her knowledge in a manner that is both natural and appropriate for the application domain.  Further, these interfaces can be used by the agent to communicate with the expert in a similar manner. The domain expert and the agent developer also decide on the nature and extent of a domain specific problem solver, based upon the type and purpose of the agent to be developed. Subsequently, the agent developer continues the customization of the agent shell with the development of the domain-specific problem solver. The software/knowledge engineer may need to develop the agent’s domain specific problem solving component by extending the existing problem solving components of the Disciple Learning Agent Shell. Other domain dependent software modules may include conversion routines to be used to import existing knowledge bases from other sources. The result of this effort is a customized Learning Agent Shell with learning capabilities and a generic problem solving component, but without a knowledge base.

            While the learning agent is under development, it resides in the customized Disciple Learning Agent Shell that is utilized by the expert during knowledge elicitation, rule learning, and rule refinement processes. At this stage, the domain expert develops the agent’s initial knowledge base and teaches it domain specific tasks. If the agent is to be his or her assistant, the expert both teaches the agent and then uses it.  If the agent is to be used by a non-expert user, the expert will initially teach the agent then release the trained agent to the user. 

            The role of the domain expert is to train the agent once the customized agent shell has been built. During this dialog, the expert interacts directly with the agent to define an initial knowledge base by engaging in a knowledge elicitation dialog in which the agent guides the expert to provide whatever knowledge he or she can easily express.  Figure 2 below presents the Concept Editor and Concept Browser windows along with two domain specific interfaces.  These interfaces are utilized to build the initial knowledge base.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 


Figure 2: Concept Editor and Concept Browser windows along with two domain specific interfaces

 

            After this initial knowledge base has been developed, the expert teaches the agent how to perform domain specific tasks using the Disciple approach. Disciple is an apprenticeship, multistrategy approach for developing intelligent agents [1].  Central to this approach is the creation of a man-machine dialog where an expert teaches the agent how to perform domain-specific tasks.  This agent training is performed in a way that resembles the manner in which an teacher would teach an apprentice, by giving the agent examples and explanations as well as by supervising and correcting its behavior [9].  The expert teaches the agent how to solve certain types of problems by providing a concrete example, helping the agent to understand the solution, supervising the agent as it attempts to solve analogous problems, and correcting its mistakes.

            In this approach the agent learns its behavior from its teacher by integrating several machine learning and knowledge acquisition techniques and by taking advantage of their complementary strengths to compensate for their individual weaknesses [2, 8, 11].  Through only a few such natural interactions, the agent is guided to learn complex problem solving rules and to extend and correct its knowledge base. This process is based on a cooperation between the agent and the human expert in which the agent helps the expert to express his/her knowledge using the agent’s representation language, and the expert guides the learning actions of the agent. As a consequence, the Disciple approach significantly reduces the involvement of the knowledge engineer as well as the time required from the domain expert in the process of building an intelligent agent. At this point, this initial knowledge base of the agent, built mainly through knowledge elicitation techniques, is assumed to be incomplete and partially incorrect.

            It is expected that the agent will require maintenance during its lifetime.  That is, the agent may need to be retrained and possibly redeveloped.  The process of retraining and redevelopment does not differ from the agent’s initial training and development. Additions could be required for the agent’s knowledge in the form of new concepts, instances, and features.  Also, the existing rule base may require refining and/or new rules may need to be learned. These new customization and knowledge requirements may be detected by the expert during the initial learning sessions or later, by the user, after the agent has been put to use.  Additional knowledge requirements may be revealed during the agent testing and evaluation phases. 

            During the agent testing and evaluation phase, the agent developer tests the agent in much the same way that traditional software systems are tested. However, as with knowledge-based systems like a Disciple agent, the expert and the users must also assess the adequacy of the agent’s knowledge and evaluate agent performance. These evaluation activities focus on measuring the completeness and correctness of the agent’s knowledge base.  Several evaluation experiments are an integral part of this methodology. Two types of experiments are necessary to measure the predictive accuracy of the agent’s knowledge base [6]; one with respect to the training expert and another with respect to a independent domain expert. These studies measure the ability of the Disciple approach to acquire the expertise of the training expert, while the other seeks to determine how well the agent has acquired general knowledge in its problem domain. In other experimental studies, both the domain expert and the agent’s potential users provide subjective survey-based evaluations. These metrics reveal specific performance aspects of the trained agent that can indicate the need for redevelopment and/or the need for additional training as illustrated in Figure 1. After the agent’s knowledge and performance are verified and validated, an agent capable of operating in a given domain is released for use. The next section describes the major phases of the Disciple agent building methodology in more detail.

 

 

3        PHASES OF AGENT DEVELOPMENT

 

            The Disciple agent development methodology is an application of software engineering principles to the orderly transformation of the needs of teachers and students into a working educational agent.  It also includes an optional phase for the integration of this agent into a target educational software package.  Moreover, this methodology includes the verification, validation and maintenance of this agent until the end of its useful life. This methodology can be broadly characterized as an iterative analysis, design and development effort. These phases of this methodology are: (1) analyzing the problem domain, defining agent requirements and defining the top level ontology of the agent’s knowledge base; (2) designing domain dependent modules, the agents task structure and problem solver; (3) customizing the Disciple shell, building the initial knowledge base and problem solver, and teaching the agent how to generate tests; (3) developing the agent with a problem solving engine and a graphical user interface; and finally (4) verifying, validating and maintaining the agent. These phases are graphically presented in Figure 3.

            The process of designing, building and integrating intelligent educational agents is comprised of four phases. The remainder of this section describes these phases within the context of the traditional development phases of software engineering: