Intrinsic motivation (artificial intelligence)

Intrinsic motivation in the study of artificial intelligence and robotics is a mechanism for enabling artificial agents (including robots) to exhibit inherently rewarding behaviours such as exploration and curiosity, grouped under the same term in the study of psychology. Psychologists consider intrinsic motivation in humans to be the drive to perform an activity for inherent satisfaction – just for the fun or challenge of it.^[1]

Definition

An intelligent agent is intrinsically motivated to act if the information content alone, or the experience resulting from the action, is the motivating factor.

Information content in this context is measured in the information-theoretic sense of quantifying uncertainty. A typical intrinsic motivation is to search for unusual, surprising situations (exploration), in contrast to a typical extrinsic motivation such as the search for food (homeostasis).^[2] Extrinsic motivations are typically described in artificial intelligence as task-dependent or goal-directed.

Origins in psychology

The study of intrinsic motivation in psychology and neuroscience began in the 1950s with some psychologists explaining exploration through drives to manipulate and explore, however, this homeostatic view was criticised by White.^[3] An alternative explanation from Berlyne in 1960 was the pursuit of an optimal balance between novelty and familiarity.^[4] Festinger described the difference between internal and external view of the world as dissonance that organisms are motivated to reduce.^[5] A similar view was expressed in the '70s by Kagan as the desire to reduce the incompatibility between cognitive structure and experience.^[6] In contrast to the idea of optimal incongruity, Deci and Ryan identified in the mid 80's an intrinsic motivation based on competence and self-determination.^[7]

Computational models

An influential early computational approach to implement artificial curiosity in the early 1990s by Schmidhuber, has since been developed into a "Formal theory of creativity, fun, and intrinsic motivation”.^[8]

Intrinsic motivation is often studied in the framework of computational reinforcement learning^[9][10] (introduced by Sutton and Barto), where the rewards that drive agent behaviour are intrinsically derived rather than externally imposed and must be learnt from the environment.^[11] Reinforcement learning is agnostic to how the reward is generated - an agent will learn a policy (action strategy) from the distribution of rewards afforded by actions and the environment. Each approach to intrinsic motivation in this scheme is essentially a different way of generating the reward function for the agent.

Curiosity vs. exploration

Intrinsically motivated artificial agents exhibit behaviour that resembles curiosity or exploration. Exploration in artificial intelligence and robotics has been extensively studied in reinforcement learning models,^[12] usually by encouraging the agent to explore as much of the environment as possible, to reduce uncertainty about the dynamics of the environment (learning the transition function) and how best to achieve its goals (learning the reward function). Intrinsic motivation, in contrast, encourages the agent to first explore aspects of the environment that confer more information, to seek out novelty. Recent work unifying state visit count exploration and intrinsic motivation has shown faster learning in a video game setting.^[13]

Types of models

Ouedeyer and Kaplan have made a substantial contribution to the study of intrinsic motivation.^[14][2][15] They define intrinsic motivation based on Berlyne's theory,^[4] and divide approaches to the implementation of intrinsic motivation into three categories that broadly follow the roots in psychology: "knowledge-based models", "competence-based models" and "morphological models".^[2] Knowledge-based models are further subdivided into "information-theoretic" and "predictive".^[15] Baldassare and Mirolli present a similar typology, differentiating knowledge-based models between prediction-based and novelty-based.^[16]

Information-theoretic intrinsic motivation

The quantification of prediction and novelty to drive behaviour is generally enabled through the application of information-theoretic models, where agent state and strategy (policy) over time are represented by probability distributions describing a markov decision process and the cycle of perception and action treated as an information channel.^[17][18] These approaches claim biological feasibility as part of a family of bayesian approaches to brain function. The main criticism and difficulty of these models is the intractability of computing probability distributions over large discrete or continuous state spaces.^[2] Nonetheless, a considerable body of work has built up modelling the flow of information around the sensorimotor cycle, leading to de facto reward functions derived from the reduction of uncertainty, including most notably active inference,^[19] but also infotaxis,^[20] predictive information,^[21][22] and empowerment.^[23]

Competence-based models

Steels' autotelic principle ^[24] is an attempt to formalise flow (psychology).^[25]

Achievement, affiliation and power models

Other intrinsic motives that have been modelled computationally include achievement, affiliation and power motivation.^[26] These motives can be implemented as functions of probability of success or incentive. Populations of agents can include individuals with different profiles of achievement, affiliation and power motivation, modelling population diversity and explaining why different individuals take different actions when faced with the same situation.

Beyond achievement, affiliation and power

A more recent computational theory of intrinsic motivation attempts to explain a large variety of psychological findings based on such motives. Notably this model of intrinsic motivation goes beyond just achievement, affiliation and power, by taking into consideration other important human motives. Empirical data from psychology were computationally simulated and accounted for using this model.^[27]

Intrinsically Motivated Learning

Intrinsically motivated (or curiosity-driven) learning is an emerging research topic in artificial intelligence and developmental robotics^[28] that aims to develop agents that can learn general skills or behaviours, that can be deployed to improve performance in extrinsic tasks, such as acquiring resources.^[29] Intrinsically motivated learning has been studied as an approach to autonomous lifelong learning in machines^[30][31] and open-ended learning in computer game characters.^[32] In particular, when the agent learns a meaningful abstract representation, a notion of distance between two representations can be used to gauge novelty, hence allowing for an efficient exploration of its environment.^[33] Despite the impressive success of deep learning in specific domains (e.g. AlphaGo), many in the field (e.g. Gary Marcus) have pointed out that the ability to generalise remains a fundamental challenge in artificial intelligence. Intrinsically motivated learning, although promising in terms of being able to generate goals from the structure of the environment without externally imposed tasks, faces the same challenge of generalisation – how to reuse policies or action sequences, how to compress and represent continuous or complex state spaces and retain and reuse the salient features that have been learnt.^[29]

See also

• reinforcement learning
• markov decision process
• motivation
• predictive coding
• perceptual control theory

References

This page was last edited on 27 November 2023, at 22:10