2 NDHB-Model/RT: Nonlinear Dynamic Human Behavior Model with Realtime Constraints – Memory and Action Selection in Human-Machine Interaction

NDHB-Model/RT: Nonlinear Dynamic Human Behavior Model with Realtime Constraints

The purpose of this chapter is to integrate the three fundamental constructs of O-SCFT (MSA, BIH and SMT), briefly introduced in Chapter 1, into a unity. The following sections provide more detailed explanations of the fundamental constructs, then define the Nonlinear Dynamic Human Behavior Model with Realtime Constraints (NDHB-model/RT).

2.1. Maximum satisfaction architecture

As described in section 1.1.3, MSA is about the realization of the purpose of living, libido – it maximizes efforts on the autonomous system. It deals with how autonomous systems achieve goals under constraints defined by BIH and SMT [KIT 07].

MSA consists of the following three parts:

  1. 1) happiness goals, i.e. basic living purposes of human beings;
  2. 2) human brain;
  3. 3) society.

2.1.1. Happiness goals

MSA assumes that the human brain pursues one of the 17 happiness goals defined by Morris [MOR 06] at every moment, and switches when appropriate by evaluating the current circumstances. Table 2.1 lists each of the happiness goals along with the name of type associated with it, and describe their relationships to social layers.

Table 2.1. Happiness goals and their relation to social layers. + denotes the degree of relevance of each goal to each layer, i.e. Individual, Community, and Social system, respectively. +++: most relevant, ++: moderately relevant, and +: weakly relevant

  happiness Types Individual level Community level Social system level
1 Target happiness The achiever +++ +++ +++
2 Competitive happiness The winner +++ +++
3 Cooperative happiness The helper +++ +++
4 Genetic happiness The relative +++ +++
5 Sensual happiness The hedonist +++ +++
6 Cerebral happiness The intellectual +++ +++ ++
7 Rhythmic happiness The dancer +++ +++
8 Painful happiness The masochist +++
9 Dangerous happiness The risk-taker +++ ++ +
10 Selective happiness The hysteric +++ ++ +
11 Tranquil happiness The mediator +++
12 Devout happiness The believer +++ ++
13 Negative happiness The sufferer +++ ++
14 Chemical happiness The drug-taker +++
15 Fantasy happiness The day-dreamer +++
16 Comic happiness The laughter +++ +++
17 Accidental happiness The fortunate +++ +++ +++

2.1.2 Society layers

Each of the happiness goals is associated with one or more layers of society: individual, family and community, and administration and enterprise. These layers have evolved from the history of human beings. Each layer is associated with its own value reflecting historical development, and thus different sets of happiness goals are relevant.

2.1.3 Brain layers

The knowledge necessary to achieve the happiness goals is partly acquired and partly inherited, which is stored in three brain layers1. At the level of the conscious state layer, knowledge, such as formal laws and social mechanisms necessary to deal with administration and enterprise, and formal social norms and common sense to deal with “family and community” and “individual”, is acquired. In contrast, knowledge such as basic functions for using language and primitive decision characteristics is inherited. Similarly, at the level of autonomous-automatic behavior control layer, knowledge such as individual experience and habit is acquired to deal with “family and community” and “individual”. However, as opposed to the inherited knowledge at the conscious state layer, all basic functions that are reproducible by the development and bodily experience are inherited in the autonomous-automatic behavior control layer.

Figure 2.1 depicts the entire relationships among happiness goals, society layers and brain layers. At every moment, an organism tries to achieve one of the happiness goals. Each happiness goal is associated with the nonlinear three-layered structure of society. The goal is achieved by using the nonlinear three-layered structure of brain: whose detail has been affected by the nonlinear three-layered structure of society on the one hand, and whose detail would impose strong constraints on how the goal is accomplished in the ever-changing environment on the other hand. It is assumed that by accomplishing the goal, the organism experiences satisfaction. The individual layers in society and in the brain are mutually related. The mechanism is complicated, but in order to understand human beings’ behavior in the ever-changing world, it is extremely needed to consider the relationships among happiness goals, society and the brain, not independently but as a whole entity.

In addition, the pieces of knowledge at each layer in the brain and the society are nonlinearly interconnected through individual experience. Nonlinearity means two important things that affect development of an individual’s brain–society system:

  1. Dissipative system: a fluctuation of the system caused by an environmental change would trigger creation of a new order or catastrophe;
  2. Sensitive dependence on initial condition (SEDIC): a small variation in the initial condition during one’s infant period would develop exponentially as one grows up.

Figure 2.1. Maximum satisfaction architecture (MSA)

This implies that individuals that pursue the same goal might have different patterns of activated networks because of SEDIC, and thus the processes to achieve the goal might be different. An intelligent autonomous agent incorporated with human–machine interactions must be sensitive to the individual differences in the processes to achieve a goal and provide sophisticated support for individuals to achieve that goal. This has a deep implication to the main theme of this book “know the users”, because this provides an important hint for the way in which to conduct “know the users” endeavors.

2.1.4. Conditions to make people feel satisfaction

The amount of satisfaction felt by a user is influenced by the factors that characterize the shape of trajectory of behavioral outcome. There are six critical factors to make people feel satisfaction (see Figure 2.2).

Figure 2.2. Conditions to make people feel satisfaction

  1. 1) Change: perceptual functions work by sensing dynamic changes. Therefore, responses while the system is stable are limited. A condition for feeling satisfaction is “change”;
  2. 2) Succession of good results: successive happiness tends to create memory traces for the best experience and the final outcome of the overall estimation of the events that have lead to successive good results;
  3. 3) Direction of absolute outcome (denoted as 2 in Figure 2.2): a change to good direction at the end of a series of events tends to create a memory trace of having felt satisfaction. The degree of the strength of the memory trace would be proportional to the degree of the change toward good direction;
  4. 4) Amplitude of success (denoted as 1 in Figure 2.2): the greater the difference between the highest event and the lowest event in terms of the degree of the strength of satisfactory feeling, the stronger the strength of a memory trace for the entire events, including the highest and the lowest. We create feelings of satisfaction as a result of overcoming the lowest evaluated situation and also tend to memorize these;
  5. 5) Absolute amount of outcome (denoted as 3 in Figure 2.2) and direction of absolute outcome (denoted as 2 in Figure 2.2): when the absolute outcome is acceptable and the contents in working memory at the time of final event are good, they jointly affect the result of estimation of entire events;
  6. 6) Bad results would not be memorized: when an event occurs that results in bad results, we strongly react to it if the degree of badness exceeds a certain threshold value. This event creates a memory trace for a must-avoid event. However, memory traces for events that are exerted while recovering from the bad situation tend to be weak because conscious processes work in their full performance.

2.2. Brain information hydrodynamics

Constraints from the environment shape how the flow of information develops along the time dimension. This is reflected in the brain as BIH. It deals with information flow in the brain and its characteristics in time [KIT 08]. At present, there is no research method for viewing the brain from a broad perspective. We suggest that theories of complex systems such as fluid are useful. We therefore propose BIH as a theory that should serve as a basis for constructing a unified theory of action selection and memory, which is traditionally conceived as an electronically based neuronal network and/or chemically based hormone field. In BIH, the influx of information from the environment is filtered at the entrance of the brain to reduce the amount of information to a tractable number of chunks. The influx flows along the terrain, which was originally shaped by genes and then transformed through experience. Immediate behavior is generated when the influx reaches the cerebellum directly. Deliberate behavior, i.e. the outcome of the cerebrum, is generated when the influx is trapped midway to the cerebellum where a number of vortices are created to transform the values of attributes of the information that the influx conveys successively to the ones finally exerted. The real-time constraint of behavior is satisfied by creating emotional vortices, as will be described in section 2.2.4 in detail, that force the flow to reach the cerebellum in a timely manner.

BIH deals with information flow in the brain, as shown in Figure 2.1, as a nonlinear three-layered structure and its characteristics in the time dimension. Biological activity can be viewed as the results of information flow in the brain and it shows the characteristics of complex systems and dissipative structure. In other words, it is best characterized by hydrodynamics at the microscopic phenomenological level and by thermodynamics at the macroscopic collective level. Table 2.2 summarizes the features of the theories.

Table 2.2. Biological Activity: Complex Systems and Dissipative Structure

Characteristics Theory
Macroscopic level Collective Thermodynamics
Microscopic level Phenomenological Hydrodynamics

2.2.1 The time axis is central to information flow

Time is the fourth dimension of our four-dimensional physical universe. However, unlike the X, Y and Z dimensions, it is not symmetric; in other words, it is not reversible. The order of our universe is being shaped as the interactions between life and the surrounding environment and develops along the one-directional time dimension. The characteristic times of brain information processing ensure sustainability of those interactions.

The functioning brain is the result of the working of a huge network of 20 billion nerve cells and synapses. It basically converts input signals from the environment to information that is necessary for acting in real time. However, the phenomena that the flow of information in the brain causes are extraordinarily complex. We suggest that this is analogous to the complexity of the phenomena exhibited by a flow of fluid and that it is useful to apply the construct of the theory of hydrodynamics metaphorically to the phenomena of information flow in the brain.

2.2.2 Cerebrum formation process

In the very early days, organisms created cerebellum-like feed-forward networks, as shown in Figure 2.3. They were most suitable for generating prompt responses to the occurrence of libido, which is the free creative energy an individual has to apply to personal development. Those networks enabled the organisms to perform the required sequence of actions very smoothly: collecting information from the external environment, taking actions for satisfying the occurring libido, achieving it and, finally, returning to the resting state.

Figure 2.3. Interaction between brain and environment based on feed-forward control

After developing the cerebellum-like feed-forward networks, organisms then developed the cerebrum. As opposed to cerebellum, the cerebrum is equipped with feedback networks for processing information as shown in Figure 2.4. These networks enabled the organisms to perform complicated information processing that was impossible for cerebellum-like feed-forward networks.

How have the feedback networks developed from the feed-forward networks? Here is our answer. When libido occurs, information from the external environment is gathered via sensory organs, eyes for visual information, nose for olfactory information and ears for auditory information. The set of information originating from the variety of sensors with different modalities constitutes a set of information flows in the brain network. They flow simultaneously and quasi-independently, and are ultimately transformed into the information for generating external actions.

Figure 2.4. Formation of cerebrum

However, the pattern of the flows is very complex because individual flows are not synchronous in time but the set of flows must converge at the time when an action associated with the input is taken. The timing of action is strictly determined by real-time constraints. Some flows may have spare time and have to wait until the other flows are ready to be integrated, or synchronized. While waiting, the flows of information may develop an order that is analogous to vortices in the stream of river. In the brain network, informational vortices may develop, drift, disappear, fission and merge. A vortex can interact with the other vortices. These vortices can be conceived as manifestation of some functions that work as part of feedback control.

2.2.3. Information flows in the brain

The brain consists of three nonlinearly connected layers as its structure and functions enabled by activating part of the structure.

  1. C layer: conscious state layer, including basic functions for using language and primitive decision characteristics, is acquired as formal laws and social mechanisms, and as formal social norms and common sense (the top layer of BRAIN as shown in Figure 2.1);
  2. A2BC layer: autonomous-automatic behavior control layer is acquired as individual experience and habit (the middle layer of BRAIN as shown in Figure 2.1);
  3. B layer: bodily state layer includes all basic functions that are reproduced by development and bodily experience (the bottom layer of BRAIN as shown in Figure 2.1).

Note that basic functions working at the C layer and B layer are inherited from the predecessors. Interlayer connections between the C layer and A2BC layer, and between the A2BC layer and B layer are established through individual experience, which should be different individual by individual and SEDIC should apply.

2.2.4. Emergence of emotion in BIH

Information flows in each layer with its specific purpose as shown by Figure 2.5. In the C layer, information is for predicting the time course of events and for coordinating relationships between the self and others. In the A2BC layer, information is for autonomously and automatically controlling a variety of parts of the body. In the B layer, information is for regulating the bodily state.

Figure 2.5. Information flows in the brain

Vortices emerge in a river when the amount of flow exceeds a certain threshold. Similarly, when the amount of information flow in the brain exceeds its threshold, informational vortices emerge in the network. These vortices correspond to some conscious states.

In BIH, emotions are regarded as the phenomena by which the information flows in the three layers are interrupted in order to take timely actions; in other words, the real-time constraints intervene in the information flows that may not converge and synchronize at the time an action must be taken. The vortices collapse immediately, i.e. conscious thinking terminates in favor of taking timely action.

2.2.5 Biorhythm of information flow

Information flow in the brain has a 1-day cycle. While sleeping, the amount of the flow stays at a minimum level. In the daytime, the flow increases to the maximum level while working intensively. However, the actual amount of flow is determined by the relationships between the state of the external environment and the desire of the self. Metaphorically, the 1-day cycle of information flow is similar to the daily changes in the ebb and flow in a narrow strait where the emergence of vortices depends on the amount of the one-directional tidal stream.

2.2.6 Role of language

The vortices emerge spontaneously when conditions are satisfied. However, as the skill of using language develops, matures and begins to be stably inherited among generations, it starts to work as a trigger for the appearance of vortices, such as pegs which cause turbulent flows that make vortices emerge.

2.2.7 Multiple personality disorder

When making decisions, a number of candidate actions are evaluated for their suitability in the current situation. However, the actions that are actually taken are largely determined by the evaluation performed by the experience-based reward system located at the junction of the cerebrum and the cerebellum. This evaluation process is unconscious.

In the human brain, there naturally coexist multiple personalities. In the cerebrum, there are a number of small-scale networks that serve as elements for defining personality. The combination of the partial elements, which is the result of information flow in the cerebrum, is determined by the reward system, and therefore there is the possibility of emergence of one personality for some situation and another for a different situation. Personality emergence depends solely on the external information that is fed to the brain, the contingency of selection of the route of information flow in the cerebrum, and the nature of the experience-based reward system.

2.3. Structured meme theory

SMT concerns the relational structure that links human beings and the environment, and thereby deals with effective information and the range of propagation [TOY 08]. The recent consensus is that the range of informational inheritance by genes is limited to physical functions and infantile behavior. Human beings need to acquire basic behavioral skills and communicational skills through experience of behaving in the environment. We propose SMT that explains acquisition and development of these skills. SMT consists of action-, behavior- and culture-level memes. These are interconnected nonlinearly and reflect the level of complexity of brain functions that map information in the environment onto internal representations. The mechanism with which the three levels of memes and genes inherit information is analogous to an information system. Genes serve as firmware that mimics behavior-level activities. Action-level memes serve as the operating system that defines general patterns of spatial–temporal behavioral functions. Behavior-level memes serve as middleware that extends the general patterns to concrete patterns. Culture-level memes serve as application tools that extend the concrete patterns to the ones that work in a number of groups of people.

2.3.1. Meme

A meme is an entity that represents the information associated with the object that the brain can recognize. The original term “meme” coined by Dawkins in 1970s [DAW 76] was conceptual and not clearly defined. However, the meme, or the structured meme, in SMT proposed in this paper is defined clearly within the framework of a nonlinear, multilayered information structure that is similar to the structure of living organisms.

A meme is defined as follows. Each object is defined as a set of elements that belong to each layer in a nonlinearly connected multilayered structure. Those elements that are recognizable as proper entities, such as shape, movement and quality, are able to exist as memes (latent memes). These latent memes change to manifest memes when they are fixated as a part of an object or memorized by the other persons as information objects through the experience that the self takes part in. As such, memes exist in the brain not only as entities that correspond to real objects that exist in the environment but also as information objects that are included in the layers the elements of the objects belong to. For human beings, the latter has been constructed by mapping environmental information onto the networks in the brain, which has established the relationships between human beings and their surrounding environment. This explains the emergence of cultural differences among living groups.

The structured meme consists of the following three nonlinear layers:

  1. – action-level memes represent bodily actions;
  2. – behavior-level memes represent behaviors in the environment;
  3. – culture-level memes represent culture.

Memes as a whole are a collection of information objects that reside in each layer. Each person will develop his/her own relationships among objects.

Figure 2.6 depicts the structure of inheritance of information in which genes, memes, and language participate.

2.3.2. Memes propagate by means of resonance

Memes propagate from person to person when the receiver estimates that the degree of reality of the meme perceived by him/her reaches a certain level. The process of feeling reality can be conceived as the process of resonance that occurs in the brain in response to the input of memes from a sender. When the meme in question resonates with some patterns associated with valued experiences endorsed by the reward system, the meme is accepted by the receiver. The entire meme structure in human society is a networked field defined by individuals’ connections. Each person’s brain forms a proper reality field, and it builds up to the entire reality field. Memes propagate in the thus constructed reality field by means of resonance.

Figure 2.6. The structure of meme

Figure 2.7 illustrates how memes propagate in the reality field. The process of propagation is facilitated by symbolization. A symbolized meme enables people to think on abstract levels.

2.3.3. Characteristics of meme propagation

A meme is defined as a matrix-like construct that consists of multiple layers and a number of elements. The feeling of reality that an individual experiences is formed by integrating responses generated by the acceptor elements whose structure is defined similarly to that of the structured meme. However, the response sensitivity of the individual’s acceptor elements is shaped by experience, and thus it exhibits individual differences depending on the individual’s experience.

While a meme is propagating in the network of individuals, the differences in reality responses by individuals also propagate. This implies that the meme may be altered in the propagation process.

Figure 2.7. Propagation of meme

Figure 2.8 depicts the cultural evolution of a meme. It also demonstrates that some amount of fluctuation could appear in the flow of meme quantity and quality because the propagation cannot completely reflect the complexity of the environment.

2.4. NDHB-model/RT2

On the basis of O-SCFT, we have developed NDHB-model/RT as an architecture model that consists of a behavioral processing system and a memory processing system that interact with each other as autonomous systems. The interactions are cyclic, and memory develops and evolves as time goes by. NDHB-model/RT represents consciousness as one-dimensional linear operations, i.e. language, corresponding to System 2 of Two Minds, and unconsciousness including emotion as a hydrodynamic flow of information in multidimensional parallel operations in the neural networks, corresponding to System 1 of Two Minds. NDHB-model/RT has autonomous memory systems that mediate between consciousness and unconsciousness to display the dynamic interactions between them.

Figure 2.8. Evolution of meme

NDHB-model/RT suggests that the brain consists of the following three nonlinearly connected layers. Behavioral decisions and action selections are made by integrating the results of operations of these three layers (repeated from section 2.2.3):

  1. C layer: conscious state layer, i.e. System 2 of Two Minds;
  2. A2BC layer: autonomous-automatic behavior control layer, i.e. System 1 of Two Minds;
  3. B layer: bodily state layer.

The B layer prioritizes the 17 behavioral goals, i.e. happiness types defined by Morris [MOR 06], such as “target happiness for an achiever”, “cooperative happiness for a helper”, “rhythmic happiness for a dancer”, and so on. The other two layers interact with each other in order to derive the next behavior that should satisfy the highest prioritized goal. In normal situations in our daily life, temporal changes in the environment impose the strongest constraint on the decision of the next behavior, and thus the A2BC layer plays a more dominant role than the C layer in organizing behavior. To put it simply, in our daily life we act more by reflex than by reasoning.

The next behavior is determined by extracting objects from the ever-changing environment and attaching values to them according to the degree of the strength of the resonance with what is stored in the autonomic memory system. This is followed by deliberate judgment by using the knowledge associated with the highly valued objects. The former is controlled by the processes in the A2BC layer, System 1; the latter, by the processes in the C layer, System 2.

2.5. MHP/RT: Model human processor with real-time constraints3

NDHB-model/RT can be simulated by the architecture model, model human processor with real-time constraints (MHP/RT) [KIT 12a, KIT 11a, KIT 13]. MHP/RT focuses on synchronization between System 1 and System 2 in the information flow under O-PDP. More specifically, MHP/RT deals with one aspect of working of NDHB model/RT, which is synchronization between conscious system and unconscious system in the ever-changing environment where human beings make decisions and action selections to behave properly.

Figure 2.9 depicts the outline of MHP/RT. It is a real brain model composed of unconscious processes of System 1 and conscious processes of System 2 at the same level. There are two distinctive information flows: System 1 and System 2 receive input from the perceptual information processing system in one way, and from the memory processing system in another way. System 1 and System 2 work autonomously and synchronously without any superordinate-subordinate hierarchical relationships but interact with each other when necessary. In Figure 2.9, solid lines and dotted lines indicate the path associated with System 1 and the System 2, respectively. These two flows are synchronized before carrying out some behavior.

Figure 2.9. Outline of MHP/RT (adapted from our article [KIT 15b], Figure 2). Solid lines indicates information for System 1 based processing and dotted lines indicates information for System 2 based processing. These two flows are synchronized before carrying out some behavior

2.5.1. MHP/RT’s basic flow4

As depicted in Figure 2.9, MHP/RT operates in two bands, the asynchronous band and the synchronous band. The bodily coordination monitoring system and the memory processing system operate in the asynchronous band. The perceptual information processing system, conscious information processing system, autonomous automatic behavior control processing system and behavioral action processing aystem operate in the synchronous band. These systems work autonomously. System 1 of the Two Minds corresponds to the autonomous automatic behavior control processing system, and System 2 corresponds to the conscious information processing system.

Figure 2.10. MHP/RT (adapted from our article [KIT 12a])

MHP/RT works as follows (Figure 2.10):

  1. 1) inputting information from the environment and the individual;
  2. 2) building a cognitive frame in working memory, which resides between the conscious process, System 2, and the unconscious process, System 1, to interface them – depicted between System 1 and System 2;
  3. 3) resonating the cognitive frame with autonomous long-term memory to make the relevant information stored in long-term memory available; cognitive frames are updated at a certain rate and the contents in the cognitive frames are continuously input to long-term memory to make pieces of information in long-term memory accessible to System 1 and System 2;
  4. 4) mapping the results of resonance on consciousness to form a reduced representation of the input information;
  5. 5) predicting future cognitive frames to coordinate input and working memory, corresponding to either decision-making or action selection depending on the time difference between the time when the prediction is made and the time when an event associated with the prediction takes palce, namely, whether the prediction is made mainly by System 2, decision-making, or by System 1, action selection.

The density of information in working memory is the product of the updating rate of the cognitive frame and the degree of fineness of the information represented in the cognitive frame. When the system is under the control of automatic behavior, i.e. under control of System 1, the updating rate of the cognitive frame tends to be high; however, the degree of fineness of the information represented in the cognitive frame is coarse. When the system is under the control of consciousness, i.e. under control of System 2, the updating rate of the cognitive frame and the degree of fineness of the information are flexibly determined by the context.

2.5.2. Basic MHP/RT behaviors

At a given time (T), MHP/RT’s state is viewed by the following two ways:

  1. 1) which part of MHP/RT is working;
  2. 2) which content MHP/RT is processing.

In the following sections, the “which part” question will be discussed in section, and the “for what” question will be discussed in section Four operation modes of MHP/RT5

At a given time (), MHP/RT’s state is considered from the viewpoint “which part of MHP/RT is working”. In MHP/RT as illustrated by Figure 2.10, behavior is the outcome of activities in System 1 and System 2 both of which use working memory to prepare for the next action. Depending on the situation, behavior is driven mainly by either System 1 or System 2. Both systems work synchronously by sharing working memory. The former is called Mode 1, and the latter is called Mode 2. However, in some situations, both work asynchronously, Mode 3, or independently, Mode 4; working memory may be shared weakly or used solely for one of these layers (see Table 2.3):

  1. Mode 1 (System 1 controls behavior): when System 1 governs behavior, the updating rate of the cognitive frame is the fastest and the system behaves unconsciously. The system refers to the memory that is activated via the resonance reaction, and the outcome of behavior is consciously monitored, which is mission of System 2 in this mode. As long as the output of behavior is consistent with the representation of the contents of activated memory, or prediction, no feedback control is applied. No serious decision-making is required but a series of unconscious action selections would result in smooth behavior. An example of this behavior mode is riding a bicycle on a familiar road.
  2. Mode 3 (System 1 and System 2 are weakly coupled): in some cases, it is not necessary to monitor the behavior with high frequency. As a result, System 2 may initiate tasks that are not directly relevant to unconscious behavior. In such a situation, consciousness is free from behavior that is tightly embedded in the environment. For example, while waiting for his/her name to be called in a lobby of a hospital, he/she may read a book. In this case, at the time when his/her name is called, he/she would be able to stand up immediately to start walking to the consultation room. In his/her working memory, the pointer to the action would be kept active while reading a book and waiting for the announcement. This mode is characterized by weak coupling of System 1 and System 2, which means that pieces of information that reside in working memory are shared by System 1 and System 2, and therefore they could trigger the processes carried out by System 1 and System 2. And then, Mode 1 or Mode 2 takes over the operation. The shared information originates from perceptual encodings of the environment.
  3. Mode 4 (System 1 and System 2 are isolated): in other cases, System 2 would initiate an independent process than System 1 is currently engaging. For example, he/she may use a mobile phone to talk with a friend while riding a bicycle, in which he/she might think deliberately to provide topics to enjoy conversation. In this case, his working memory would be used for two independent processes: talking with the friend over phone and riding bicycle safely. When encountering a dangerous situation, the system needs to take care of it primarily, which means that he/she needs to quit the phone conversation and use his/her working memory for controlling bicycle. Switching the part of memory used for the phone call to the bicycle ride would cause a certain amount of delay in action. This mode is characterized by isolation of System 1 from System 2, which means that each uses different portion of working memory for the respective processes. System 2 could be either totally detached from System 1, e.g. daydreaming, or in the deliberate thinking mode like Mode 2, in which System 2 mainly controls behavior and System 1 works under the control of System 2 by using the area of working memory for this process. Mode 3 and Mode 4 are similar because the process System 1 takes control and the Mode 3 and Mode 4 initiated by System 2 are carried out quasi-independently, but they are different in terms of the usage of working memory, i.e. Mode 3 has the area in working memory that holds information available to the two processes but Mode 4 does not.
  4. Mode 2 (System 2 controls behavior): when System 2 governs behavior, the systems try to behave according to the image System 2 created or meditate with no bodily movement. The least resources are allocated for initiating behavior according to input from the environment. This corresponds to a situation in which the amount of flow of information in System 1 is small. Working memory is occupied by activities related to System 2. However, the sensory-information filter functions so that the system can react to a sudden interruption from the environment (e.g. a phone call).

Mode 1, or System 1 control mode, would require least cognitive resources for stringing pieces of behavior in the ever-changing environment. On the other hand, Mode 2, or System 2 control mode, would consist of resource consuming activities including reasoning, recalling weak memory, etc. System 1 control may break down due to unexpected changes in the environment, which would be detected by monitoring activity of System 2, leading to System 2 control mode for searching for procedures for escaping from the undesirable situation. Note that, in daily life, human beings are normally in System 1 control mode because human beings normally prefer effortless behavior, but are occasionally forced to operate in System 2 control mode for the purpose of resuming “normal” System 1 control mode easily and as soon as possible.

Table 2.3. Four operation modes of MHP/RT and their relationships to decision-making and action selection

Synchronous Mode
Mode 1: System 1 controls behavior Skilled performance, i.e. no serious decision-making is necessary for most of behavior but still necessary to decide whether to continue, change, or terminate actions, but most of behavior can be regarded as a series of effortless action selections.
Mode 2: System 2 controls behavior Unskilled activities, e.g. learning, thinking, etc.; a series of serious decision-makings will be required.
Asynchronous Mode
Mode 3: System 1 and System 2 are weakly coupled Concentrating on skilled activities; Shared use of working memory; Easy to resume to Mode 1 or 2 when necessary
Mode 4: System 1 and System 2 are isolated Unconcentrated activities; Separate use of working memory; Time lag in resuming to Mode 1 or 2 activities when necessary Four processing modes of MHP/RT6

MHP/RT assumes that at a particular time before the event, say Tbefore, we engage in conscious processes of System 2 and unconscious processes System 1 concerning the event. At a particular time after the event, we engage in conscious processes and unconscious processes. What we can do before and after the event is strongly constrained by the Newell’s time scale of human action as shown by Figure 2.11. It indicates that System 2 carries out the processes surrounded by a round-cornered rectangle with dotted lines, whereas System 1 does those surrounded by a round-cornered rectangle with solid lines. MHP/RT works under the following four processing modes, ordered from the past to the future:

  1. System 2 before mode: conscious use of long-term memory before the event, i.e. System 2’s operation for anticipating the future event or decision-making;
  2. System 1 before mode: unconscious use of long-term memory before the event, i.e. System 1’s operation for automatic preparation for the future event or action selection;
  3. System 1 after mode: unconscious use of long-term memory after the event, i.e. System 1’s operation for automatic tuning of long-term memory related with the past event;
  4. System 2 after mode: conscious use of long-term memory after the event, i.e. System 2’s operation for reflecting on the past event.

Figure 2.11. Newell’s time scale of human action [NEW 90] and behavioral characteristics of each band

Figure 2.12 illustrates the four processing modes along the time dimension expanding before and after the event, which is shown as a boundary event. Table 2.4 shows the resulting four processing modes of in situ human behavior; at each moment, along the time dimension one behaves in one of the four processing modes and he/she switches among them depending on the internal and external states.

Figure 2.12. How the four processing modes work (adapted from our article [KIT 13]) Four processing modes and adaptation7

For MHP/RT, an event corresponds to a branch point where it can select an action from the alternatives under a specific environmental condition. The environment makes chaotic changes, and human beings, modeled by MHP/RT, are required to develop an adaptive system that is capable of dealing with a set of events that take place in such an environment. An event may result in selection of an action that is associated with one of four possible action categories, as defined by four processing modes. An event could be a future event or a past event, and it could be processed consciously or unconsciously. These are the four possibilities when we think about an event along the time dimension. An action selection will affect the future course of action selections, and the execution of the selected actions will change environmental conditions. The mode selection is carried out empirically. Under the condition of strong time constraints and the chaotic environment, the results of execution of selected actions would become unreliable, and it is required to repair the undesirable situations. In this situation, flexibility is required in selecting appropriate actions in response to the unpredictable changes in the environment. The mechanism of switching among the four processing modes relative to a series of events makes possible the high-level empirical adaptation to the ever-changing environment.

Table 2.4. Four Processing Modes [KIT 11a]

  System 2 Conscious Processes System 1 Unconscious Processes
Before After Before After
Time Constraints none or weak exist none or weak exist
Network Structure feedback feedback feed-forward + feedback feed-forward + feedback
Processing main serial conscious process + subsidiary parallel process main serial conscious process + subsidiary parallel process simple parallel process simple parallel process
Newell’s Time Scale Rational/Social Rational/Social Biological/Cognitive Biological/Cognitive

2.6. Two Minds and emotions8

As described in the previous sections, human behavior can be viewed as the integration of output of Systems 1, i.e. unconscious automatic processes, and System 2, i.e. conscious deliberate processes. System 1 activates a sequence of automatic actions in System 1 before mode. System 2 monitors the performance of System 1 according to the plan it has created, and it activates future possible courses of actions as well in System 2 before mode. At the same time, when these forward processes are working, System 1 and System 2 deal with the outcome of the forward processes by estimating the results of the performance of System 1 in System 1 after mode and the performance of System 2 in System 2 after mode. The result of estimation could be either good or bad in terms of the active goal at the moment controlled by MSA discussed in section 2.1, and it would generate emotions depending on the degree of goodness or badness of the estimation in the context of the current goal. Emotions are generated through the dynamics of the parallel processing of System 1 and System 2, which is called O-PDP discussed in section 1.2.1. This section discusses how emotion generation process is integrated with MHP/RT.

2.6.1. Dynamics of consciousness–emotion interaction: an explanation by MHP/RT Interaction between consciousness and emotion

The processes in the A2BC layer and those in the C layer are not independent. Rather, they interact with each other very intensely in some cases but very weakly in other cases. We investigate this issue in more detail below. Onset of consciousness

With the onset of arousal, the sensory organs begin to collect environmental information. This information flows into the brain, and the volume of information flow grows rapidly. As the information flow circulates in the neural networks, the center of the flow gradually emerges. It corresponds to the location where the successive firings of the neural networks concentrate. At this time, the center of information flow induces activities in the C layer via the cross-links in the neural networks. Conscious activities

Figure 2.13 depicts the state of the brain when consciousness starts working. The location of consciousness is indicated as a dot in the C layer. In many cases, the working of consciousness includes such cognitive activities as comprehension of self-orientation and the individual’s circumstances. The judgment on what decision-making is needed for the current situation is equivalent to initiating some action to move the location of consciousness to an appropriate direction. The direction of movement is determined by the information needs at that time. It could move either in the direction in which the initial information will be deepened (left in the figure) or to the direction in which the initial information will be widened (right in the figure). The density of information would change depending on how far the center of consciousness would have moved. However, the location of the consciousness would not move when carrying out a routine task.

Figure 2.13. Onset of consciousness and emergence of emotion (adapted from our article [KIT 15b]) Emergence of emotion

After the onset of consciousness, a new thread of information coming into the brain via the sensory organs triggers successive firing within the neural networks. This causes a new information flow in the brain that reflects the past experience that resonates with the input information. If there is a discrepancy between the new information flow (the dotted line in the figure) and the existing information flow (the solid line in the figure), emotion emerges. Emotion works to reduce the amount of discrepancy. Determination of next behavior

When the A2BC layer works continuously within its capacity, consciousness does not interfere with the working of the A2BC layer but monitors the individual’s behavior, prepares for the next behavior, and/or ponders issues that come to mind. However, if the A2BC layer has difficulty in determining the next behavior, the C layer takes over and determines it. However, note that decision-making deals with planning for future behavior in the “System 2 before mode”. Actions that will be taken actually in the ever-changing real world are determined by the system flexibly in an ad-lib fashion in the “System 1 before mode”.

Figure 2.14. Determination of next behavior (adapted from our article [KIT 15b]) Summary

The following points depict the flow of the processes that takes place (see Figure 2.14):

  1. 1) consciousness determines the next behavior by considering the current state of emotion and the self-recognition;
  2. 2) consciousness tunes the orientation of the sensory organs in preparation for initiating the next behavior just determined;
  3. 3) consciousness commands initiating the next behavior;
  4. 4) the behavior results in changes in the information flow;
  5. 5) the direction of emotion changes;
  6. 6) the new state of emotion affects the process of determining the next behavior. Synchronization between the C layer and the A2BC layer: MHP/RT’s perspective

We assume that the C layer and the A2BC layer operate together in order to determine the next behavior. However, as described above, the interaction between them could be weak or strong, depending on the situation. There thus needs to be a synchronization mechanism for them to work together appropriately. The most important assumption of the MHP/RT is that the human brain works under real-time constraints governed by the environment, largely uncontrollable from the brain. Therefore, detection of discrepancy is closely related with the mechanism of synchronization between consciousness and unconsciousness. The degree of discrepancy could be measured by the amount of efforts to reestablish good synchronization between the two systems.

We suggest that the visual frame reconstruction process in the C layer should be used for establishing synchronization between the C layer and the A2BC layer. In Figure 2.9, this synchronization process is indicated schematically as an arrow with the label “Two Minds”. The C layer predicts the representation of the visual frame that should appear in the future and uses it for synchronization. The information flow for this process is indicated in the dotted lines in Figure 2.9, which occurs in the characteristic times surrounded by a round-cornered rectangle with dotted lines as shown in Figure 2.11. On the other hand, when the A2BC layer mainly controls the behavior, the visual frame rate would be around 10 frames per second. The information flows as indicated by the solid lines in Figure 2.9, with the characteristic times in the a round-cornered rectangle with solid lines, as shown in Figure 2.11.

When the C layer controls the behavior as in the former case, the rate would become lower and vary depending on the interest of consciousness. In the latter case, the C layer would monitor the self-behavior by occasionally matching the expected visual frame and the real visual frame in the A2BC layer. For the former situation, the visual frame density is high but the information density is low; for the latter situation, the visual frame density is low but the information density is high. Discrepancy would be detected easier in the former case than the latter.

2.6.2. Taxonomy of emotions: behavioral perspective

Table 2.5 summarizes the relationships between Two Minds and emotions as a combination of the states of the C layer and A2BC layer. The top half of the table lists the kinds of decision-making that the C layer would do before some event takes place. Depending on the intensity of the signals emitted from the A2BC layer and the self-estimate of the state of the system, the C layer decides to do something with large effort, small effort, or just do nothing, or do nothing intentionally. Note that no emotion will take place at the time when the C layer makes decisions concerning the system’s future.

Table 2.5. Relationships between Two Minds and emotions (adapted from our article [KIT 15b])

System 2’s before-event expectation
Case C layer’s before-event decision-making Signasl emitted from A2BC layer C layer’s estimate when expectation was formed
1 do something with small effort stable (no signal) relaxed
2 do nothing intentionally bad prepared for bad
3 do something with large effort bad positive
4 do something with large effort good strongly positive
5 do nothing good calm
System 2’s after-event decision-making
Case C layer after-event decision-making Signals emitted from A2BC layer C layer’s estimate when decision-making was done Result of A2BC layer’s action Emotion after A2BC layer’s action was taken
6 do something with small effort good good + satisfaction
shock, lostness
7 do nothing intentionally bad bad + amazement, pleasure
regret, despair
8 do something with large effort bad uneasy + self-praise
9 do nothing good fearful + relief

On the other hand, as shown in the bottom half of the table, emotions will emerge when actions are carried out by the A2BC layer. A specific emotion type would emerge depending on the combinations of the possible states of the following four conditions:

  1. 1) the signal intensity of the A2BC layer;
  2. 2) C-layer’s estimate of the system state;
  3. 3) the nature of C layer’s decision-making;
  4. 4) the result of A2BC layer’s action.

For example, in Case 9, though the A2BC layer emits good signals, the C layer estimates the situation to be fearful. It decides not to do anything. However, the A2BC layer reacts to the situation autonomously and the situation eventually turns good. The C layer feels relieved. In summary, this table provides a taxonomy of emotions in terms of the activities of the A2BC layer and C layer.