5 Cognitive Chrono-ethnography – Memory and Action Selection in Human-Machine Interaction

Cognitive Chrono-ethnography

In the previous chapters, a cognitive architecture for action selection and memorization has been discussed. It provides the mechanism for analyzing how human beings would select their next actions in an ever-changing environment by synchronizing unconscious fast processes and conscious slow processes, and what would do by using MD memory frames, which themselves evolve over time as human beings interact with the environment by performing actions and reflecting on the results of performance. On the basis of this understanding of human beings’ action selection and memorization, this chapter introduces a theoretically motivated study methodology for understanding human beings’ activities in real life situations.

5.1. Understanding people’s behavior in real life

A series of a person’s action selections can be regarded as a result of moment-by-moment problem-solving activities. The problem in this situation is in realizing a goal state by successively applying moves in the problem space. Each move causes a transition from one state to another and the move finally chosen at a specific state depends on the available resources that he or she can manipulate during the allowed time. Remember that moves can happen in a varied timescale as indicated by Newell’s time scale of human action (see Figure 2.11). If enough time is provided, it is possible for him or her to select a move that has a greater expected benefit. In this situation, he or she can utilize as much knowledge as possible to achieve better solutions. However, if not enough time is allowed, there is nothing for him or her but to choose an ordinary solution. In most situations, this is the one he or she has adopted most frequently in the similar situations to the current one in the past. This decision is done without deliberate consideration concerning possible future development of the course of actions. Using the terms introduced in section, the situations can be rephrased as follows: the former case is dominated by System 2 before mode, whereas the latter by System 1 before mode, respectively.

Understanding people’s behavior in real life can be achieved by two-tiered research approach, that is domain-independent theorization and domain-dependent instantiation, or, theoretically motivated case studies. Domain-independent theorization deals with what people would do at abstract levels. This book suggests that NDHB-model/RT is the theory and MHP/RT provides a way for simulating human action selections on the basis of NDHB-model/RT.

This chapter and the following two chapters describe domain-dependent instantiation for understanding people’s behavior in real life. This chapter explains a methodology cognitive chrono-ethnography (CCE) to be used in this research activity. CCE is derived by considering requirements that are imposed on a study methodology for dealing with people’s action selections in time-critical real-world settings. It starts by describing the requirements for the methodology and providing the top level description of the CCE methodology.

The chapters to follow illustrate case studies to demonstrate what people actually do in real-world navigation. Navigation is an appropriate behavioral category to study with MHP/RT because: (1) both external and internal situations change as far as navigation continues, and synchronization of conscious processes with unconscious processes is definitely needed to produce coherent behavior in the ever-changing environment; (2) navigation can be either self-paced (internal control) or environment-paced (external control); and (3) navigation can be either fast or slow. Navigation can happen either in a real physical environment or in a virtual informational environment. No matter in which environment navigation takes place, action selections and memorization can be considered in the framework of NDHB model/RT and simulated by MHP/RT. This book takes two case studies concerning navigations with different characteristics. Chapter 6 concerns a relatively slow navigation, i.e. investigation of how elderly people use guide signs at train stations when they have to transfer lines, in addition to using facilities such as restrooms, lockers, elevators, telephones and so on. This situation is characterized by self-paced active information gathering to select next actions for accomplishing consciously managed explicit goals. Chapter 6 concerns a relatively fast navigation, i.e. investigation of valuable and useful information for drivers when driving unfamiliar roads. This situation is characterized by environment-paced passive information gathering to select next actions for accomplishing unconsciously managed implicit goals.

5.2. Cognitive chrono-ethnography

Our 24-hour day is roughly divided into three categories. The first is the hours for work to earn money, the second is the hours for biological activities to live and the third is the hours for the rest to spend time for feeling satisfaction or happiness, otherwise called leisure activities, such as playing sports, watching TV, driving a car, playing PC games, traveling, going to the movies, surfing the Web and so on. Not all but most activities categorized in the third category involve HMI and therefore its design should affect the performance of such activities. When it is well designed, the users have satisfactory experiences. Traditionally, the first two categories have been dealt with in such study fields as human factors and ergonomics. However, the third category has been little studied because of the diversity involved in such activities. This section starts by describing requirements for the methodology (CCE) to study people’s behavior in real life that belongs to the third category and suggests four critical factors to be considered as the primary causes of the diversity, and must be disentangled by CCE to obtain coherent understanding of how daily behavior is organized.

5.2.1. Requirements for the methodology (CCE) to study human beings’ behavior in real life What to understand

Understanding human beings’ daily action selections should come down to understanding relationships between memes that are active at the time the behavior was undertaken and the overtly observed behavior by taking into account those factors as Two Minds, the multiplicity of goals and the nature of memory processes (see section 5.2.3 for the detailed explanation concerning meme and these factors, which are called behavior shaping factors). The following are specific questions that a CCE study has to deal with for respective behavioral events:

  1. 1) Which memes were activated?
  2. 2) Under which conditions were the memes activated?
  3. 3) How had the meme been formed?

The answers will be analyzed to construct models that explain and predict human beings’ behavior in the study field. How to understand

This section discusses what kinds of data are available for deriving answers to the above questions. The origins of the data will be the results of observation of human beings’ daily action selection processes in real-world settings instead of in simulated laboratory experimental settings. A list of data that are obtainable with little interference with the participants’ activities are as follows:

  1. 1) Behavior observation records: Investigators will record the participants’ behavior without any intervention with their activities.
  2. 2) Behavior measurement records: Sensors will be attached to the participants to record their physiological activities, e.g. a pin microphone to record their vocalizations, a small ear-mounted camera to record the scene they are viewing, an electrocardiograph to record their physiological responses to the events and so on.
  3. 3) On site self-reports: Study participants themselves will take photos, brief notes, voice recording, etc., concerning their activities while their memories of the events remain fresh.
  4. 4) Utterances in retrospective interviews: Behavioral observation records, behavioral measurement records, and on site self-reports will be used to reconstruct their active memes at the time of events by conducting a series of retrospective interviews. Their utterances in the interview sessions will be recorded.

5.2.2. CCE procedure

CCE is carried out in the following six steps as shown in Figure 5.1:

  1. 1) Phase 1 of “preliminary socio-ecological modeling”: It defines the study field. It is important to specify the study field sufficiently to undertake successful CCE studies. Manifestations of behavior selection shaping factors under the characteristic atmosphere of the study field, which must be understood in terms of the effects of behavior selection shaping factors, will be observed in the study field.

Figure 5.1. The CCE procedure

  1. 2) Phase 2 of “preliminary socioecological modeling”: It defines critical parameters. Critical parameters are initial hypotheses about the behavior selection shaping factors that should work when human beings’ activities are organized in the study field. To do this, it is necessary to examine the structure and dynamics of the study field in order to ensure the existence of chronological changes in the people in question, to construct hypotheses about the critical parameters and to carry out a preliminary test. Steps 1 and 2 are conducted interchangeably to define the parameter space to be explored.
  2. 3) Monitor recruiting: It selects elite monitors. In order to conduct CCE, study participants (elite monitors) are selected. Each point in the parameter space has values, continuous or discrete. The study question is “what such and such people would do in such and such way in such and such circumstance (not an average behavior)”. Therefore, elite monitors, such and such persons, are selected by consulting the parameter space. In this process, it is necessary that the points in the parameter space, which correspond to the elite monitors, are appropriate for analyzing the structure and dynamics of the study field. Monitor selection is conducted by purposive sampling rather than by random sampling.
  3. 4) Monitor behavior observation: It records the monitors’ behavior. The elite monitors are expected to behave as they normally do at the study field. Their behavior is recorded in such a way that the collected data are rich enough to consider the results in terms of the parameter space and as unintrusively as circumstances allow.
  4. 5) Individual model construction: It conducts retrospective interviews. The collected data are used to clarify the structure of the meme of the elite monitors by conducting a series of structured interviews. The results of the interviews are analyzed for the purpose of defining the basis of the representations of the collected data. The analysis involves finding common terms used in the interviews and common activities that are defined by combinations of the common terms, as well as statistical analysis of the activities, e.g. factor analysis and cluster analysis.
  5. 6) Socioecological model construction: It construct models. The last step of CCE is to construct models that address “such and such people would do what in such and such way in such and such circumstances”.

5.2.3. Behavior selection shaping factors

CCE studies are carried out by keeping the following critical factors for understanding human beings’ daily behavior in mind. Two Minds

As described in the introduction, Two Minds is one of the crucial factors for understanding human beings’ action selection processes. System 1 of Two Minds corresponds to the A2BC layer of NDHB-model/RT and System 2 corresponds to the C layer, respectively, as described in section 2.4, and its operation is simulated by MHP/RT, as described in section 2.5. System 1 is a fast feed-forward control process oriented toward immediate action (System 1 before mode), and is experienced passively, outside of conscious awareness (System 1 after mode). In contrast, System 2 is a slow feedback control process oriented toward future action (System 2 before mode), and is experienced actively and consciously (System 2 after mode). There is a huge difference in processing speed between the two systems; rational processing typically takes minutes to hours, whereas experiential processing typically extends from hundreds of milliseconds to tens of seconds (see Figure 2.11 ). A large part of human beings’ daily activities are immediate actions and are therefore under control of System 1. System 2 intervenes with System 1 to better organize the overall outcome of the processing through consciously envisioning possible futures. Human beings’ behavior is the outcome of the operation of System 1 and System 2 along the time dimension, which work in parallel and at the same time work in synchronous with the development of the states of the environment. The observed data concerning the study participants’ behavior at a study field must be understood from the viewpoint of the operation of Two Minds along the time dimension. Understanding “how” human beings carry out action selection processes in the ever-changing environment should reduce to understanding how the four processing modes of MHP/RT are coordinated in real life situations. Meme that mediates individuals and society

Moment-by-moment decision-making and action selection is carried out by utilizing knowledge that is activated from long-term memory in response to the recognized objects that exist in the external world. A meme is an entity that represents the information associated with the object that a person can recognize. Therefore, active memes play a critical role in decision-making and action selection processes. The term “meme”, originally coined by Dawkins [DAW 76], was conceptual and not clearly defined. However, meme can be defined more clearly by assuming that meme itself is structured in accord with the structure of living organisms, which is characterized by nonlinear, multilayered information structure.

As described in section 2.3.1, the structured meme consists of the following three nonlinear layers, each of which is associated with multidimensional memory frames described in section 3.1.2:

  1. – action-level memes represent bodily actions, corresponding to PMD, BMD, and MMD;
  2. – behavior-level memes represent behaviors in the environment, corresponding to PMD and RM;
  3. – culture-level memes represent culture, corresponding to PMD and WMD.

Understanding “what” human beings carry out decision-making and action selection processes in the ever-changing environment should reduce to identifying active memes considering functional flow structure and evolving cyclic network structure in the layered structure of multidimensional memory frames as shown in Figure 3.5, while performing actions with MHP/RT, in conjunction with the memorization process as shown in Figure 3.3. Multiplicity of behavioral goals

Morris [MOR 06] defined 17 happiness goals. It is assumed that a person pursues one of the 17 goals at every moment, and switches to another when appropriate by evaluating the current circumstances. CCE needs to identify the current goal and elucidate the goal enabling conditions by consulting maximum satisfaction architecture described in section 2.1 working at the B layer of NDHB-model/RT (section 2.4). A person feels satisfaction when a goal is accomplished. The amount of satisfaction felt is influenced by the factors that characterize the shape of trajectory of behavioral outcome. There are six critical factors to make people feel satisfaction (see section 2.1.4 and Figure 2.2). Usage of memory under strong interaction between environment and behavior

Memory processes play a crucial role on the course of decision-making and action selection processes and their results, i.e. the behavior he or she ultimately selects and carries out. As Newell [NEW 90] described in his seminal book, Unified Theories of Cognition, human beings operate differently according to the characteristic times of the environment they are in (see Figure 2.11). When real-time constraints are strong, slow memory processes, which use long-term memory, should not participate in the whole processing. In other words, only the unconscious side of the Two Minds (System 1) should work. On the contrary, when there are little real-time constraints, both System 1 and System 2, consciousness and unconsciousness systems work collaboratively in some cases and independently in other cases. Memory processes stay in the back of meme activation, and therefore when we want to envision active memes at the specific time when the event occurred, he or she has to take into account the plausible memory processes that should have operated given the time constraint imposed on the participant at that time.