8 Designing for Future Needs – Memory and Action Selection in Human-Machine Interaction

Designing for Future Needs

This chapter describes how people’s future needs are derived by applying CCE as discussed in Chapter 5. CCE assumes that people select their next behavior to maximize their satisfaction for given behavioral needs by appropriately coordinating available cognitive resources [KIT 13, KIT 12a]. CCE starts by defining critical parameters for understanding people’s behavior by considering the nature of behavior selection processes in the field in question, and then designing ethnographical field observations. The participant’s behavior is recorded, followed by a series of structured retrospective interviews. An analysis of the interview results aids in developing models of present behavior selections and their chronological changes in the past, which should trace the changes in people’s behavioral needs and the structure of satisfaction. This chapter states that these models should serve as defining future needs of persons who would follow the same developing paths with a certain amount of time delay.

8.1. Introduction

The straightforward question for identifying people’s future needs would be “what do you want in the near future?”. However, it is inherently impossible for people to provide reliable answers to this question because they will not be able to accurately observe the state of the future world where their needs will be satisfied. Therefore, the concept future needs should be regarded as the concept inaccessible to those who are asked to express it at the present time. Since it is inherently impossible for a person to express his/her future needs reliably and accurately, a methodology for defining future needs should inevitably derive the person’s future needs from outside of the person, e.g. from other individuals. This chapter illustrates how CCE has been successfully applied to derive future needs of people, drawing an example from spectators of professional baseball games. We suggest that the CCE-based methodology to derive future needs of people would be generally applicable to a variety of domains of human activities including IT devices equipped with HMI.

8.2. Making inaccessible future needs accessible: t-translation invariant principle

This section describes the idea of how to make inaccessible future needs accessible; in other words, how to define future needs of a person reliably and accurately that he/she cannot define as such by him/herself. A person has moment-by-moment desires for spending satisfactory and happy times in his/her living environment. Each person has his/her own unique living history. The personal history is stored in his/her long-term memory and significantly influences his/her way of reacting to his/her current environment with the given desire. What he/she wants to achieve at present can be considered as his/her present needs, D(t = 0), and what he/she will consider at a certain time in the future, t, he/she wants to achieve given the present needs are satisfied,

can be considered as his/her future needs, which are not accessible at present. We want to know them without directly asking him/her what he/she thinks his/her future needs are.

Suppose a person, A, whose desire state at present, t = 0, is equal to the desire state of another person, B, in the past, t = − τ < 0, i.e.

and this also applies to A’s history up to the past time,

when A began to pay attention to the object toward which his/her desire is oriented now, and if the environment can be considered stable compared with the characteristic time of the change in the state of a person who interacts with this environment, it is likely that the person A will take the same developmental path as the person B has taken.

This situation is depicted by the following equation:

In other words, A is behind B with the amount of time, τ, in terms of the change of desire state. In this scheme, defining future needs of A during the period spanning from the present, t = 0, to the future time, t = τ, reduces to revealing the history of desire state of B from the past time, t = − (σ + τ), to the present, t = 0. It is suggested that this scheme should be called the “t-translation invariant principle” for making inaccessible future needs of a person accessible to those who want to understand and use the person’s future needs at the present time. Figure 8.1 depicts it schematically.

Figure 8.1. t-Translation invariant principle

The issue of predicting people’s future needs has been dealt with in the field of ergonomics under the name of prospective ergonomics, which is about the conception and design of future systems, products or services [ROB 09]. However, as described, the problem of defining a person’s future needs should be reduced to the problem of describing the other person’s behavioral selection history, who is experiencing a certain amount of time ahead of the person in question; this is done by understanding a person’s behavioral selection process in the past, which is, in large part, governed by memory processes, and therefore, it is the issue of the field of cognitive sciences.

The following sections describe a case study that illustrates the usefulness of CCE for extracting otherwise inaccessible people’s future needs reliably and accurately.

8.3. A case study of CCE: why do fans repeat visits to the ballpark?

This section describes a case study of CCE. This study was started at the beginning of the 2008 regular season of Japan’s professional baseball pennant race for the purpose of establishing a set of hypotheses concerning the processes of developing repeaters who regularly attend games hosted by the Hokkaido Nippon-Ham Fighters1 at Sapporo Dome. Any person at the status of repeater must have reached at that status from lower statuses. The following sections describe in detail how the CCE study was conducted.

8.3.1. Steps 1 and 2 of CCE

The field of study was the ballpark of a Japanese professional baseball team, the Hokkaido Nippon-Ham Fighters. When the study was conducted, the Fighters had been there for 5 years. Those who had reached the status of repeater were expected to be able to reconstruct their histories to show how they had evolved from 2004 up until the time of the study. This study focused on the repeat visiting behavior of loyal fans of the Fighters. The specific study questions were: “Why do loyal fans repetitively visit Sapporo Dome to watch professional baseball games?” and “how have they evolved to their current status of loyal fans?”.

As to how fans are structured, the following was concluded through a brief survey (Figure 8.2). Loyal fans, or repeaters, of a professional baseball team have their own individual histories in arriving at their current fan stage. They started in the pre-fan stage, passed through the fan stage and ultimately reached their current loyal-fan stage. In the pre-fan stage, fans know little about the team, or at most they pay a certain amount of attention to the team and/or have some interest in the team. However, their attitude toward the team is passive, and they exert no aggressive action. Starting from this pre-fan stage, they advance to the fan stage, when they aggressively desire to have a relationship with the team. For example, fan stage individuals display emotion toward the results of the games, and start to become interested in watching live games at the stadium. However, they do not have much interest in information about the team. A fan stage person advances to a loyal fan by breaking through these passive characteristics. Loyal stage fans aggressively collect information about the team, go to the stadium to watch live games when time allows or even arrange their activities so as to give top priority to watching live games at the stadium.

Figure 8.2. Fan structure

A number of aspects of MHP/RT’s simulation of spectators are relevant to the CCE study at the ballpark. Referring to the Newell’s time scale of human action [NEW 90], which is included in MHP/RT (Figure 2.11), the cognitive band with the characteristic times of ~100 msec to a few seconds is relevant to understand how a person reacts to the scene he/she is observing, i.e. real-time reactions to the live event. A “cheering” person would react to the events unconsciously but the other “analytic” person would analyze scene by scene as if he were the coach of the team. The former would construct memory different from the one the latter would create. These differences might result in different courses of development toward the loyal-fan stage. In fact, Kitajima et al. [KIT 11b] found that their participants at a movie theater had created two essentially different memories of the same event on the screen depending on the degree of participation of consciousness while encoding events.

On the other hand, the rational band with the characteristic times of ~10 minutes to hours is relevant to understand how a person would expect or reflect on a specific game or specific segments within a game. Some persons would engage in these activities intentionally but the others would not. This aspect might be considered in the “cheering versus analytic” dimension as well.

Finally, the social band with the characteristic times of approximately weeks to months is relevant to reflecting on the past seasons and deciding how to behave for the coming season, i.e. pennant race, or participate in the events during the off-season. It was assumed that the number of cycles of on-season and off-season would affect the overall attitude of how to enjoy visiting ball park.

In this way, two critical parameters were derived for the study: one was the viewing attitude parameter and the other was the length of fan history parameter. In addition, the latter two bands, rational and social, are also related with the purposes of visiting and enjoying ball games at the stadium; a variety of purposes, or goals, are listed in MSA [KIT 07], a subtheory of MHP/RT. One may enjoy him/herself, one may want to have his/her family members enjoy, one may want to purchase a special premium goods, etc. These aspects would be considered in the interview step, step 5.

8.3.2. Step 3 of CCE: monitor recruiting

We conducted a Web survey for 3 consecutive days (11 June, 2008 to 13 June, 2008) and selected nine highly loyal fans, i.e. elite monitors, from the Fighters’ fan club members who had different attitudes toward professional baseball, and had visited Sapporo Dome several times since the Fighters moved to Sapporo in 2004. The number of responses was 3,687. The selection process consisted of two stages. The first stage was to select 30 respondents as candidates for the elite monitors. We conducted a series of group interview sessions with six candidates as one group for the purpose of confirming their responses to the Web survey and, most importantly, their attitudes toward the study: we wanted elite monitors who had little problem in expressing their own fan history and their behavior at the stadium, i.e. talkative and cooperative. We finally selected nine elite monitors. The nine selected fans were supposed to represent different “fan styles” and had different histories in reaching their current fan status as defined by the hypothesis as described. The number “nine” came from several constraints for conducting the study, i.e. the amount of budget, the schedule of the games at the stadium, the location and the number of seats reserved for the study, the number of equipment for recording, etc.

Figure 8.3. Illustration of the field observation: a) three elite monitors in their seats watching a game, b) ear-mounted CCD camera, c) the view of the ear-mount camera and d) an electrocardiograph and an accelerometer

8.3.3. Step 4 of CCE: field observation

We had the elite monitors visit Sapporo Dome three times to watch designated Fighters-hosted games. We recorded their viewing behavior using a DVD camera recorder located three rows in front of the monitors’ seats to capture their game-viewing behavior (Figure 8.3(a)), installing a small ear-mounted CCD camera to record the scene they were viewing (Figures 8.3(b) and (c)), recording their vocalizations with a pin microphone and using an electrocardiograph and an accelerometer to capture their physiological responses to the events of the game (Figure 8.3(d)). The designated games were a three-game series with the Softbank Hawks in July, a three-game series with the Orix Buffalos in August and a three-game series with the Rakuten Golden Eagles in September. Each elite monitor was asked to attend all three series.

Figure 8.4. A screen shot from the interview session

8.3.4. Step 5 of CCE: conduct retrospective interviews

We conducted structured interviews after each visit to Sapporo Dome, replaying the behavior recordings, the viewing scene recordings and the broadcasted TV video of the game for the characteristic events, including scoring scenes, field events between innings and events for which the participants exhibited remarkable changes in physiological data (Figure 8.4). Each participant was interviewed three times.

Each interview lasted 90 minutes, and therefore each elite monitor was interviewed for 270 minutes in total. The purpose of the first interview was to understand how the participants enjoyed the game. The purpose of the second interview was to understand how participants developed their loyalty from the pre-fan stage several years ago, to the fan stage a few years ago and then to the current repeaters stage. The purpose of the third interview was to understand what triggered the state changes and what factors helped them retain each fan stage. The interviewers required to keep the workings of MHP/RT, e.g. a large part of behavioral selections is unconscious and therefore hard to tell intellectual “reasons” for them, in mind not only while conducting the interview sessions but also when analyzing the collected data.

8.3.5. Step 6 of CCE: socioecological model construction

We compiled the results of the interviews in the form of a fan loyalty evolution diagram (FLE diagram) that represented in detail how individual participants had evolved their loyalty by specifying triggers for stage changes, circumstances that made them stay at a particular stage and activities in both the regular season and in the off-season (Figure 8.5 and Table 8.1). Nine FLE diagrams were created. We then compiled them to derive models of developmental processes of repeaters, which will be described in the following section.

Figure 8.5. A participant’s fan history during the period from 2004 to 2008 Results: developmental processes of repeaters

The following section describes results of analysis of the evidence collected during the interview sessions that focused on triggers that caused monitors to step up a stage (i.e. from pre-fan stage to fan stage, and from fan stage to loyal-fan stage), and the conditions that made or make them stay in a particular stage. These triggers and conditions define a rough qualitative model of the developmental process of fan loyalty along the time dimension.

Table 8.1. The events that happened in respective years that characterize the fan level of events

Number Year On/Off Event
0 2004 On Not interested in baseball. Not interested in the Fighters coming to Sapporo.
1 2004 On Obtained information about the team and the players in relation to his job as journalist.
2 2004 On Had come to pay attention to some players.
3 2004 Off Knew about the players’ trade, salary, etc.
4 2005 On Joined the official fan club and net communities. Watched TV.
5 2005 Off Got access to the information during season off, but not actively.
6 2006 On Cheered with a player’s T-shirt. Began to repeat at the end of the season.
7 2006 Off Participated in the parade for championship, events and talk shows.
8 2007 On Went to the Sapporo Dome stadium after work when possible. Cheered with replica uniform.
9 2007 Off Participated in the events. Purchased the passport ticket.
10 2008 On Visited the farm and visitor games. Began to purchase goods. Started to limit the amount of purchases.
11 2008 Off Got access to the information during the season off, but not actively. Progressing from the pre-fan stage to the fan stage

Three common triggers were found in the study for advancing the elite monitors from the pre-fan stage to the fan stage.

  1. 1) “Retirement of a star player” and “expectation of league championship”: In the 2006 regular season, two events triggered three participants who had little knowledge about professional baseball, and another three participants who had knowledge about professional baseball but did not have enough interest in it, to progress to the fan stage. One event was an announcement by the then star player, outfielder Tsuyoshi Shinjo, that he was retiring, relatively early in the regular season. This news was reported frequently in various media. The other event was that the Fighters were in the first championship race of the league and Japan’s professional baseball leagues.
  2. 2) “Watch the fans cheering”: Two participants who had little knowledge about professional baseball and one participant who had little interest in professional baseball advanced to the fan stage after watching live cheering in the stadium.
  3. 3) “Know the players and the team” and “unexpected talent of players outside baseball”: Regardless of their knowledge level of professional baseball, knowing players and the team triggered participants to progress to the fan stage. Three participants who knew professional baseball reacted to the players’ behavior outside baseball, causing them to advance to the fan stage. Advancing from the fan stage to the loyal-fan stage

Ten common triggers were found in the study for advancing the elite monitors from the fan stage to the loyal-fan stage:

  1. 1) “Watching live games at the stadium”.
  2. 2) “Knowing the rules of baseball and the team”.
  3. 3) “Watching games by oneself”, “one’s wife became a fan by following his lead”, “communication with his/her friends at the stadium” or “meeting persons who visited the stadium”: The common feature of these triggers is the establishment of an environment where fans could comfortably watch the games at the stadium with someone who contributed to building a relationship with them (e.g. spouse or friends).
  4. 4) “Presence of players who always come to mind”: Participants who had little knowledge about baseball or professional baseball, those who were fans of other professional baseball teams, and those who became fans at the end of the regular seasons tended to find opportunities that should provide information about players, teams and the Fighters in particular. These participants were eager to attend off-season events such as talk shows and advanced to loyal fans in the next regular season.
  5. 5) “Collecting the Fighters’ merchandise”.
  6. 6) “Recording events of live games” and/or “collecting the recordings as proof of watching the games”.
  7. 7) “Expectation of the climax series and the Nippon series”, and “eagerness to watch those series”.
  8. 8) “Communication with the other fans when watching live games”.
  9. 9) “Network community” that they accessed during live games to exchange information and post opinions.
  10. 10) “Seeing the players closely, e.g. visiting camp in Okinawa”, and those who had special interest (or who followed pro-baseball) said that their greatest interest was in seeing live action on a professional field.

8.3.6. Results: developing from a pre-fan to a repeater

While they were in the pre-fan stage, the nine elite monitors were classified into three categories in terms of their interest in baseball or professional baseball:

  1. 1) three elite monitors did not have any interest in baseball;
  2. 2) another three were interested in baseball in general but did not have interest in professional baseball;
  3. 3) the rest had interest in professional baseball but were not interested in purchasing tickets to visit Sapporo Dome for watching Fighters’ games.

Figure 8.6 illustrates the cases of the Group A and Group C. The pre-fans who did not know baseball well, Group A, have developed into either repeaters who enjoy cheering or those who enjoy watching games (Figure 8.6, top). The pre-fans in the Group C category have developed into repeaters who enjoy watching games (Figure 8.6, bottom). Defining future needs by CCE

This section starts by discussing how the t-translation invariant principle could be applied for the fans of the professional baseball team. In this study, the trajectories of the elite monitors from 2004 to 2008 could be considered as typical developmental paths from their pre-fan stage, to their fan stage and ultimately to their loyal-fan stage. The common triggers were identified for developing from the pre-fan stage to the fan stage, and from the fan stage to the loyal-fan stage (see [KIT10] for more details). The principle claims for a fan’s future needs as follows, if we go back to the year of 2008, for example:

A fan at his/her pre-fan stage now, i.e. the year of 2008, whose attitude toward the professional baseball team is the same as one of the elite monitor’s, say E, will take the same developmental path from 2008 to 2012 as E has taken from 2004 to 2008 given the appropriate triggers are to be provided to him/her timely.

As shown in Figure 8.6, the developmental paths of the nine elite monitors depended on their attitude toward the professional baseball team when they had been in the pre-fan stage. This suggests that questionnaires concerning their attitudes toward professional baseball, e.g. “do you have knowledge about baseball?”, “do you know about professional baseball?”, “do you purchase tickets for watching professional baseball game at the stadium”, would be useful to classify a fan into the fan similar to those shown in Figure 8.6. Effective triggers may be different, as discussed in section

Figure 8.6. The evolution trajectory of each elite monitor. Left: the paths for those who had not known baseball when they were at the pre-fan stage. Right: the paths for those who had not been fan of this specific team when they were at the pre-fan stage

8.4. Discussion

8.4.1. Selection of elite monitors

In this case study, nine elite monitors were selected from 3,687 respondents of the Web survey. The selection process of elite monitors is similar to purposeful sampling that is normally used in qualitative studies. Purposeful sampling is significantly different from random sampling in which the purpose of the study is to understand average characteristics of the population from which the samples are selected randomly.

CCE is interested in the study question “what would such and such people do in such and such a way in such and such a circumstance?”. It is not “what would average people do in such and such a circumstance?”. CCE is also interested in understanding people’s behavior along the time dimension. Therefore, CCE naturally involves the processes of recording monitors’ behavior in the real field and retrospective interview sessions for identifying monitors’ memory that should have been active when the behavior was recorded. From these data, the investigators try to reconstruct the elite monitors’ individual histories with a focus on the critical parameters. The histories thus created could be compared within the parameter space defined by the critical parameters to derive useful insights that should provide deeper understanding of the people, among whom the elite monitors should be appropriately located. In this way, elite monitors are the precious resource for the study and needed to be selected carefully.

8.4.2. CCE interview

The main purpose of a CCE study is to reconstruct the participant’s behavioral trajectory in order to answer the question “what would such and such people do in such and such a way in such and such a circumstance?”. What we observed in the study field is used in the interview sessions in which the investigators and the participant would engage in the project of reconstructing the participant’s history relevant to the purpose of the study. Figure 8.5 and Table 8.1 illustrate the typical results of the interview sessions.

What the investigators should extract in the interview sessions are the descriptions of the contents of long-term memory that are related with the contents that are active in the participant’s working memory, which originally come from the recorded data shown to the participant serving as the cues to make available the contents in long-term memory. Figure 8.4 illustrates such an interview session. The image displayed on the screen would be used to create some representations in working memory which would then serve as the initial cues for activating relevant portions of long-term memory.

In general, investigators tend to search for causal relations among events; when an event comes after another one, they are inclined to understand the event sequence in such a way that “the latter should have caused the former, and therefore there should be a reason why the participant did that way as evidenced by the event, observed or described”. However, as Two Minds processing strongly suggests [KAH 03], people often make intuitive decisions that inevitably would lead to unconscious behavior, which should not be associated with any rational reasons. On the other hand, the participants tend to provide “intellectual” answers when they are asked “why did you do that way?” because they want to be evaluated as a rational being. What the investigators would obtain are spurious reasons that may lead to the wrong understandings of the participants behavioral trajectories. We suggest that reconstruction of the participants’ behavioral trajectories should be carried out with the focus of how the contents of long-term memory should have evolved considering the memory processes of MHP/RT [KIT 13].

8.4.3. Applicability of CCE

The interview step is key for the reconstruction process. However, the format of interview, namely, the investigators typically ask question verbally and the participants give answers verbally, or via symbols with the shared meaning, should place restrictions on the range of the reconstruction. This is again related with the memory process. As shown in Chapter 3, each system has its own memory, i.e. RMD frame for the conscious process and BMD frame for the unconscious process, and both may be integrated to construct a memory that is accessible from the conscious process and/or the unconscious process.

However, there are cases where a BMD frame is not integrated with a RMD frame, which was actually evidenced in the study concerning how people enjoy watching short films [KIT 11b]. Since the interview uses symbols as the representations for getting access to long-term memory of the participants and for having the participants describe the contents of long-term memory, the pieces of memory that are not integrated with the RMD frame are not accessible via symbolic representations. For example, when a participant is shown a scene that he/she has actually watched and to which he/she showed strong physiological reactions, and he/she is asked to report the consequence of a scene; if he/she has only the memory of the scene in the BMD frame, he/she will not be able to associate the memory with any reportable symbols. He/she will be able to activate certain memory traces in the BMD frame that is associated with the scene, but they are not labeled as “the consequence of the scene” because this requires integration with the RMD frame. Therefore, he/she will not be able to provide any useful answers to construct his behavioral trajectories. As mentioned in section, he/she is working in MHP/RT Mode 3 or MHP/RT Mode 4 in which the conscious process of System 2 and the unconscious process of System 1 work asynchronously, and therefore integration of the RMD frame and the BMD frame has not happened. In other words, CCE is applicable for those who work in MHP/RT Mode 1 or MHP/RT Mode 2 in which the conscious process and the unconscious process work in parallel and memory will be constructed by integrating RMD frame and BMD frame.

8.5. Conclusions

This chapter began with the claim that people would never be able to imagine their future accurately and reliably, nor what they would need in an unknown future. However, there are needs in a variety of fields for knowing people’s future needs. This chapter proposed a solution by introducing the idea of people’s behavioral trajectories – some of which may advance a certain amount of time from the others – and the principled way of reconstructing people’s behavioral trajectories, CCE, whose basis is provided by the cognitive architecture, MHP/RT, that is capable of simulating people’s decision-making or action selection processes in daily life.

Understanding people’s decision-making in daily activities is crucial for CCE studies. There are two forms of understanding: quantitative understanding and qualitative understanding. Quantitative understanding is derived through statistical data analyses applied to a large collection of data, implicitly assuming that noise in the collected data should obey some form of probabilistic distribution. This form of understanding involves identifying causal relationships among entities in the data. Rational interpretations of the data are expected. This might be possible if the data were generated solely by the rational processing system. In contrast, qualitative understanding is derived through studies in natural settings that try to uncover the regularities in the behavioral trajectories, such as common triggers for promoting them to higher fan stages.

Considering the nature of decision-making and action selection at the site in question, the observed behavior should in a large part be the result of immediate actions controlled by the experiential processing system. Such decision-making and action selection may not be rational, but is controlled by the bounded rationality principle and the satisficing principle uncovered by Simon [SIM 56] and further studied by Kahneman [KAH 03] in generating situated decision-making and action selection. As Newell [NEW 90] suggested, the conscious processes and the unconscious processes occur in the different bands with different characteristic times. It is not wise to try to understand the phenomena in the higher bands by extrapolating the findings in the lower bands, i.e. to try to understand rational behaviors by treating unconscious behaviors as subsidiaries as ACT-R [AND 98, AND 07]. This book has advanced the line established by Simon [SIM 56] and Kahneman [KAH 03] even further by introducing the cognitive architecture, MHP/RT, for simulating people’s daily decision-making and action selection, whose differences are discussed in Kitajima and Toyota [KIT 13], and the methodology, CCE, for conducting field study to understand people’s decision-making and action selection process, of which prescription is given by MHP/RT.

This chapter illustrated a case study that applied the CCE methodology for reconstructing history of fans of a professional baseball team. The unique feature is the selection of monitors, called elite monitors, who should represent respective segments defined by combinations of values of critical parameters that should be effective to distinguish among the behaviors of fans. There are those fans who are a certain amount of time, say 2 years, behind an elite monitor and would follow the same behavioral trajectory as the elite monitor. However, we need a method to find those fans, which should be challenged in the future.

The case study revealed histories of nine elite monitors, which demonstrated how they moved through the fan stages, from the pre-fan stage to the fan stage and ultimately to the loyal-fan stage. We identified features that motivated participants to advance from the fan stage, and those that motivated them to advance from the fan stage to the loyal-fan stage. These features should suggest possible paths that potential loyal fans follow and define possible their needs when they are at the pre-fan stage and those when they are at the fan stage in the future.