By Peter Cooper,2014-05-16 01:34
6 views 0

    Pervasive Cognitive Assistance in Smart Homes

    aaaJérémy Bauchet, Denis Vergnes, Sylvain Giroux, abHélène Pigot and Jean-Pierre Savary

    a DOMUS Laboratory

    Department of Computer Science,

    Université de Sherbrooke

    2500 boul. Université,

    Sherbrooke, Canada J1K 2R1

    {Sylvain.Giroux, Helene.Pigot, Jeremy.Bauchet}

    b France Telecom Research and Development division, France


    Cognitively impaired people may suffer from deficits of attention, initiation, memory, and planning. A pervasive cognitive assistant (PCA) able to address them is deployed in an experimental smart apartment. The illustration of the prototype focuses on the morning routine of a patient. Assistance is either provided on the assistant initiative or on the patient request. Assistance may consist in cues and interactions with the assistance system or may involve caregivers synchronously or asynchronously.

    1 Introduction

    Networks, microprocessors, memory chips, smart sensors and actuators are faster, more powerful, cheaper and smaller than ever. They enable to build brand-new communicating objects and pervasive information systems capable to improve the life of elderly and disabled people in their home and outside. Accordingly technology can transform conventional homes into smart homes where distributed information systems assist people and foster their autonomy.

    Many research works led aim at transforming cognitive prosthetics into a reality in a near future. At the DOMUS lab, our target is pervasive assistance to cognitively impaired people [1]. Our work focuses on adults suffering from head injury or schizophrenia, and on a lesser extent on elders suffering from dementia. In this paper, we show how pervasive computing may give a hand in their activities of daily living (ADLs). At present, cognitively impaired people are often very dependent on caregivers. If their relatives cannot take care of them, they need to live in a supervised home. The deployment of distributed information systems in their homes can help them to recover the necessary competence needed to complete their ADLs. Their environment will become able to support them by giving pieces of advice and by taking appropriate actions when hazardous behaviours are detected. As competence rhymes with autonomy, they will be able to stay at home longer in a safer environment for themselves and, consequently, for their relatives.

    In this paper, a pervasive cognitive assistant (PCA) is presented. We show how this assistant observe the concrete actions of the assisted person, and help him to perform his ADLs in a domestic environment by providing cues to guide the person to reach his goal,

    or by communicating with caregivers. First section describes existing research works on cognitive prosthetics and on smart homes. Then section 3 shows how an assistant can

    cope with cognitive deficits of initiation, planning, memory, and attention. Section 4 gives a glimpse at the PCA and the current prototype. Section 5 addresses control of the PCA and communication with caregivers.

    2 Related works on cognitive prosthetics and smart homes

    Yet some research works on cognitive prosthetics have already given concrete results. However, their scope for ADL assistance is limited as they address specific problems, or as their implementation does not allow providing a more global assistance. Section 2.1 presents three of them related to cognitive assistance on mobile devices: Autominder, ISAAC, and opportunity knocks. Section 2.2 then shows that pervasive computing is emerging as an alternative or a complement to mobile devices. Section 2.3 discusses pervasive assistance in a smart home.

    2.1 Cognitive prosthetics

    Autominder, a collaborative project between University of Michigan and Carnegie Mellon University, reminds intelligently the completion of specific activities during the day [2]. Implemented on a robot, the system collects data by the means of sensors. Data are employed to infer what the person is doing. When necessary, the system gives reminder through the robot screen and speakers. Autominder provides limited assistance. First, the data collected are not sufficient to infer the completion of complex activities. Indeed, the robot is equipped with only few sensors, and the reach of the sensors is conditioned by the robot position. Secondly, the robot screen and speakers do not allow to give the person relevant cues directly in his/her environment. The person has to focus his/her attention specifically on the robot devices.

    Various cognitive prosthetics are implemented on mobile devices, in particular personal digital assistant and mobile phones. The ISAAC System provides procedural information about activities completion (“how to” guidance line) [3]. This mobile tool is an efficient support for a step by step activity completion. It remedies mnesic deficits which prevent the person to perform even easy activities in a correct way. Opportunity knocks, from the Washington University, assists the person in his/her routine trips [4]. Implemented on a mobile phone, it manifests itself when the path followed by the person in not correct according to his/her usual. It gives indications about the correct path, notably in regard to the means of conveyance.

    2.2 Smart homes

    In the field of smart homes, several projects are engaged for the addition of technological devices and automation systems in domestic environment. Such devices are traditionally designed to assure security for physically impaired people or elderly people [5]. Nevertheless, since few years, a new axis of research concerning smart homes has emerged. It focuses on the assistance for cognitively impaired people [6]. Cognitive assistance in smart homes takes all it sense with the emergence of a new paradigm, which is now taking place in domestic environments: pervasive computing. In this approach, the conventional centralised computer vision is left for a pervasive one: computer is everywhere in the environment, while being transparent for the user [7].

    Activity recognition, the first step of cognitive assistance, is already based on the treatment of information collected through sensors distributed in the home environment [8]. With the development of new networks technologies, sensors can be distributed and communicate together in order to diagnose the status of the occupant. The next step is then to display an intervention in the environment, which will interact with the occupant to assist him/her in case of difficulty.

    2.3 Pervasive cognitive assistant

    According to the pervasive computing paradigm, we propose a cognitive prosthetic which interacts with the person and assist him through the environment. The cognitive assistant supports the person during the completion of ADLs in a non-intrusive way. Assistance services are distributed in the home avoiding the need for the person to carry a device or to focus on a specific place to receive assistance. Instead, the whole environment is engaged in a cognitive assistance process by gathering information about the actions and, if necessary, providing cognitive aids to assist or prevent them.

    Pervasive cognitive assistant offers a relevant way to interact with the person and increase the effectiveness of the assistance. The aim is to enhance the occupant’s autonomy and enable him to stay in a safe and powerful environment. The objectives are twofold:

    ; to elicit possible cognitive deficits at the origin of a ill-performed ADL by means

    of information collected by a sensor network,

    ; to assist the occupant by the means of effectors taking into account the diagnosed


    Therefore pervasive computing is at the root of the solutions we propose by involving the entire environment in the detection phase as well as in the assistance phase. 3 Addressed cognitive deficits

    The cognitively impaired population considered in this research is composed of persons suffering from Alzheimer disease, head trauma, and also schizophrenia. In a study led both in Quebec and France, several cognitive deficits were pointed out as being possibly addressed with advanced technologies [9]. According to the caregivers the four cognitive deficits elicited in that study are mainly responsible of the autonomy disruption in the daily life. There are initiation, planning, attention and memory deficits. The autonomy disruption could be due to difficulties in remembering which activity to perform and how to do it. It is also due to the difficulty in focusing on the activity in process or even to begin it. Such behaviours may impair the health and well being of the person, as the essential activities, like eating or taking medicine, could not be completed in time. Then the person needs continuous prompts from her caregiver, which could affect the relationship [9].

    For each deficit, impacts in the daily life are presented and how they are transcribed in the context of smart home. The PCA role is decomposed in two steps, first the diagnosis of the cognitive deficit, then the aim of the assistance provided to the occupant.

3.1 Initiation deficit

    The initiation deficit leads to inactive periods whereas the person is supposed to perform actions [10]. For example, during breakfast time, standing in the kitchen for a long time could be attributed to an initiation deficit.

    The PCA detects an initiation deficit when no action is observed during a period where the occupant is supposed to be active. To diagnose it, the PCA must combine the information from three features: first the lack of actions detected by the sensors, second the period of inaction and third the occupant habits. The third feature is used to compare the actual activity with the one presupposed. It is essential to avoid detecting initiation deficit in a period where the occupant is usually inactive, as for example during a nap. The PCA infers a deficit of initiation when it notices that, contrary to the information stored about the occupant habits, the sensors indicate no action during a period. The aim of the assistance is to urge the person to begin the activity.

    3.2 Planning deficit

    The planning is the identification and organization of the steps and elements needed to achieve a goal [10]. The planning deficit leads to the difficulty to perform an appropriate sequence of actions in order to achieve a goal. To prepare tea, it is at least necessary to put a tea bag in a cup, to boil the water, and to pour it in the cup. The appropriate sequence requires that the third task must be performed after the two others. Performing actions in an inappropriate sequence indicates a planning deficit.

    Given an activity in progress, three cases are distinguished according to the current action performed:

    (1) The current action is related to the activity under progress, but should not occur at

    that time. Previous actions have to be performed before this one. For example,

    pouring the coffee before the water is hot.

    (2) The action under progress is not related to the activity. But this action takes place in

    the same location as the activity one. It could be inferred that the person encounters

    difficulties to formalize the next step that has to be completed. She then tries an

    action without following the goal. For example, she opens the cabin doors instead of

    the drawer to pick up a spoon.

    (3) The activity is engaged, but the current step average duration has run out. The

    occupant seems to be unable to perform the next step.

    Time duration is involved in the third case as shown previously for the initiation deficit. But, here the activity has yet begun and the person is lost, exhibiting difficulty to find the next step, instead of waiting without beginning the activity. In the second case, the location where the non-relevant action takes place allows identifying the nature of the deficit. Outside the activity area it could be due to a stimulus leading to an attention deficit, inside the activity area it is attributed to the difficulty to find the next step. The PCA detects a planning deficit according to the three types of unfair actions presented below. To diagnose one of these cases, the PCA must combine the information from the following four features:

    ; The location of ADL under progress,

; The location of the current action,

    ; The sequence of actions in an activity,

    ; The average duration of a step completion.

    Given the activity under progress, given the current action, the PCA diagnoses a planning deficit if the current action is not the one to be performed at that time even if performed in the same location, or if it takes too much time to perform the next action. The aim of the assistance is to recall the next step in the sequence of actions for the current activity.

    3.3 Attention deficit

    The attention concept is linked to the processing of external stimuli [10]. During task completion, the person shifts her attention from the activity under progress to a stimulus causing interference. The person demonstrates difficulty to focus on the activity to be performed and as a consequence, the current activity should be forgotten and never completed.

    The PCA detects attention deficit when the actions performed are not related to the current activity. To distinguish it from a planning deficit, the location of the new location is crucial. As explained before, if the current action location is different from the activity one then the PCA diagnoses an attention deficit, otherwise a planning deficit. Given the activity under progress, given the current action, the PCA diagnoses an attention deficit if the current action is not the one to be performed at that time and if it is performed in another location than the activity under progress.

    The aim of the assistance is to help the person continuing the current activity. It will then recall the goal of the activity to the activity to help her keeping the focus on it. 3.4 Memory deficit

    The memory processes refer to information storage and retrieval [10]. Suffering from memory deficits could lead to difficulties to remember the activity to perform, the steps of the activity or the locations of the tools and materials involved in that activity. It is inferred in this study that the person is aware of her memory deficits. She will then take the initiative to ask for assistance.

    The PCA diagnoses the memory deficit by the kind of demand asked by the occupant. The aim of the assistant is to provide the forgotten information.

    4 The PCA system

    In the previous section, we presented the cognitive deficits addressed by the PCA, how it detects them and how the aim of the assistance is adjusted to each deficit. Information on the activities performed in the smart home is gathered by the means of sensors, then it is analysed to diagnose the appropriate deficit and to propose the adequate assistance. The representation of the activity used in the PCA is presented in the next section. The smart home is then described, first the sensors used to detect the actions, second the effectors to provide the assistance.

4.1 Activity representation

    According to numerous cognitive and computer theories, an activity is performed in a hierarchical mode. In order to achieve an activity, one divides it in sub goals, which in turn could be divided in other sub goals until an atomic action is reached. For example, in order to prepare tea, one elaborates three sub goals, the one to boil water, the other one to put a tea bag in a cup and the third one to pour boiling water in the cup. The first one is decomposed into several sub goals such as filling up the kettle, turn on the stove and so on. The atomic actions are those detected by the sensors, for example opening a door or turning on an appliance. The next section will precise other actions. The hierarchical aspect of the activity is kept in a tree representation, where the root indicates the activity, the intermediate nodes the sub goals and the leaves the actions.

    Figure 1. The activity hierarchical representation. Dashed nodes are not described. Constraints are given with grey background.

    Some activity features were previously highlighted in order to diagnose the deficits: the activity sequence, the habits, the location and duration of the activity, the location and duration of the action. These features lead to constraints for the detection of deficits, and then for the assistance process. The hierarchical representation of the activity takes in account these constraints by specifying each node with the task features [11]. The figure 1 illustrates the location and habits constraints specified on the root, and the duration constraint on the sub task B. The constraint of activity sequence is exemplified on the root task and one the sub task B, the former is unspecified, as the latter needs the three sub actions to be made in a precise order.

    The habits of the occupant are composed of the list of the usual ADLs and the way they are realized. They are specified in the PCA by the hierarchical representation and the several constraints on each node. When an action is performed, the PCA compares it with the usual ADL(s) ones. If the action corresponds to one of the low level nodes, the

    hierarchical structure is employed to process the action. The information is propagated through the hierarchical structure, from the action node to abstract levels, in accordance with the ADL description. At each level, the concerned node validates if this information is complying with its constraints [12]. If not, the PCA diagnoses one of the four cognitive deficits presented below.

    In addition to the detection of the cognitive deficits, the representation of the activity and the knowledge of the habits are useful to carry out the assistance. This structure informs the PCA to the appropriate next step to complete. Then, when necessary, this information is provided to the occupant.

    4.2 Sensors in the smart home

    Distributed sensors provide information about the occupant behaviour. This concrete information (e.g. open a cupboard) is used by the system to infer the abstract goal of the

    occupant (e.g. prepare a coffee).

    In the DOMUS laboratory, a typical one-bedroom apartment, various types of sensors are used, as illustrated in figure 2. It allows monitoring several types of actions. In order to follow different ADLs, doors (including cupboards and closets ones) and drawers are all equipped with electromagnetic contacts. These sensors provide information about the opening and closing actions. These are relevant information for the monitoring of heavy cognitive load activities, such as meal preparation. Some specific objects are essential for the completion of an ADL. For example, the coffee pot indicates whether or not the activity preparing coffee is under progress. Such crucial objects are equipped with

    electronic tags by the mean of Ultra Wide Band (UWB) technology. They allow to localise the object and to infer when they are moved.

    It is essential for the PCA to locate the occupant to distinguish between the attention and planning deficit as previously noticed, but also to provide assistance at the right place. The infrared sensors are installed in each room of the DOMUS laboratory. Some critical areas are covered with infrared sensors or sensitive rugs. Infrared sensors provide general information about the location of the occupant (room scale), sensitive rugs allow to delimit small area at strategic places (e.g. in the front of the cooking-range or in the bed).

    Figure 2. Distribution of sensors in the environment: (a) infra red, (b) sensitive rug, (c) flowmeter, (d) electromagnetic contact and (e) electronic tag.

    4.3 Assistance in the smart home

    The objective of the assistance is to provide, in the environment, relevant information to the occupant to help him completing the impaired ADL. Media used to that purpose are called effectors. Effectors constitute the interface between the occupant and the PCA. The pervasive computing paradigm leads us to use different types of effectors according to three types: 1) traditional Graphical User Interfaces (GUI), 2) Tangible User Interfaces (TUI) and 3) Multimedia approach.

    A traditional GUI takes place in a screen/pointing device environment. It allows the system to display textual information and, the user to interact with the system by selecting options. In the DOMUS laboratory, touch screen panel are used to display GUIs. However, this traditional approach is not really adapted for cognitive impaired people. The screen-centered vision does not allow offering the information where the occupant needs it. The GUI necessitates abstraction to transfer the information displayed on a screen in the real environment. This could be beyond the scope of the clientele. In the TUI paradigm, every day artefact becomes a medium for the interaction between the system and the user [7]. The object is used as a support of information relevant to the information and to the object function. As example, a plate modifies its colour to inform the user about the temperature of its content.

    This approach is very relevant to the PCA as it assists the occupant in his environment in the core of the ADL completion. Two TUIs are used in the PCA to provide assistance. The UWB tags, previously used to locate the objects, are also equipped with luminescent diodes and acoustic device. This is used to highlight specific objects in order to recall to the occupant its location or to focus on it. Objects are then at the core of the human PCA interaction. The DOMUS laboratory is also equipped with Ariane Controls power line communication systems. This system enables to use lighting devices dispatched in the

    environment to focus on specific elements. Tags and power line system are the firsts steps of a more global approach of TUIs for cognitive assistance, transforming the environment in a mediated space for the ADL completion [7].

    Finally, the multimedia approach, more particularly audio cues, is also used to provide relevant information to the occupant. Speakers, present is all the rooms, are used as effectors. A visual presentation of all these sensors is shown in figure 3.

    Figure 3. Distribution of effectors in the environment: (1)a tag used to highlight a food container, (2) a lighting device used to highlight a cupboard, (3) touch screen and (4) speakers.

    5 Results: Providing assistance for the morning routine

    Based on the detection of the cognitive deficits and on the pervasive computing paradigm, the PCA assists the cognitively impaired people to complete her ADL. Each cognitive deficit necessitates a specific approach. The three types of interface allow mixing a specific cognitive deficit with a specific interface in order to increase the scope of the advice displayed. We present here the major principles used to detect the need of assistance and to provide it. The cognitive assistance is delivered according to two axes: 1) on the PCA initiative when a wrong action is detected and 2) on the occupant initiative when he feels no competent. The cognitive assistance is delivered inside the smart home but the occupant or the actions performed may require a caregiver intervention. In those two cases, a caregiver is involved with, respectively, synchronous and asynchronous ways to exchange about the difficulties met by the occupant.

    The results concern the morning routine chosen for its regular behaviours. This routine contains significant ADLs, like hygiene activity or meal preparation. In this section, we focus particularly on two activities: preparing coffee and taking medicine.

5.1 Providing assistance on the PCA initiative

    The PCA initiative to provide assistance addresses initiative, planning and attention deficits.

    When the PCA detects an initiative problem during the morning routine, it points out the first task the occupant has to complete. As the occupant looks distressed the aim is to provide him cues in order to help him beginning the activity. The traditional interface GUI gives the opportunity to display textual information about the first task to complete. The TUI allows bringing cues in the environment to motivate the occupant. This is done by highlighting a specific screen in the kitchen environment, with a lighting device controlled with power line communication system. Using the screen allows to provide a more abstract information than showing directly in the environment the first concrete action to perform. In order to initiate the activity preparing coffee, the PCA displays the

    message "to take a coffee, you should first take a cup", instead of highlighting the cups' cupboard with no apparent reason. However, if the occupant is then missing the cupboard opening, the appropriate cupboard is highlighted.

    At the contrary, highlighting the cups' cupboard is the approach retained to address planning deficits. It is necessary to recall the next concrete step to perform, because the occupant is still aware of the activity he wants to achieve but he doesn’t know how to do it. Lighting system and tags are used to show directly in the environment the elements involved in the next task. This step-by-step assistance guides the occupant through the activity completion.

    When an attention deficit is detected, speakers provide audio cues. The acoustic prompt "You should finish to take your coffee" is aimed to provide a substantial impact on some one who is easily distracted by other stimulus. As the person has been moved, the visual and acoustic cues issued from the activity location, are not likely to draw the person back to the room. An acoustic signal has a good potential to attract the occupant attention. 5.2 Providing assistance on the occupant initiative

    The occupant initiative to ask for assistance addresses memory deficits. The PCA provides a tool, namely the touch screen, which can be used by the occupant to express that he needs assistance. The occupant can 1) ask the PCA to show him on a map the objects implied in one activity completion and 2) ask the PCA to show him the objects in the environment.

    In the first case, a map of the smart home is displayed on the screen. The objects involved in the activity are graphically represented with coloured points, and localised thanks to the localisation system. For the activity preparing coffee, the occupant may be informed

    upon the sugar pot position. This traditional graphical user interface is the first way the PCA addresses a memory deficit concerning objects implied in one activity completion. When the occupant prefers to visualise the objects in his environment, the TUIs are used. The cupboard where the sugar is stored is highlighted thanks to a lighting device. The same approach is used for the cups' cupboard. The containers may also be spotted by the means of visual and acoustic cues providing with their tag. The occupant is then informed of the objects position directly in his environment.

Report this document

For any questions or suggestions please email