International Journal of Internet of Things

2018;  7(1): 10-15

doi:10.5923/j.ijit.20180701.02

 

Behavior Sensing in Intelligent Space System

Shoichi Okazaki1, Akiko Watanabe2, Hidetoshi Kambe3, Masahiro Inoue4, Hisao Koizumi5

1Mobile Computing Promotion Consortium, Tokyo, Japan

2Department of Architecture, Tokyo Denki University, Tokyo, Japan

3Faculty of Science and Technology, Tokyo Denki University, Saitama, Japan

4Department of Electronic Information Systems, Shibaura Institute of Technology, Saitama, Japan

5NPO Organization M2M/IoT Study Group, Kanagawa, Japan

Correspondence to: Shoichi Okazaki, Mobile Computing Promotion Consortium, Tokyo, Japan.

Email:

Copyright © 2018 The Author(s). Published by Scientific & Academic Publishing.

This work is licensed under the Creative Commons Attribution International License (CC BY).
http://creativecommons.org/licenses/by/4.0/

Abstract

The Internet of Things (IoT), connecting devices with the internet, is making progress. IoT connects various world objects to internet, such as information and communications equipment, electrical home appliances, and medical equipment. Furthermore, IoT provides services through mutual communication such as equipment failure prediction, remote monitoring, and telemedicine. A smart house is one IoT application. It autonomously controls household appliances, including air conditioners and water heaters, to achieve optimal home energy consumption. Whereas a smart house specifically examines energy consumption, intelligent space systems are also making progress, providing a secure and safe living environment by sensing of human motion in a residential space. We comprehend human motion in an intelligent space system and propose a method by which people can live in a house comfortably, securely, and safely. We adopted a behavior sensing method that ascertains the timing of a person's movement between zones and implements abnormality detection by remote observation.

Keywords: Intelligent space, Smart house, IoT, Behavior sensing, Sensor

Cite this paper: Shoichi Okazaki, Akiko Watanabe, Hidetoshi Kambe, Masahiro Inoue, Hisao Koizumi, Behavior Sensing in Intelligent Space System, International Journal of Internet of Things, Vol. 7 No. 1, 2018, pp. 10-15. doi: 10.5923/j.ijit.20180701.02.

Article Outline

1. Introduction
2. Significance of Behavior Sensing in Intelligent Space System and Relevant Studies
    2.1. Significance of Behavior Sensing in Intelligent Space System
    2.2. Relevant Studies of Behavior Sensing and Abnormality Detection
3. Proposal of Behavior Sensing Method in Intelligent Space System
    3.1. Characteristics of the Proposed Method
    3.2. Constitution of Intelligent Space System
    3.3. Behavior Sensing Method
    3.4. Abnormality Detection Method for Behavior Sensing
4. Mounting
    4.1. Mount Configuration of Intelligent Space System
    4.2. Mounting of Behavior Sensing in Intelligent Space System
5. Evaluation and Discussion
6. Conclusions

1. Introduction

The development of sensing technologies [1] has produced sophisticated intelligent space systems that observe human motion in spaces such as residences and which detect the behavior and abnormal state of people in such spaces. An intelligent space system can provide a comfortable living environment by controlling equipment such as robots and electrical home appliances properly. It can perform watching tasks by detecting an abnormal state of a person according to the results of observing human motion. An intelligent space [2, 3] refers to a series of these processes.
Various methods exist for watching people living alone with diverse sensor data in space. The major methods include the following:
- a method by which a target person to be watched wears an instrument on the body, and the person's movement and health conditions are observed remotely,
- a method by which cameras and equipment are installed in space and human motion is observed constantly, and
- a method by which various sensors are installed to the floor, a door, furniture, etc. in a house, and human motion is observed.
The Internet of Things (IoT) [4-8] connecting devices with the internet is attracting attention. IoT can adjust the temperature and illuminance in a house automatically, or can perform remote watching, etc. by connecting various things to the internet, such as information and communications equipment, electrical home appliances, and medical equipment.
We propose a behavior sensing method in an intelligent space system and a method for detecting the occurrence of an abnormal state using IoT technologies. For detecting this abnormal state, it is important that detection equipment be installed easily in a home. In addition, it is fundamentally important to adopt methods to keep the target person not to be noticed that they are being "looked at," "monitored," etc. and to provide an environment implementing this. For this study, we have adopted and estimated a method to detect abnormality by sensing of a person's movements between rooms in an intelligent space system. Chapter 2 of this paper describes conventional methods of relevant studies. Chapter 3 proposes a method of abnormality detection by behavior sensing. Then Chapter 4 describes details of mounting of our proposal. Chapter 5 explains an evaluation and presents discussion.

2. Significance of Behavior Sensing in Intelligent Space System and Relevant Studies

2.1. Significance of Behavior Sensing in Intelligent Space System

The purpose of an intelligent space in a dwelling is to provide persons who live in a house with a comfortable life, and to secure their relief and safety. The significance of behavior sensing in an intelligent space system is the following.
(i) Situations and human motion in a space are observed using sensors installed in the space.
(ii) Sensing of human behavior is performed for extracting useful information, which is provided to people and equipment.
(iii) Robots and equipment in a space are controlled properly, providing people with a comfortable life and a secure and safe environment.
An intelligent space system handles not only data from various sensors but human behavior as sensing data, and provides service to people through actuators, etc. Figure 1 presents a typical constitution of an intelligent space system. Figure 1(a) presents the hub of an intelligent space system, which gathers measurements from various sensors (b) and the data of human motion detected with motion detection sensors, and transmits the data to an intelligent space application on a cloud (c). An application on a server performs a data analysis process, and transmits the result to the hub (a). The hub controls actuators in the intelligent space system (d) or provides people with required information using the received data.
Figure 1. Concept model of intelligent space system

2.2. Relevant Studies of Behavior Sensing and Abnormality Detection

We surveyed past related studies [9-12] of two points: methods of comprehending human behavior and methods of detecting an abnormal state of a person in an intelligent space system. Major methods used in studies related to the points above are explained below.
Conventional behavior comprehension methods use robots, infrared sensors, tags carried by people, tags and their readers/writers placed in a space, and monitoring cameras. In this case, it is necessary to ascertain whether a target person carries an instrument [13, 14] such as a wearable device, or large-scale equipment [15] such as cameras be placed in the space.
Conventional abnormality detection methods keep grasping human behavior constantly using monitoring cameras, infrared sensors, etc. These methods observe human behavior at all hours, and detect abnormality based on the observation data.
The survey results of relevant studies have convinced us that a handy method is fundamentally important in an intelligent space system in which a person to be watched need not wear any special instrument, but the behavior of the person in a house can be grasped and abnormality can be detected.

3. Proposal of Behavior Sensing Method in Intelligent Space System

3.1. Characteristics of the Proposed Method

The aims of the proposed method in an intelligent space system are the following.
(i) To grasp a person's moving between zones corresponding to the rooms of a house by sensing of human behavior in an intelligent space system.
(ii) To detect abnormality based on sensing of human behavior for sustaining a comfortable life.
A person is deemed as being in an abnormal state when the person’s behavior deviates from daily life behavior patterns that are established in advance. The purpose of abnormality detection is to secure safety in remote watching and to provide an assisted living function by analyzing details of the abnormality and predicting an abnormal state.
The proposed method takes the following measures against the subjects described in relevant studies of Section 2.2.
For sensing of human behavior, this method uses beacons and motion detection sensors placed in each room to ascertain the room with a target person to be watched. The target need not wear any special instrument. In addition, using a smart device as a gateway even when the gateway is placed in living environment, a target's behavior can be observed without the resident's discomfort.
Regarding abnormality detection, this method specifically examines movement among rooms corresponding to a zone and determines whether a moving pattern is normal or abnormal. Furthermore, an abnormality is deemed as occurring when the duration of stay in each zone exceeds a definite period of time. Because a smart device such as a smart phone is adopted as a gateway, a target can be alarmed when it is determined as abnormal using the voice or video of the smart device. Furthermore, this method is readily applicable to various structures of houses. It merely requires measurement of data of moving between zones.

3.2. Constitution of Intelligent Space System

Figure 2 depicts the basic architecture of this method. Data collected by sensors such as motion detection sensors and a temperature sensor placed in a space are gathered by an intelligent space sensor device (a) in Fig. 2. The gateway manager of the intelligent space gateway (b) gathers data through the Bluetooth Low Energy (BLE), and transmits the data to the intelligent space manager of server (c) on the internet. The gateway manager conducts data processing using a database that manages the constitution of sensors and actuators. An application on the server analyzes data collected from motion detection sensors, etc. based on a database that manages the sensor data of the whole intelligent space system. Then it transmits data of the analysis results to the gateway.
Figure 2. Intelligent space system structure
We prototyped a sensor device that can detect human motion using an infrared sensor and an intelligent space sensor microcomputer. The mounting layout of the sensor device is depicted in Fig. 3. The sensor device has a sidewall dimension of about 70 mm × 50 mm, with approximately 40 mm height including an infrared sensor. The sensor device is driven by a lithium cell with low self-discharge.
Figure 3. External appearance and size of sensor device

3.3. Behavior Sensing Method

Figure 4 depicts the proposed behavior sensing method. The proposed system divides a space into multiple zones, and determines a zone where a target person is present. This method does not grasp the location of a person. It is a behavior sensing method that detects the timing of movement of a person between zones. Comprehension of movement between zones is performed based on data from motion detection sensors placed in each zone.
Figure 4. Behavior model in intelligent space
Figure 4 divides the inside of a space into multiple zones from Zone - 1 to Zone - n. A zone corresponds to a room in a house. A person moves freely among zones. Each of motion detection sensor devices 1 - n is placed, respectively, in each zone. It performs sensing of human motion. The motion detection sensor placed in each zone notifies to the server on the cloud via the gateway when a person's motion is detected. The notification to the server is given when the motion of a person across a zone border is detected.

3.4. Abnormality Detection Method for Behavior Sensing

A person is deemed in an abnormal state when the person’s behavior deviates from the normal behavior pattern, which is defined in advance.
As for comprehension of human behavior according to behavior patterns, for example, it is determined that a person is either going out or coming home if a motion detection sensor at the entrance door reacts. It is determined that a person comes home when no sensor is reacting initially, then a motion detection sensor at the entrance reacts, and a motion detection sensor in the living room reacts. Otherwise, it is determined that a person goes out when a person is detected in the living room, then the sensor at the entrance door reacts, and all the sensors stop reaction.
Consequently, the daily life behavior of a person in an intelligent space system can be categorized into various behavior patterns using behavior as a variable by behavior sensing. Behavior patterns such as outdoor (going out) → entrance (coming home) → kitchen → washroom → living room are deemed as normal states. Alternatively, it is judged as an abnormal condition when detection of a person from motion detection sensors does not match the behavior patterns, for example, if there is no reaction after a person is detected in the bathroom, or if there is no reaction after coming home.
We investigated whether machine learning is applicable as an abnormality detection method for this proposal. Application of machine learning to abnormality detection requires large amounts of training data. However, practically speaking, almost no abnormal state can be anticipated to occur. Therefore, we inferred that machine learning would be difficult based on data for the occurrence of an abnormal state. Possible methods to apply machine learning without using the abnormal data occurrence include a method in which unsupervised learning using the abnormal data occurrence, but with learning only of normal patterns is conducted. An abnormality is regarded as occurring when a person's behavior deviates from normal patterns. Nevertheless, even in this case, large amounts of data must be used for learning of normal patterns. For this reason, we adopted an abnormality detection method using a behavior pattern table.

4. Mounting

4.1. Mount Configuration of Intelligent Space System

Figure 5 presents the mounting layout of an intelligent space system. Gateway (a) was prepared using an Android tablet with a built-in short-distance radio communication function. Motion detector (b) was prototyped using a motion detection sensor and an intelligent space sensor microcomputer. The motion detector transmitted the sensor data to the gateway manager on the intelligent space gateway through BLE. The gateway manager transmitted the sensor data to the intelligent space application on the server by 4G/Wi-Fi. The intelligent space application accumulated and analyzed the data.
We were able to detect human motion using motion detection sensors placed in the intelligent space system and to communicate with a target person in the house from a distant place with a speaker and a microphone.
Figure 5. System configuration of applied system

4.2. Mounting of Behavior Sensing in Intelligent Space System

Table 1 depicts the table of rules for detecting abnormality from behavior patterns. Whether human behavior in an intelligent space system is normal or abnormal is determined by observation of human behavior in which room a person is present and to which room the person moves next based on sensor data transmitted from sensors.
Table 1. Rule of anomaly detection
     
The movement of a target person between rooms is expressed with from/to in the list. The behavior pattern of the person is determined as normal or abnormal. "OK" and "x" in the list respectively denote a normal and abnormal pattern. It is also judged as abnormal when the duration of stay in each room exceeds a preset maximum period.
Figure 6 depicts measurement result examples of normal/abnormal determination in the mounting case of Fig. 5. Regarding the method of detecting a person in a room, when data were received from multiple sensors simultaneously, a dataset from a sensor with the most intense field intensity (dB value) was adopted. Figure 6 presents a target person's movement history between rooms based on detection data from motion detection sensors placed in the entrance, living room, kitchen, washroom, and bathroom. The solid line and dotted line respectively represent behavior at a normal state and when abnormality is detected.
Figure 6. Error detection by abnormal pattern
Normal behavior patterns in daily life include entrance – living room, living room – kitchen, living room –washroom and living room – bathroom. Normal behavior can be inferred through sequences and combinations. Movements (a) → (b), (b) → (c), (c) → (d), (d) → (e), and (e) → (f) portrayed in Fig. 6 are regarded as normal behavior patterns by the table presented in Table1. Although (f) → (g) is daily behavior, because a response from other than the bathroom stopped after (g) and timeout occurred, this case was judged as abnormal.

5. Evaluation and Discussion

(1) Remote watching
We implemented remote watching and an assisted living function in an intelligent space system. We established communications with a resident in an intelligent space system as an application on a tablet used as a gateway to the intelligent space. We also implemented speech such as an announcement of an action schedule to elderly people using a speech recognition function or a text read-out function supported by Android 4.4 using Android APIs.
(2) Behavior sensing
We verified not only that a comfortable lifestyle can be implemented but that remote watching and abnormality detection can be performed by behavior sensing, through control of electrical home appliances in a house as an intelligent space system. We conducted monitoring of human behavior as the basis of an intelligent space model, which confirmed the effectiveness of the model that determines a normal/abnormal state based on behavior patterns.
We infer that it is possible to control instruments using a combination of various sensors/actuators that constitute an intelligent space system with human motion. We also assume that this method can accommodate the abnormality detection of human behavior flexibly because human behavior in space takes no finite form.
(3) Sensor application technology
Adoption of a smart device as an intelligent space gateway shortened the development cycle of this intelligent space system. A liquid crystal display and a voice input–output function of a tablet implemented applications such as remote watching and an assisted living function easily. Mounting of an assisted living function by combining a smart device and an intelligent space microcomputer demonstrated that the whole intelligent space system can be treated as a feedback control system, as a kind of robot.
(4) Future subjects
The abnormality detection method of the proposal was able to judge an abnormal state according to whether the person's behavior pattern was normal or abnormal, or by the timeout of the duration of stay in each zone, based on a behavior pattern table. Behavior patterns change greatly according to differences among individuals and in the constitution of rooms. This method was able to accommodate these differences easily by adjusting the behavior pattern table.
Our future subjects include accumulation of data of normal behavior patterns as the next step, and then detection of the occurrence of abnormality by machine learning based on normal behavior patterns.

6. Conclusions

We have proposed an intelligent space system with sensors placed in spaces used by people, such as a house or a building. The system secures a comfortable life, with relaxation and safety for elderly people. Moreover, we have conducted mounting, evaluation, and discussion of the system, and verified the effectiveness of the detection method of the abnormal state of a person in an intelligent space system.
Remote watching and abnormality detection of elderly persons living alone will become important future issues with the progress of an aging society in the future. We anticipate that inexpensive and handy intelligent space systems will be based on the proposed method. Our future tasks include advanced study of this method, especially improvement of its precision for abnormality detection.

References

[1]  Tadashi Terasaki, “Innovative First Step toward Mechanoluminesecent Ubiquitous Light Source for Trillion Sensors”, Sensors and Materials, Vol.28, No.8, pp.827-836, 2016.
[2]  Shoichi Okazaki, Toru Watanabe, Masahiro Higuchi, Akiko Watanabe, Tetsuo Shiozuki, Hisao Koizumi, “An Intelligent Space System and its Communication Method to Achieve the Low Energy Consumption”, IEEJ Transactions on Electronics, Information and Systems Vol. 136 No.12 pp.1804-1814, (2016) (in Japanese).
[3]  Hidenori Hashimoto, Akiko Watanabe, “Design of Intelligent Space”, NHK Publishing, Inc., (2004) (in Japanese).
[4]  David Bosworthick, Qmar Elloumi, Oliver Hersent, “M2M Communications”, John Wiley & Sons Ltd, (2012).
[5]  Luigi Atzoria, Antonio Ierab, Giacomo Morabitoc: "The Internet of Things: A survey", Computer Networks, Vol. 54, Issue 15, pp. 27872805 (2010-10).
[6]  Mohamed Ali Feki, Fahim Kawser, Mathieu Boussard, and Lieven Trappeniers: “The Internet of Things: The Next Technological Revolution”, IEEE Computer, Vol.46, No.2, pp24-25 (2013).
[7]  Steve Hodges, Stuart Taylor, Nicolas Villar, James Scott, Dominik Bial, and Patrick Tobias Fischer: “Prototyping Connected Devices for the Internet of Things”, IEEE Computer, Vol.46, No.2, pp26-34 (2013).
[8]  K. Chang, A. Soong, M. Tseng, and Z.Xlang: "Global Wireless Machine to Machine Standardization", IEEE Computer Society, Vol.15, No.2, pp.64-69 (2011).
[9]  Mai Tsuda, Morihiko Tamai, Keiichi Yasumoto, “A Monitoring Support System for Elderly Person Living Alone through Activity Sensing in Living Space”, IPSJ SIG Technical Report, Vol. 2013-MBL-66, No16, (2013) (in Japanese).
[10]  K. Oshima, R. Urushibara, A. Fujii, H. Noguchi, M. Shimosaka, T. Sato, T. Mori, “Behavior Labeling Algorithms from Accumulated Sensor Data matched to Usage of Livelihood Support Application”, No.09-4 Proceedings of the 209 JSME Conference on Robotics and Mechatronics, Fukuoka, Japan, May 24-26, 2009 (in Japanese).
[11]  Kazuya Suginohara, Kenichi Morooka, Tokuo Tsuji, Ryo Kurazume, “Indoor Human Behavior Estimation by combining Hierarchical Hidden Markov Model”, No.15-2 Proceedings of the 2015 JSME Conference on Robotics and Mechatronics, Kyoto, Japan, May17-19, (2015) (in Japanese).
[12]  Yasuyoshi Hyodo, Takehiro Kohashi, Taisuke Yamanaka, “Method for Work Analysis Using Human Activity Sensing Based on IoT Technology”, Toshiba Review, Vol.71, No.5, pp.72-75, (2016) (in Japanese).
[13]  Takuya Maekawa, Masaki Samejima, “Daily Activity Sens ing Technologies using Wearable Devices and Their Applications to Animal Behaviour Recognition”, Journal of the Robotics Society of Japan, Vol.35, No.2, pp.105-109, (2017) (in Japanese).
[14]  Kazushige Ouchi, Miwako Doi, “Living Activity Recognition Technology Using Sensors in Smartphone”, Toshiba Review, Vol.68, No.6, pp.40-pp.43, (2013) (in Japanese).
[15]  Taketoshi Mori, “House with MIMAMORI Function”, Journal of the Robotics Society of Japan, Vol.32, No.3, pp.255-pp.258, (2014) (in Japanese).