1. Introduction

For products used in space, aeronautics, automotive, and other mission-critical fields, developers use design methods to detect defects in functional elements and interfaces from the design stage. For example, Architecture Analysis and Design Language (AADL) [1], System Theoretical Accident Model and Processor (STAMP) [2] [3], and System Theoretic Process Analysis (STPA) [2] are applied. In third-party verification using the Japan Aerospace Exploration Agency (JAXA) publisher's IV & V process [4] [5], concrete methods for design reviews by third parties based on the upper design document are reported.

For consumer products, the authors published a third-party verification method for system products including defect estimation with detection via third-party verification [6]. This method emphasizes the fact that Internet of Things (IoT) system operation can be expressed as a linkage of subsystems. From the failure result detected by third- party verification, by tracing the linkage of the subsystems, it is possible to estimate the subsystem as the failure occurrence site and its cause. For faults detected using third-party verification, developers must reproduce and analyze fault occurrence to correct faults, but the location and cause of the fault are known in third-party verification on a subsystem basis. If results can be shown in that manner, a developer’s work is expected to be easier.

In an IoT system, sensors and actuators located at data generation sites are connected to the cloud over the network. Data transmission, analysis, feedback of results, etc. are accomplished independently of human intervention. An IoT system has various components. It characteristically comprises physical (physical) objects such as sensors and actuators and virtual (cyber) objects on the cloud. Because of the diverse technologies and applications involved, commercializing an IoT system demands that one ascertain the product aim and the basic technology base, and that one ascertain the issues and targets to be solved and to orient them. By creating an IoT system prototype and by verifying the degree of achievement of the main specifications at an early stage, the tasks to be done before completion can be clarified. And the period until commercialization can be shortened. Therefore, objective evaluation must be done to ascertain whether it can be commercialized by additionally developing functions and performance for the prototype. The authors published a method to verify the IoT prototype system from a third-party perspective. Based on the verification results, the authors calculate the difficulty of developing a prototype to product completion, reporting it as a distance [7] [8].

However, because the product specifications are established at the beginning of development, it remains unclear whether a product satisfies the user's request or has appealing features. Therefore, we confirmed the reaction of the market with restricted users for a prototype at the stage of prototype verification. Based on the evaluation results and the awareness of developers obtained during prototype development, we propose a method of establishing product specifications by clarifying the purpose of use and the realization method.

For this study, we applied a product specification determination method by third-party verification of an IoT system prototype to a hydroponic cultivation system and drone system for deterring harmful birds. We then confirmed the effectiveness of this method.

2. Direction to Product Development by Verification Based on a Third-party Perspective of the IoT Prototype

2.1. Prototype to Product Orientation

To commercialize the IoT system, it is necessary to determine the product aim and the basic technology base, and to direct development in a way that defines issues and targets to be solved. By verifying the prototype from a third-party perspective, one can ascertain the ease of development: whether the prototype can be developed as it is, and whether the function and the performance can be developed into a finished product. Furthermore, product specifications can be confirmed by prototype type verification with a few users. Consequently, the direction of development can be achieved, as shown in Figure 1. In (1) prototype in the figure, (2) third-party perspective verification is conducted, and (3) a verification result is obtained. From this verification result, we obtain distance (5) from (1) prototype to (4) product completion, and obtain (6) definite product specification with a clearer user request and implementation method.

Figure 1. Direction from prototype to product

2.2. Ascertain the Distance to Product Completion by Third-party Perspective Verification of a Prototype

The authors published a method to verify the IoT prototype system from a third-party perspective and calculate the difficulty of development as a distance from prototype to product completion based on the verification result [7] [8]. To quantify the verification results, we use the Analytic Hierarchy Process (AHP) [9] [10]: a problem-solving decision-making method. The following three items used from the viewpoint of verification should be the AHP evaluation criteria.

(a) Achievement of specifications as a prototype

(b) Scalability to products

(c) Hazard control

We define the evaluation level of the verification result against three evaluation criteria in three stages: good, normal, and inadequate.

Validation results to be implemented based on evaluation criteria are expressed objectively as the number of successful cases / the total number of items to be verified. They are evaluated according to the three evaluation levels. We use two alternatives product and prototype for AHP and use absolute comparison for comparison among alternatives. Because the total evaluation value of product is represented by "1,” if the total evaluation value of prototype is multiplied by 100, then the prototype verification result can be expressed as a percentage of product. This value is the distance to complete the product.

Using the method explained above, at the time of prototype creation, the concept of distance from prototype to product completion was introduced to demonstrate the ease of development. However, the product specification at this point is the requirement definition defined at the start of development: whether it meets the user purpose, whether the developer intention is reflected or not, and whether the implementation method remains ambiguous. Therefore, it is necessary to confirm the product specifications.

2.3. Establishing Product Specifications by Third-party Perspective Verification of the Prototype

Evaluate the IoT system prototype for a trial by a few users. Based on this evaluation result, developers visualize user requests, extract development subjects, and clarify product specifications. Items to be clarified are the following two.

(a) Clarification of purpose of use

(b) Clarification of implementation method

Figure 2. Clarification of product specifications

An outline of this method is presented in Figure 2. For the prototype in (1) in the figure, (2) the limited users suitable for the tester of the prototype such as age, number of people, sex, experience etc. perform the user evaluation. Then (3) the evaluation result is obtained. Based on this, the developer creates the characteristic tag of the limited user and the user story vote of the user request. The developer (4) visualizes the user request. From the user story vote, the developer clarifies (5) clarifies the purpose of use, (6) clarifies the realization method, and (7) obtains definite product specifications.

The user evaluation is conducted such that after a few users operate the prototype for a certain period of time, the developer interviews the users. The method from interview to finalization of product specifications comprises the following three steps.

1. Evaluation items of restricted users (creation of operation record): Because it is necessary to extract implicit requests of which the user is not aware, the developer presents questions to the user using the contextual inquiry method [11] [12]. The developer infers the user experience from the context of the user answers. Because the few users are conscious of operations, it is necessary to devise questions for which they can recall the operation later. For this reason, we request creation of an operation record at the time of using the prototype, which consists of items with the time and operation clarified to the users. Table 1 shows operation record items.

Table 1. Items of operation record

2. Evaluation result (creation of interview record): First, tell the user the theme of "What I want you to teach." From the operation records in Table 1, find work items that are regarded as singular points such as performing repetitive operations, performing treatment at a time when the treatment result is not good, and performing at a time not meeting the work cycle. Next, start an interview with "What did you do?" and "Please tell us about that time." to infer the user's behavior based on the date. Then let them tell you the user experience in question form. We will not ask the question "why" in that interview. Instead, by asking for detailed information such as "Please tell us about this time.” "How long have you been doing this?" "What you did before?" and "What you did after that?” Listen to the experiences. Developers must aim to ascertain the user experience. As evaluation results, we recorded all utterances by users during an interview of about 30 min.

3. Visualization (Create User Story Vote): Extract from the interview record items representing user's characteristics such as the user profile, actions to be episodes to themes, symbolic comments and related photographs. Next, summarize the user's request that this is the case. To this, we create a user story vote in the form of who will operate what on one leaf. Repeat rearrangement operations such as arranging user story votes in chronological order or grouping and extracting purposes of use. Then, we associate development technologies to realize this extracted use purpose. Figure 3 presents an example of operation for extracting the usage purpose from the user story vote and corresponding implementation method. As a result, the user request and the realization method are clarified. The definite product specifications are obtained.

Figure 3. Extracting user purpose from user story votes

3. Implementation

Implementing this verification method for evaluation was conducted on a prototype of hydroponic cultivation and on a prototype of unmanned aerial vehicle (UAV, drone) deterrence of harmful birds. The former is one in which the device does not move. The latter is one in which the device moves.

3.1. Hydroponic Cultivation Prototype

3.1.1. Function of Hydroponic Cultivation Prototype

Large-scale systems such as lettuce are already commercialized in hydroponic culture systems. For plants other than leafy plants, many reports have described IoT systems aimed at collecting breeding data for controlling the illumination intensity and irradiation time of blue / red light, temperature, humidity, and culture solution to shorten the growing period and to facilitate stable cultivation [13].

The authors developed prototypes using device gateways and commercial cloud Amazon Web Service IoT (AWSIoT) with the aim of developing a practical system with the purpose of accumulating various plant growth data.

Figure 4 shows the system configuration of the hydroponic prototype developed this time. (1) The device constitutes a hydroponic cultivation prototype with sensors and actuators such as a temperature humidity sensor, illuminance sensor, water temperature sensor, pH measurement sensor, EC measurement sensor, and color LEDs. Irrigation of the culture solution is conducted manually. Temperature, humidity, and the solution concentration detected by the sensor are transmitted to (3) cloud via (2) gateway. The plant growth data of the device are displayed to the control terminal (4) owned by the operator using the application on the cloud. According to training plan (4), the operator gives an instruction to change the LED illuminance from the control terminal. Instruction data of the LED illuminance are transmitted to the cloud (3). The application on the cloud transmits the value of the color LED illuminance to the device (1) via gateway (2). Temperature control is maintained with an externally installed cooler. The main specifications of the hydroponic cultivation prototype are compared with the product specifications and are presented in Table 2, which shows (a) the specifications of the product, and (b) the function realized by the prototype.

Figure 4. Prototype system structure for Hydroponic culture

Table 2. Product and Prototype Specifications (Hydroponic culture)

3.1.2. Establishing Product Specifications by Third-party Verification of the Hydroponic Cultivation Prototype

Table 3 presents operation record items of the hydroponic prototype recorded by a few users every day. Based on this table, we selected irregular items, frequently executed items, visually conducted items, and manual operation items. We interviewed the few users with the first question items as to what they did by designating the execution date. Results clarified the following three purposes of use shown in Table 4 and the realization method corresponding to it. (1) The automatic irrigation function with the automatic water level adjustment function can prevent the user's water level observation error. (2) The room temperature can be made uniform using a blower fan. (3) By saving web camera data on a daily basis, one can confirm the past growing conditions.

Table 3. Item of operation record of hydroponic cultivation prototype

Table 4. Purpose of use and implementation method obtained from evaluation results of hydroponic cultivation prototype

3.2. UAV Deterrence of Harmful Bird Prototype

3.2.1. Function of UAV Deterrence of Harmful Bird Prototype

Fruit orchards are heavily damaged by crows and starlings. Even if fireworks and protective nets are installed as measures, the birds soon become accustomed to them. Such measures have little effect. Some reported systems with UAVs simulate natural enemies such as hawks and eagles, which these pest birds fear [14]. The authors developed a prototype for evaluating the effectiveness to deterring birds corresponding to a specific orchard. First, we created a robust drone flight and then combined it with a surveillance camera to monitor the penetration of birds. We launched the drone and flew back on a preset route and implemented the function. The prototype system structure for UAV deterrence of harmful birds is portrayed in Figure 5. Figure 5(1-2) shows a fixed surveillance camera that detects the appearance of a harmful bird. The detected image is transmitted to (3) the server. Then the application on the server notifies the detection event of the appearance of harmful birds to the control terminal (4) of the operator. Upon receiving the notification, the operator starts (1-1) the device UAV using the (1-3) UAV starting device. (1-1) The UAV flies along a preset route and returns. UAV sends altitude, GPS information, temperature atmospheric pressure via (2) gateway to the server (3) with the installed sensor. Table 5 shows the main specifications of the UAV deterrence of harmful bird prototype against product specifications. The table presents (a) the requirement definition specification of the product and (b) the function realized by the prototype.

Figure 5. Prototype system structure for UAV deterrence of harmful birds

Table 5. Product and Prototype specification (UAV deterrence of harmful birds)

3.2.2. Establishment of Product Specifications by Third-party Verification of a UAV Harmful Bird Deterrent Prototype

Table 6 presents the operation record table of the UAV deterrence of harmful bird prototype by a few users during UAV operation. Based on this table, we selected many operations, visual items, manual operations, and the harmful bird deterrence result. We interviewed the few users with the first question items as to what they did by designating the execution date. We clarified the following four purposes shown in Table 7 and the realization method corresponding to it. (1) By increasing the number of surveillance cameras according to the crow recognition rate, erroneous crow recognition can be reduced. (2) By improving the landing site accuracy by GPS and by performing automatic charging, one can make the UAV fly repeatedly against a returning crow. (3) By installing a web camera in the drone, a developer can collect realization data of the crow tracking function. (4) Waterproofing functions are necessary for flights in rainy weather, but such functions are unavailable now.

Table 6. Item of operation record of UAV deterrence of harmful bird prototype

Table 7. Purpose of use and implementation method obtained from evaluation results of UAV deterrence of harmful bird prototype

4. Evaluation and Consideration

4.1. Evaluation

Based on evaluation results obtained from a few users of the prototype, we propose a method to clarify the purpose of use and the method of realization, and determine the development specifications by visualizing user requests. Furthermore, the method was applied to a hydroponic prototype and UAV prototype for deterrence of harmful bird and was realized as a product. Results were obtained from comparing the original development specifications and the product specifications realized based on the usage purpose clarified from a user evaluation and the corresponding realization method.

1. Hydroponic cultivation prototype

(1) For automatic irrigation, a water level adjustment function exists for improving user operability.

(2) With regard to air blowing, in addition to the purpose of covering the leaves and preventing CO₂ from being absorbed, the objective of making the room temperature uniform is realized.

(3) Although similar to the requirement definition that a plant image is required as a web display, it also realizes a past image data display function as well as current day data, which is a user request.

2. UAV harmful bird deterrence prototype

(1) For control of 10 monitoring cameras, we realized a function of adding cameras according to the crow recognition rate.

(2) Regarding the flight time, the requirement definition specification was 30 min. However, for deterring crows, the GPS accuracy of the landing point is improved and automatic charging can be performed rather than extending the flight time, making repeat flights possible.

(3) Although not defined in the initial specifications, the developer devised the addition of a crow tracking function to raise the effect of deterring crows. To gather development data for this purpose, we implemented the web camera as a product on the drone.

(4) Because crows start to appear from the rainy season, flight during times of rainy weather was requested. Although a waterproof function is necessary, we decided against implementation now because it departs from the premise of the initial requirement specifications.

As described above, compared to the original development specifications, the development purpose corresponding to the purpose of use and the corresponding development method were clarified. Therefore we conclude that this method is effective for clarifying product specifications.

4.2. Consideration

As proposed in this report, we considered a prototype evaluation method by a few users.

1. One must consider differences between a few users and real users. Regarding the number of people who perform user evaluation, Jacob–Nielsen's theory holds that five people can represent the requests of 85% of all users [8]. However, because the IoT field has widely diverse applications, it is probably necessary to perform user evaluation for each user layer when the users using the system can not be narrowed down.

2. Implementing the Contextual and Inquiry Method (question form). Because this method is a method of forming questions to compel users to describe the user experience, we prepared no questionnaire in advance. However, if nothing is prepared, one might expect that some users will say nothing. For this reason, it is necessary to prepare to listen carefully in accordance with the prototype system, preparing tools that are reminiscent of operations such as a user's operation record or asking about operations on the spot.

3. The timing of evaluating prototypes by limited users varies depending on the target IoT system. In IoT systems targeting agricultural fields and homes, the possibility exists of releasing prototypes particularly addressing limited users early in the development that all functions are not implemented. In fields close to mission critical fields such as manufacturing and automobiles, we believe it is necessary to conduct evaluation with expert evaluators such as test drivers at the stage of near-completion.

5. Conclusions

As described herein, as a method of establishing the product specifications of the IoT system, a prototype system evaluation by a few users is performed, the developer ascertains the user experience as a user episode, and creates a user story vote corresponding to the user operation by extracting development tasks. Subsequently, we proposed a method to clarify the purpose of use and the corresponding development method. As a result of applying this method to a hydroponic cultivation prototype and UAV harmful bird deterrence prototype, we obtained the finding that limited user evaluation of prototypes is effective. I plan to apply this method to the remote control system of household electric appliances that is expected to expand in the future. Furthermore, I intend to consider a method that does not depend on experience in creating User Story Vote. Because the application range of IoT systems is expected to expand to new markets, we intend to accumulate data of evaluation results and increase application examples as future tasks.

ACKNOWLEDGMENTS

I thank Mr. Nakajima of VSN Co., Ltd. for cooperation in implementation verification.