JP2024039333A - Emotion estimation program, emotion estimation method, and information processing device - Google Patents

Abstract

To estimate emotion with high accuracy.SOLUTION: An information processing device acquires physiological response information of a person specified from vital data of the person. The information processing device acquires physical information of the person specified from video data obtained by photographing the person. The information processing device generates a psychological characteristic showing a unique characteristic of the person by inputting the acquired physiological response information and physical information to a first machine learning model. The information processing device estimates emotion of the person on the basis of the generated psychological characteristic.SELECTED DRAWING: Figure 1

Description

The present invention relates to an emotion estimation program, an emotion estimation method, and an information processing device.

In recent years, with the development of wearable terminals, it has become easier to obtain physiological response information such as heart rate and breathing rate, and emotion estimation technology that utilizes physiological response information is being used. For example, a technique is known that acquires physiological reaction information of a target person during a telephone conversation, estimates emotions from the acquired physiological reaction information, and uses the estimated emotions to prevent special fraud.

Unexamined Japanese Patent Publication No. 2016-106689 JP 2019-153303 Publication Japanese Patent Application Publication No. 2013-46691 US Patent Application Publication No. 2014/0323817 US Patent Application Publication No. 2013/0238394

However, it is difficult to say that the emotion estimated from the physiological reaction information is highly accurate. For example, there are large individual differences in the link between psychological states and physiological responses; some people's heart rate becomes faster when they are anxious, while others' heart rate does not change even when they are anxious, so it can be estimated from physiological reactions. feelings lack accuracy.

In one aspect, it is an object of the present invention to provide an emotion estimation program, an emotion estimation method, and an information processing device that can estimate emotions with high accuracy.

In the first proposal, the emotion estimation program causes a computer to acquire physiological reaction information of a person identified from the person's vital data, and acquire physical information of the person identified from video data of the person. Then, by inputting the acquired physiological reaction information and the physical information into a first machine learning model, psychological characteristics indicating the unique personality of the person are generated, and the psychological characteristics are applied to the generated psychological characteristics. The method is characterized in that a process is executed to estimate the emotion of the person based on the emotion of the person.

According to one embodiment, emotions can be estimated with high accuracy.

FIG. 1 is a diagram illustrating an information processing apparatus according to a first embodiment. FIG. 2 is a diagram illustrating the reference technology. FIG. 3 is a diagram illustrating problems with the reference technology. FIG. 4 is a diagram illustrating a specific example of psychological characteristics. FIG. 5 is a diagram illustrating the system configuration according to the first embodiment. FIG. 6 is a functional block diagram showing the functional configuration of the information processing device according to the first embodiment. FIG. 7 is a diagram illustrating the first training data DB. FIG. 8 is a diagram illustrating the second training data DB. FIG. 9 is a diagram illustrating the third training data DB. FIG. 10 is a diagram showing an example of the victim's emotions. FIG. 11 is a diagram showing an example of emotions when a crime occurs. FIG. 12 is a diagram showing the relationship between refund fraud and emotional patterns. FIG. 13 is a diagram illustrating the relationship between me-or-me fraud and emotional patterns. FIG. 14 is a diagram illustrating generation of training data. FIG. 15 is a diagram illustrating machine learning of a characteristic estimation model. FIG. 16 is a diagram illustrating machine learning of an emotion estimation model. FIG. 17 is a diagram illustrating machine learning of a crime risk estimation model. FIG. 18 is a diagram illustrating estimation of psychological characteristics. FIG. 19 is a diagram illustrating emotion estimation. FIG. 20 is a diagram illustrating estimation of crime risk. FIG. 21 is a diagram illustrating an example of identifying emotional patterns. FIG. 22 is a flowchart illustrating the flow of machine learning processing. FIG. 23 is a flowchart illustrating the flow of crime risk estimation processing. FIG. 24 is a diagram illustrating an example of a hardware configuration.

Embodiments of the emotion estimation program, emotion estimation method, and information processing apparatus disclosed in the present application will be described in detail below with reference to the drawings. Note that the present invention is not limited to this example. Moreover, each embodiment can be combined as appropriate within a consistent range.

(Description of Information Processing Device)
Fig. 1 is a diagram illustrating an information processing device 10 according to Example 1. The information processing device 10 shown in Fig. 1 is an example of a computer device that estimates emotions with high accuracy using imaging data including videos and images of subjects when performing special fraud using telephones, detection of suspicious persons in a store, detection of employee stress in the workplace, and the like.

In recent years, there has been a significant increase in damage caused by phone scams such as "Ore-Ore" scams and refund scams. A method has been proposed that analyzes the audio of phone calls to prevent fraud. However, as criminals' methods become more complex and sophisticated, the problem is that these methods soon become unapplicable. Therefore, a detection method that does not depend on the method of crime is required.

Therefore, by focusing on emotional changes (such as nervousness and impatience) that occur in victims, which are common to all types of fraud, it is possible to detect whether a victim is involved in a fraud by estimating their emotions. In recent years, emotion estimation technology exists that utilizes physiological response information such as heart rate and breathing rate, which can be easily acquired due to the development of wearable devices.

FIG. 2 is a diagram illustrating the reference technology. As shown in Figure 2, the reference technology acquires physiological reaction information from a wearable device worn by a person during a call, and inputs the physiological reaction information into a trained machine learning model to estimate the person's emotions. presume. In the reference technique, the estimated emotions of the person are used to determine whether or not the person has been subjected to special fraud. However, there are large individual differences in the linkage between physiological response information and emotions, and there are limits to emotion estimation based only on physiological response information.

FIG. 3 is a diagram illustrating problems with the reference technology. Note that the left axis in FIG. 3 indicates the heart rate, and the right axis is an index used in psychology to express a psychological state (excitement level, etc.), and a higher value indicates a more excited state. FIG. 3(a) shows a person whose excited state decreases and changes to a calm state as the heart rate decreases, and whose physiological reactions and psychological state are linked. On the other hand, (b) of FIG. 3 shows a person whose psychological state does not change even if the heart rate decreases, and whose physiological reactions and psychological state are not linked.

In this way, there are differences in physiological responses and psychological states between individuals. In particular, in the case of people whose physiological reactions and psychological states are not linked, it is considered difficult to estimate emotions based on physiological reactions alone, so we consider using personal characteristic information as features for emotion estimation. Therefore, in this embodiment, by developing a method that absorbs the above-mentioned individual differences in emotion estimation using physiological reactions, accuracy is improved, and fraud detection is realized through highly accurate emotion estimation.

Here, psychological characteristics, which are an example of personal characteristic information employed in this embodiment, will be explained. FIG. 4 is a diagram illustrating a specific example of psychological characteristics. Psychological characteristics are characteristics that a person has, regardless of their emotions and psychological state. This psychological characteristic is also information similar to a person's personality, individuality, and characteristics, such as being anxious, not paying attention to details, and having a tendency to be suspicious of others. For this reason, there is a duality between the psychological characteristics that the person himself or herself recognizes and the psychological characteristics that people related to the person feel about the person. It can be specified by

In this embodiment, as shown in FIG. 4, the characteristics of suspicion and anxiety are used. For example, the psychological characteristics of user A are suspiciousness (50) and anxiety characteristic (24), and the psychological characteristics of user B are suspiciousness (44) and anxiety characteristic (24).

Note that as a scale representing suspicion, a scale used in POMS2 (Profile Of Mood States Second Edition) can be used. For anxiety traits, a trait anxiety scale measured by STAI (State-Trait Anxiety Inventory) can be used. Note that these measures may be calculated values themselves or may be normalized values. However, other indicators can be used as long as they are characteristics unique to a person, regardless of their emotions and psychological state.

(System configuration)
Next, a special fraud detection system using the information processing device 10 will be explained. FIG. 5 is a diagram illustrating the system configuration according to the first embodiment. As shown in FIG. 5, this system is a system in which the public and private sectors cooperate, including a special fraud prevention solution 1, a special fraud group 2, a user's home 3, and a local government 4. This system uses a crime detection model trained using criminal psychology to detect special fraud in real time, prevent the occurrence of special fraud, and train the evolving features of special fraud to help local governments and This is an example of a special fraud prevention system that provides effective guidance.

Special fraud prevention solution 1 is a service run by companies, etc., and generates and provides a crime detection model that combines criminal psychology and machine learning. This crime detection model is a model trained on the emotional patterns of users when a special fraud occurs, as determined by criminal psychology. For example, crime detection models include a machine learning model that estimates the user's emotions from sensing data, which is an example of feature data that indicates the user's characteristics, and a machine learning model that estimates the possibility of special fraud from emotional patterns. The possibility of special fraud is estimated from the real-time emotions of the user during the call.

The special fraud group 2 is criminals who call the user's home 3 and commit crimes such as special fraud. For example, special fraud group 2 includes known special frauds such as "Ore-Ore" fraud, savings fraud, cash card fraud, fictitious fee fraud, refund fraud, financial product fraud, gambling fraud, relationship brokering fraud, etc., and combinations of these frauds. and conducting unknown special frauds, including new frauds.

The user's home 3 is the home of a user, such as an elderly person, and is the user's home that is the target of fraud by the special fraud group 2. This user's home 3 includes an information processing device 10, a camera that captures video and facial images, a microphone that collects audio, a millimeter wave sensor that measures heart rate and breathing rate, and a Non-contact sensing terminals, including wearable terminals that measure waves, etc., have been installed. Further, the information processing device 10 has a crime detection model made by the special fraud prevention solution 1, and is communicably connected to a user's electronic device (for example, a telephone) and each sensing terminal.

Local government 4 will provide guidance and seminars for the elderly, and will implement various measures to prevent damage from special fraud. The training sessions exemplified here are not limited to those held by local governments such as the local government 4, but also include services provided by the private sector, training sessions held by neighborhood associations, and training sessions held at schools.

In such a system, the information processing device 10 acquires physiological reaction information (respiration rate, pulse, heartbeat, etc.) of a person identified from the person's vital data. The information processing device 10 acquires physical information (gender, age, etc.) of a person identified from video data obtained by photographing the person. Then, the information processing device 10 inputs the acquired physiological reaction information and physical information into the first machine learning model to generate psychological characteristics indicating the unique personality of the person. The information processing device 10 estimates a person's emotions based on the generated psychological characteristics.

Thereafter, the information processing device 10 estimates whether or not a special fraud against the person has occurred, based on the pattern of changes in the person's emotional state that is estimated at any time during the call. As a result, the information processing device 10 can more accurately detect whether the target person has fallen victim to special fraud using emotion estimation with improved accuracy.

(Functional configuration)
FIG. 6 is a functional block diagram showing the functional configuration of the information processing device 10 according to the first embodiment. As shown in FIG. 6, the information processing device 10 includes a communication section 11, a storage section 12, and a control section 20.

The communication unit 11 is a processing unit that controls communication with other devices, and is realized by, for example, a communication interface. For example, the communication unit 11 receives sensing data from a sensing terminal such as a wearable terminal or a millimeter wave radar, receives video data from a camera, and transmits alarms, messages, etc. to the user.

The storage unit 12 is an example of a processing unit that stores various data including sensing data, imaging data, etc., programs executed by the control unit 20, and the like, and is realized by, for example, a memory or a hard disk. This storage unit 12 stores a first training data DB13, a second training data DB14, a third training data DB15, a characteristic estimation model 16, an emotion estimation model 17, and a crime risk estimation model 18.

The first training data DB 13 is a database that stores first training data used for machine learning of the characteristic estimation model 16. FIG. 7 is a diagram illustrating the first training data DB13. As shown in FIG. 7, the first training data stored in the first training data DB 13 includes "physical information" and "physiological reaction information" that serve as explanatory variables during machine learning, and "physical information" and "physiological reaction information" that serve as objective variables during machine learning. ``psychological characteristics'' are associated with each other.

The "physical information" stored here is information related to the person's body such as gender, age, and height, and is set as numerical information such as 1 for male and 0 for female. "Physiological reaction information" is an example of information specified from sensing data and vital data that can be sensed from a person, such as heart rate and breathing rate. Information can also be set. "Psychological characteristics" refers to characteristics unique to a person, regardless of emotion and psychological state, and includes, for example, a scale showing suspiciousness and anxiety characteristics. In the example in Figure 7, a man in his 70s (1), who has psychological characteristics of 50 on the scale of suspicion and 20 on the scale of anxiety, has a heart rate of 70 and a respiratory rate of 13 ( normalized value).

The second training data DB14 is a database that stores second training data used in the machine learning of the emotion estimation model 17. FIG. 8 is a diagram illustrating the second training data DB14. As shown in FIG. 8, the second training data stored in the second training data DB14 is data in which "physical information," "physiological response information," and "psychological characteristics," which are explanatory variables during machine learning, are associated with "emotions," which are objective variables during machine learning.

The "physical information", "physiological reaction information", and "psychological characteristics" stored here are the same information as in FIG. 7, so a detailed explanation will be omitted. "Emotion" is information indicating the emotion of the person, and information that quantifies each emotion such as "angry" or "sad" can also be set. For example, "0" is set for "angry feelings" and "1" is set for "anxious feelings." In the example of FIG. 8, the heart rate of "70s, male (1)" who has the psychological characteristics of "suspicion scale of 50" and "anxiety characteristic scale of 20" is 70 and respiratory rate of 13 ( normalized value), the emotion is set to be “emotion A”.

The "emotion" set here is the user's facial expression while on the phone, and general indicators such as happy or sad can also be used, and it can also be used as a general indicator such as "anger to hostility" or "hostility" defined by POM2. It is also possible to adopt seven scales: confusion to perplexity, ``depression to depression,'' ``fatigue to lethargy,'' ``tension to anxiety,'' ``liveliness to vitality,'' and ``amity.''

The third training data DB 15 is a database that stores third training data used for machine learning of the crime risk estimation model 18. FIG. 9 is a diagram illustrating the third training data DB15. As shown in FIG. 9, the third training data stored in the third training data DB 15 has an association between "emotional pattern", which is an explanatory variable during machine learning, and "special fraud", which is a target variable during machine learning. This is the data obtained. The "emotional pattern" stored here indicates the pattern of changes in the user's emotions during a phone call, and indicates the pattern of changes in the user's emotions during a phone call, within a preset time after a predetermined operation such as a response operation of the call button on the telephone is detected. It is a pattern of changes in a person's emotions. "Special fraud" is information that specifies a special fraud associated with the emotion pattern, and is a criminal act that corresponds to the emotion pattern among a plurality of criminal acts. In the example of FIG. 9, it is shown that the special fraud when the emotion changes to emotion A, emotion B, and emotion A is "fraud AA."

Here, the "emotions" used in Example 1 will be explained from both sides: the user (victim) who receives a call from a special fraud group, and the special fraud group (perpetrator) who calls the user. First, let's explain emotions in detail. FIG. 10 is a diagram showing an example of the victim's emotions. Here, as examples, we will explain "friendship," "liveliness/vitality," and "confusion/bewilderment." As shown in FIG. 10, each emotion is identified by the user's state, the user's tone and atmosphere, and the content of the user's emotion.

For example, the state of emotion "friendship" is a state in which the user is attentive and interested in the other party of the conversation, such as introduction and trust. In this state of "friendship," the user's tone and atmosphere are "polite, cheerful, and sympathetic," and the user's emotional content is characterized by "having a sense of trust and being interested in" the other party. It is.

The emotional state of "confusion to bewilderment" is a state in which the user becomes agitated with the other party in the conversation, causing heart rate fluctuations. In this state of ``confusion/bewilderment,'' the user's tone and atmosphere become ``polite and flat,'' and the user's emotional content is characteristically ``promoting impatience and desire'' in the conversation partner.

The state of emotion "liveliness ~ vitality" is a state in which the user's arousal level increases, such as promoting action, in the person with whom the user is interacting. In this state of "vigor - vitality," the user's tone and atmosphere are "polite and flat," and the user's emotional content is characterized by "taking action."

Next, we will explain the perpetrator's guidance and the victim's emotions when the damage occurs. FIG. 11 is a diagram showing an example of emotions when a crime occurs. As shown in Figure 11, in order to make the victim feel the above-mentioned feelings of "friendship," the perpetrator uses a tone and atmosphere such as "polite, cheerful, and agreeable" to give the victim a sense of trust, interest, and interest. The purpose of the conversation is to make the children have the same information as possible, and the conversation includes keywords such as ``I'm at the government office, but I'll get my money back.Didn't my documents arrive?'' As a result, victims develop unconditional trust in public institutions, expect a refund, and develop feelings of ``friendship.''

Additionally, in order to make the victim feel the above-mentioned feelings of confusion and bewilderment, the perpetrator uses a tone and atmosphere such as ``polite and flat,'' with the purpose of ``prompting the victim to become impatient and stimulate desire.'' Have a conversation that includes keywords such as ``The refund deadline has passed, Special/Just for you, Passbook and ATM card, Refund via ATM.'' As a result, the victim becomes impatient, develops doubts, is persuaded, and begins to feel ``confusion'' and bewilderment.

In addition, in order to make the victim feel the above-mentioned feeling of ``vitality and vitality'', the perpetrator uses a ``polite and flat'' tone and atmosphere, and asks the victim to ``take the bankbook and other items and head to the ATM.'' The goal is to have a conversation that includes keywords such as "on the phone with the person in charge and follow instructions." As a result, the victim feels uneasy about operating the ATM, tries to follow instructions, and begins to feel ``vigorous.''

As mentioned above, the perpetrator uses skillful speech to induce the emotions of the victim in order to commit special fraud. In other words, the emotional changes of victims often vary depending on the type of fraud. Therefore, the machine learning of Crime Risk Estimation Model 18 uses training data that correlates "emotional patterns" and "special fraud" based on past history, criminal psychology analysis, and questionnaires of perpetrators and victims. .

Here, as an example, a combination of a special fraud and an emotional pattern will be described. FIG. 12 is a diagram showing the relationship between refund fraud and emotional patterns. As shown in Figure 12, in refund fraud, the perpetrator calls the user and starts telling them things that are interesting, such as ``You'll get your money back,'' and then saying things like ``The refund deadline has passed.'' They often tell stories that make them feel anxious, and then eventually encourage them to take action, such as saying, ``It's okay as long as you pay the fee.'' In this case, the user's emotions will be in a "friendly" state due to stories that interest the user, "confused to perplexed" due to stories that irritate the user, and then become "lively to vitality" due to stories that encourage action. In other words, in "refund fraud," a change from positive emotions to negative emotions occurs. As a result, the emotion patterns of ``friendship, confusion/embarrassment, liveliness/vitality'' are associated with ``refund fraud.''

FIG. 13 is a diagram illustrating the relationship between me-or-me fraud and emotional patterns. As shown in Figure 13, in the ``Ore-Ore'' scam, the first criminal pretending to be a relative of the user calls the user and starts making irritating stories such as ``I lost the company's money.'' On behalf of the second culprit, say things like, ``I'll pay for it, so it's okay,'' to instill a sense of trust. Afterwards, the first offender often makes another irritating statement, such as ``I want you to transfer money,'' and the second culprit often makes an encouraging message, such as ``I will support you over the phone.'' In this case, the user's emotions become "confused to bewildered" due to the story that irritates the user, become "friendly" state due to the story that creates a sense of trust, and then become "confused-embarrassed" again due to the story that irritates the user. , stories that encourage action lead to a state of ``liveliness~vitality''. In other words, in "Ore Ore Fraud", a change from negative emotions to positive emotions occurs. As a result, the emotional patterns of "confusion-embarrassment, friendship, confusion-embarrassment, liveliness-vitality" are associated with "I'm me fraud".

Returning to FIG. 6, the characteristic estimation model 16 is a machine learning model included in the crime detection model that estimates psychological characteristics. Specifically, the characteristic estimation model 16 is a first machine learning model that outputs psychological characteristics according to the input of physical information and physiological reaction information, and estimates corresponding to each index indicating psychological characteristics. This is a multi-value judgment model that outputs a value (probability). Note that various mathematical models such as a neural network can be adopted as the characteristic estimation model 16.

The emotion estimation model 17 is a machine learning model that is included in the crime detection model and estimates emotions. Specifically, the emotion estimation model 17 is a second machine learning model that outputs emotions according to the input of physical information, physiological reaction information, and psychological characteristics, and the emotion estimation model 17 is a second machine learning model that outputs emotions according to the input of physical information, physiological reaction information, and psychological characteristics. This is a multi-value decision model that outputs an estimated value (probability). Note that various mathematical models such as a neural network can be adopted as the emotion estimation model 17.

The crime risk estimation model 18 is a machine learning model that is included in the crime detection model and estimates the risk of occurrence of crimes such as special fraud. Specifically, the crime risk estimation model 18 is a machine learning model that outputs a crime risk according to the input of an emotional pattern that includes at least one emotion, and calculates the occurrence risk of each crime (out of a plurality of crimes). This is a multi-value decision model that estimates the probability. Note that various mathematical models such as a neural network can be adopted as the crime risk estimation model 18.

The control unit 20 is a processing unit that controls the entire information processing device 10, and is realized by, for example, a processor. The control section 20 includes a machine learning section 30, an operation section 40, and a notification section 50. Note that the machine learning unit 30, the operation unit 40, and the notification unit 50 are realized by an electronic circuit included in a processor, a process executed by the processor, or the like.

The machine learning unit 30 includes a data generation unit 31, a first training unit 32, a second training unit 33, and a third training unit 34, and performs processing to generate various machine learning models prior to real-time detection of special fraud. Department.

The data generation unit 31 is a processing unit that performs psychological experiments and generates training data for machine learning. FIG. 14 is a diagram illustrating generation of training data. As shown in FIG. 14, the data generation unit 31 acquires the psychological characteristics of the test subject and the physical information of the test subject through a questionnaire administered to the test subject before the experiment (S1). Next, the subject wears a wearable terminal that can measure physiological reactions (S2).

After that, the data generation unit 31 gives the subject an arbitrary stimulus such as listening to a fraudulent voice, and records the physiological reactions before and after the stimulus measured by the wearable terminal for a certain period of time (S3). Furthermore, the data generation unit 31 similarly records the emotions before and after the stimulation through a questionnaire given to the subject (S4).

By doing so, the data generation unit 31 collects data in which physical information, psychological characteristics, a certain emotion, and physiological reaction information during a certain emotion are associated for each subject. Then, the data generation unit 31 stores the training data in which "physical information, physiological reaction information, psychological characteristics" are associated in the first training data DB 13, and stores the training data in which "physical information, physiological reaction information, psychological characteristics, The training data associated with "emotion" is stored in the second training data DB 14.

The first training unit 32 is a processing unit that generates the characteristic estimation model 16 by machine learning using first training data stored in the first training data DB 13. Specifically, the first training unit 32 uses supervised learning to generate a characteristic estimation model 16 that estimates "psychological characteristics" according to the input of "physical information and physiological reaction information."

FIG. 15 is a diagram illustrating machine learning of the characteristic estimation model 16. As shown in FIG. 15, the first training unit 32 uses the explanatory variables "physical information (gender: 1, age: 70), physiological reaction information (heart rate: 70, breathing rate: 13)" and the objective variable "psychological information". The first training data having the following characteristics (suspicion: 50, anxiety characteristics: 20) is input into the characteristic estimation model 16, and the output result of the characteristic estimation model 16 is obtained. Then, the first training unit 32 adjusts the characteristic estimation model 16 so that the error between the output result of the characteristic estimation model 16 and the target variable "psychological characteristics (suspicion: 50, anxiety characteristics: 20)" is minimized. Update various parameters.

The second training unit 33 is a processing unit that generates the emotion estimation model 17 by machine learning using second training data stored in the second training data DB 14. Specifically, the second training unit 33 uses supervised learning to generate an emotion estimation model 17 that estimates an “emotion” according to the input of “physical information, physiological reaction information, and psychological characteristics.”

FIG. 16 is a diagram illustrating machine learning of the emotion estimation model 17. As shown in FIG. 16, the second training unit 33 uses explanatory variables such as physical information (gender: 1, age: 70), physiological reaction information (heart rate: 70, breathing rate: 13), psychological characteristics (suspicion The second training data having the target variable "emotion (emotion A)" and "heart: 50, anxiety characteristics: 20)" are input to the emotion estimation model 17, and the output result of the emotion estimation model 17 is obtained. Then, the second training unit 33 updates various parameters of the emotion estimation model 17 so that the error between the output result of the emotion estimation model 17 and the target variable "emotion (emotion A)" is minimized.

The third training unit 34 is a processing unit that generates the crime risk estimation model 18 by machine learning using the third training data stored in the third training data DB 15. Specifically, the third training unit 35 uses supervised learning to generate a crime risk estimation model 18 that estimates the "probability of occurrence of special fraud" according to the input of "emotional patterns" that are time-series changes in emotions. do.

FIG. 17 is a diagram illustrating machine learning of the crime risk estimation model 18. As shown in FIG. 17, the third training unit 34 uses the third training data having the explanatory variable "emotion pattern (emotion A→emotion B→emotion A)" and the target variable "crime AA" to the crime risk estimation model 18. and obtain the output results of the crime risk estimation model 18. Then, the third training unit 34 updates various parameters of the crime risk estimation model 18 so that the error between the output result of the crime risk estimation model 18 and the objective variable "crime AA" is minimized.

The operation unit 40 includes an acquisition unit 41, a characteristic estimation unit 42, an emotion estimation unit 43, and a risk estimation unit 44, and uses various machine learning models generated by the machine learning unit 30 to extract special information from the contents of the user's phone call. This is a processing unit that detects the risk of fraud. That is, the operation unit 40 estimates the risk of occurrence of known special fraud and unknown special fraud in real time.

The acquisition unit 41 is a processing unit that acquires sensing data of a user. Specifically, the acquisition unit 41 acquires various sensing data from a camera installed in the user's home 3, a millimeter wave radar installed in the user's home 3, and a wearable terminal worn by the user.

For example, when the acquisition unit 41 detects the start of a call by detecting the start of a call by detecting a ring tone in the case of a landline phone or an operation to start a call by wireless communication in the case of a mobile phone, the acquisition unit 41 detects the start of a call. Obtain the video data. The acquisition unit 41 then acquires the user's physical information from the acquired video data using known image analysis or a machine learning model that estimates physical information from image data.

Further, upon detecting the start of a call, the acquisition unit 41 acquires physiological reaction information measured by the wearable terminal of the user during the call at any time during the call. Then, the acquisition unit 41 associates the acquired physical information and physiological reaction information, stores them in the storage unit 12, and outputs them to the characteristic estimation unit 42.

The characteristic estimation unit 42 is a processing unit that inputs the physical information and physiological reaction information acquired by the acquisition unit 41 into the characteristic estimation model 16 and estimates the psychological characteristics of the user. FIG. 18 is a diagram illustrating estimation of psychological characteristics. As shown in FIG. 18, the characteristic estimation unit 42 learns the physical information of the user obtained by analyzing the video data of the user during the call and the physiological reaction information of the user during the call. input into the completed characteristic estimation model 16.

Then, the characteristic estimation unit 42 estimates psychological characteristics based on the output results of the learned characteristic estimation model 16. For example, the characteristic estimation unit 42 specifies the measure with the highest estimated probability as the estimation result of suspicion among the measures and estimated probabilities of suspicion included in the output result, and the characteristic estimation unit 42 specifies the measure with the highest estimated probability as the estimation result of suspicion, and and the estimated probability, the scale with the highest estimated probability is identified as the estimation result of the anxiety characteristic. Then, the characteristic estimation section 42 outputs the physical information and physiological reaction information acquired by the acquisition section 41 and the estimated psychological characteristics to the emotion estimation section 43.

The emotion estimation unit 43 uses the physical information and physiological reaction information acquired by the acquisition unit 41 and the psychological characteristics estimated by the characteristic estimation unit 42 by inputting them into the learned emotion estimation model 17. This is a processing unit that estimates people's emotions.

FIG. 19 is a diagram illustrating emotion estimation. As shown in FIG. 19, when a psychological characteristic is estimated using the characteristic estimation model 16, the emotion estimation unit 43 combines the estimated psychological characteristic with the physical information and physiological reaction information used for estimation. It is input into the emotion estimation model 17 to obtain an emotion estimation result. Then, the emotion estimating unit 43 identifies, among the emotion estimation results, an emotion with an estimated probability equal to or higher than a threshold value or an emotion with the highest estimated probability, as the estimation result. The emotion estimation unit 43 stores the estimation results in time series in the storage unit 12 or the like.

The risk estimation unit 44 is a processing unit that estimates the risk of occurrence of special fraud using the emotion estimated by the emotion estimation unit 43 and the crime risk estimation model 18. For example, the risk estimating unit 44 generates an emotional pattern by combining the estimated emotions in time series, inputs it into the learned crime risk estimation model 18, and obtains the estimation result of the risk of occurrence of special fraud.

For example, the risk estimating unit 44 determines that a special fraud call has been made to the person when the person's emotion estimated by the emotion estimating unit 43 satisfies the condition that the person's emotion changes from a negative state to a positive state within a predetermined period. Estimate that.

In addition, the risk estimation unit 44 inputs the estimated emotional pattern into the crime risk estimation model 18 to obtain the magnitude of the risk of occurrence of each of the plurality of criminal acts, and calculates the risk of occurrence of each of the plurality of criminal acts. Identifying a criminal act against a person from among multiple criminal acts based on magnitude. For example, the risk estimating unit 44 identifies, as the estimation result, a crime with an estimated probability equal to or higher than a threshold value or a crime with the highest estimated probability in the estimation result of the occurrence risk obtained from the crime risk estimation model 18. When a crime is detected, the risk estimation unit 44 outputs information regarding the estimated crime to the notification unit 50.

Here, the estimation of crime risk will be explained. FIG. 20 is a diagram illustrating estimation of crime risk. As shown in FIG. 20, the operation unit 40 acquires (1) physical information and (2) physiological reaction information as external information and internal information of the user during a telephone conversation, and (3) To obtain the psychological characteristics inherent to a person regardless of their emotions and psychological state. The operation unit 40 associates each piece of information (1), (2), and (3) acquired from time to time and inputs it into the emotion estimation model 17 to estimate the person's emotion from time to time.

Then, the operation unit 40 generates an emotion pattern using information regarding each emotion that is sequentially estimated from the start of the conversation. For example, the operation unit 40 generates emotion patterns of "liveliness to vitality", "confusion to bewilderment", and "liveliness to vitality" in the chronological order of conversation time, inputs them into the crime risk estimation model 18, and inputs them into the crime risk estimation model 18, Obtain an estimation result such as "Special fraud estimation degree: 〇%".

The emotion pattern is generated by using emotion information estimated within a predetermined period of time, such as every 5 minutes, for example. FIG. 21 is a diagram illustrating an example of identifying emotional patterns. As shown in FIG. 21, when the telephone call starts at time T0, the operation unit 40 performs emotion inference at any time to estimate emotion A at T1, emotion B at T2, and emotion at T3. A is estimated, emotion C is estimated at T4, and emotion B is estimated at T5.

In this case, the operation unit 40 includes emotion A, emotion B, and emotion A using emotion B and emotion C estimated within a specified time starting from T1 at which the emotion (emotion A) was first estimated. Emotion pattern 1 is generated and crime risk estimation is performed. For example, the operation unit 40 inputs the emotion pattern "emotion A, emotion B, emotion A" into the crime risk estimation model 18 and estimates "no crime risk."

Next, the operation unit 40 uses the emotion A and emotion C estimated within the specified time starting from T2, where the emotion (emotion B) was estimated next to T1, to calculate emotion B, emotion A, and emotion C. The emotion pattern 2 including the above is generated and the crime risk is estimated. For example, the operation unit 40 inputs the emotion pattern "emotion B, emotion A, emotion C" into the crime risk estimation model 18 and estimates "no crime risk."

Next, the operation unit 40 uses the emotion C and emotion B estimated within the specified time starting from T3, where the emotion (emotion A) was estimated after T2, to calculate emotion A, emotion C, and emotion B. The emotion pattern 3 including the above is generated and the crime risk is estimated. For example, the operation unit 40 inputs the emotion pattern "Emotion A, Emotion C, Emotion B" into the crime risk estimation model 18 and estimates "There is a crime risk."

In this manner, the operation unit 40 repeatedly estimates the crime risk by generating an emotion pattern using emotions within a predetermined interval while the telephone conversation continues. Note that the emotion pattern generation interval shown in FIG. 21 is an example, and the settings can be changed as desired. For example, an emotional pattern can be generated using T, T2, and T3, and then an emotional pattern can be generated using T4, T5, and T6.

Returning to FIG. 6, the notification unit 50 is a processing unit that notifies related parties when a crime risk is estimated by the operation unit 40. For example, when a "risk of crime occurrence" is estimated with an estimated probability equal to or higher than a threshold value, the notification unit 50 sends a message to the speaker installed in the user's home 3 saying, "The person on the other end of the call may be a fraud group. ” or vibrate the user's mobile phone or wearable device to alert them. The notification unit 50 can also send an emergency message such as "Mr. ___ is on the phone and there is a possibility of special fraud" to a pre-registered emergency contact. The notification unit 50 can also notify the police or local government.

(Flow of machine learning processing)
FIG. 22 is a flowchart illustrating the flow of machine learning processing. As shown in FIG. 22, when the machine learning unit 30 is instructed to start processing (S101: Yes), the machine learning unit 30 conducts a questionnaire on the psychological characteristics of the subject to acquire the psychological characteristics (S102).

Then, the machine learning unit 30 listens to the audio of the specialized fraud to collect physiological response information (S103), and after the audio ends, the machine learning unit 30 administers a questionnaire to the subject about their emotional state to obtain the emotions they felt when experiencing the specialized fraud (S104). If the experiment is to continue (S105: No), the machine learning unit 30 repeats S102 and subsequent steps.

On the other hand, when the machine learning unit 30 finishes the experiment (S105: Yes), the machine learning unit 30 generates first training data for the characteristic estimation model 16 using the experiment results (S106), and generates second training data for the emotion estimation model 17. Data is generated (S107).

Thereafter, the machine learning unit 30 generates the characteristic estimation model 16 using the generated first training data (S108), generates the emotion estimation model 17 using the generated second training data (S109), and prepares it in advance. The crime risk estimation model 18 is generated using the third training data obtained (S110).

(Flow of estimation process)
FIG. 23 is a flowchart illustrating the flow of crime risk estimation processing. As shown in FIG. 23, when the operation unit 40 is instructed to start processing (S201: Yes), it acquires video data (S202), analyzes the video data, and analyzes the physical characteristics of the person in the video. Acquire features (S203).

Subsequently, the operation unit 40 acquires physiological reaction information from the wearable terminal of the person depicted in the video data (S204). Then, the operation unit 40 inputs the physical information and physiological reaction information into the characteristic estimation model 16 to estimate the psychological characteristics of the person (S205). Further, the operation unit 40 inputs the physical information, physiological reaction information, and psychological characteristics into the emotion estimation model 17 to estimate the person's emotion (S206). Here, the operation unit 40 repeats S202 and subsequent steps before a predetermined time has elapsed since the start of the call (S207: No).

On the other hand, when a predetermined period of time has passed since the start of the call (S207: Yes), the operation unit 40 generates an emotion pattern using the estimated emotion (S208). Subsequently, the operation unit 40 inputs the emotional pattern into the crime risk estimation model 18 to estimate the crime risk (S209).

Then, when a crime risk is detected by the operation unit 40 (S210: Yes), the notification unit 50 notifies related parties of the crime risk (S211). Thereafter, the operation unit 40 executes S202 and subsequent steps when continuing the process (S212: No), and ends the crime risk estimation when ending the process (S212: Yes). Note that if the crime risk to be notified is not detected in S210 (S210: No), S212 is executed without executing S211.

(effect)
As described above, the information processing device 10 estimates psychological characteristics based on physical characteristics and physiological reactions estimated from video data and the like. In emotion estimation, the information processing device 10 uses physiological reaction information and personal characteristic information (physical characteristic information and psychological characteristic information) as feature quantities. The information processing device 10 can minimize the influence on estimation accuracy caused by individual differences by using the emotion estimation model 17 that takes psychological characteristics into consideration, and can improve estimation accuracy.

In addition, among personal characteristic information, psychological characteristic information requires the implementation of psychological questionnaires, which is difficult and expensive to obtain, and conventional methods cannot estimate emotions using both personal characteristic information and physiological reaction information. Not realized.

On the other hand, by adding psychological characteristics, the information processing device 10 can improve the accuracy of emotion estimation based on physiological reactions and realize accurate fraud detection focusing on changes in the victim's emotions.

Since the information processing device 10 can infer a crime based on a person's emotional pattern, it is possible to prevent criminal damage caused by known special fraud and unknown special fraud. In addition, the information processing device 10 can generate a machine learning model that performs accurate inference using accurate training data, and achieves faster estimation processing than when unnecessary training data is used. be able to.

The information processing device 10 estimates the risk of a criminal act against a person using a crime risk estimation model 18 trained on the relationship between emotional patterns and crime. Therefore, even if a new crime occurs, the information processing device 10 can estimate the risk of the crime occurring from the emotional pattern of the person related to the crime, so that even unknown special fraud crimes can be effectively processed. can be prevented.

The information processing device 10 generates emotional patterns obtained from past performance and criminal psychology for each special crime, and trains the crime risk estimation model 18. Therefore, the information processing device 10 can train on past performance and also on emotional patterns at the time of an unknown crime that can be inferred from past performance.

Now, the embodiments of the present invention have been described so far, but the present invention may be implemented in various different forms in addition to the embodiments described above.

(Numbers, etc.)
The data set examples, training data examples, sensor examples, number of data, time span, examples and numbers of emotions, number and types of special frauds, emotional patterns, combinations of special frauds and emotional patterns, etc. used in the above examples are as follows: This is just an example and can be changed as desired. Furthermore, the flow of processing described in each flowchart can be changed as appropriate within a consistent range.

(Model form)
In the above embodiment, an example in which a multi-value judgment model (multi-value classification model) is used as each machine learning model has been described, but the present invention is not limited to this. For example, the characteristic estimation model 16 can also use a binary classification model that determines whether physiological reactions and psychological states are linked or not.

(Estimation of emotions)
In the above embodiment, an example was explained in which a person's emotion is estimated using the emotion estimation model 17, but the present invention is not limited to this. For example, the information processing device 10 can estimate emotions using psychological characteristics using rules that associate psychological characteristics with emotions, rules that associate changes in psychological characteristics with emotions, etc. I can do it.

(Example of using emotions)
In the above embodiment, an example was described in which highly accurately estimated emotions are used to detect special fraud, but the present invention is not limited to this. For example, the estimated emotions can be used to detect suspicious persons in stores, employee stress in the workplace, etc.

(Example of presumed special fraud)
In the above embodiment, an example has been described in which the crime risk of special fraud is estimated from the emotional pattern using the crime risk estimation model 18, but the present invention is not limited to this. For example, the information processing device 10 can hold a determination rule that associates emotional patterns with special fraud, and use the determination rule to estimate special fraud that corresponds to the identified emotional pattern.

(Example of electronic equipment)
In the above embodiment, the case where the user's home 3 is on the phone is explained as an example, but the present invention is not limited to this. For example, the information processing device 10 can provide similar services to users who use ATMs (Automatic Teller Machines) set up in stores such as banks and convenience stores. In this case, the information processing device 10 estimates the crime risk by acquiring sensing data from cameras and microphones installed around the ATM, and when a crime risk is detected, makes an announcement in the store or notifies the person in charge. I do things.

(system)
Information including processing procedures, control procedures, specific names, and various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

Furthermore, each component of each device shown in the drawings is functionally conceptual, and does not necessarily need to be physically configured as shown in the drawings. That is, the specific form of distributing and integrating each device is not limited to what is shown in the drawings. In other words, all or part of them can be functionally or physically distributed and integrated into arbitrary units depending on various loads and usage conditions. For example, the machine learning section 30 and the operation section 40 can be implemented using separate computers (housing). That is, it can also be realized by an information processing device that performs the same function as the machine learning section 30 and an information processing device that performs the same function as the operation section 40.

Furthermore, all or any part of each processing function performed by each device may be realized by a CPU and a program that is analyzed and executed by the CPU, or may be realized as hardware using wired logic.

(hardware)
FIG. 24 is a diagram illustrating an example of a hardware configuration. As shown in FIG. 24, the information processing device 10 includes a communication device 10a, an HDD (Hard Disk Drive) 10b, a memory 10c, and a processor 10d. Further, each part shown in FIG. 24 is interconnected by a bus or the like.

The communication device 10a is a network interface card or the like, and communicates with other devices. The HDD 10b stores programs and DB that operate the functions shown in FIG.

The processor 10d reads a program that executes the same processing as each processing unit shown in FIG. 6 from the HDD 10b, etc., and deploys it in the memory 10c, thereby operating a process that executes each function described in FIG. 6, etc. For example, this process executes the same functions as each processing unit included in the information processing device 10. Specifically, the processor 10d reads a program having the same functions as the machine learning section 30, the operation section 40, the notification section 50, etc. from the HDD 10b. The processor 10d then executes a process that executes the same processing as the machine learning unit 30, the operation unit 40, the notification unit 50, and the like.

In this way, the information processing device 10 operates as an information processing device that executes an information processing method by reading and executing a program. Further, the information processing device 10 can also realize the same functions as in the above-described embodiments by reading the program from the recording medium using the medium reading device and executing the read program. Note that the programs in other embodiments are not limited to being executed by the information processing device 10. For example, the above embodiments may be applied in the same way when another computer or server executes the program, or when these computers or servers cooperate to execute the program.

This program may be distributed via a network such as the Internet. In addition, this program is recorded on a computer-readable recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO (Magneto-Optical disk), or DVD (Digital Versatile Disc), and is read from the recording medium by the computer. It may also be executed by being read.

10 Information processing device 11 Communication unit 12 Storage unit 13 First training data DB
14 Second training data DB
15 Third training data DB
16 Characteristic estimation model 17 Emotion estimation model 18 Crime risk estimation model 20 Control section 30 Machine learning section 31 Data generation section 32 First training section 33 Second training section 34 Third training section 40 Operation section 41 Acquisition section 42 Characteristic estimation section 43 Emotion estimation section 44 Risk estimation section 50 Notification section

Claims (6)

to the computer,
Obtain physiological reaction information of a person identified from the person's vital data,
Obtaining physical information of the person identified from video data of the person;
Generating psychological characteristics indicating the unique personality of the person by inputting the acquired physiological reaction information and the physical information into a first machine learning model,
estimating the emotion of the person based on the generated psychological characteristics;
An emotion estimation program characterized by executing processing. The estimating process is
estimating the emotion of the person by inputting the psychological characteristics, the physiological reaction information, and the physical information into a second machine learning model;
The emotion estimation program according to claim 1. 3. The emotion estimation program according to claim 1, wherein the psychological characteristic is a characteristic unique to the person regardless of emotion and psychological state. The acquisition process is as follows:
Generating physiological reaction information of the person using the electronic device based on the vital data of the person,
The acquisition process is as follows:
obtaining physical information of the person identified from video data of the person using the electronic device;
The process to generate the
Generating the psychological characteristics by inputting the physiological reaction information and the physical information of the person into the first machine learning model,
The estimating process is
Estimating whether or not a special fraud has occurred in the person using the electronic device, based on a pattern of emotional changes of the person estimated based on the psychological characteristics;
The emotion estimation program according to claim 1. The computer is
Obtain physiological reaction information of a person identified from the person's vital data,
Obtaining physical information of the person identified from video data of the person;
Generating psychological characteristics indicating the unique personality of the person by inputting the acquired physiological reaction information and the physical information into a first machine learning model,
estimating the emotion of the person based on the generated psychological characteristics;
An emotion estimation method characterized by performing processing. Obtain physiological reaction information of a person identified from the person's vital data,
Obtaining physical information of the person identified from video data of the person;
Generating psychological characteristics indicating the unique personality of the person by inputting the acquired physiological reaction information and the physical information into a first machine learning model,
estimating the emotion of the person based on the generated psychological characteristics;
An information processing device comprising a control section.

Images