CN104704797B - Virtual protocol communication for electronic equipment - Google Patents

Abstract

Method and apparatus for being communicated between virtual protocol, wherein virtual protocol are associated with the user of the electronic equipment through at least one network connection.First user can indicate associated first virtual protocol calling and the communication session with associated second virtual protocol of second user.In order to call the communication session, the first virtual protocol can send outflow communication to the second virtual protocol and outflow communication can indicate that the second virtual protocol executes at least one action for the first user.Virtual protocol associated with different user can alternatively communicate with one another in the case where lacking user's interaction to execute cooperative action.

Description

Virtual agent communication for electronic devices

Background technique

In recent years, electronic devices such as smartphones and tablet computers have often included applications known as virtual agents or assistants that help users perform functions such as searching for content on a network (eg, the Internet) and interacting with other applications. Users can interact with the virtual agent using typing or touch to perform these functions. Additionally, some devices include voice interfaces to enable users to interact with virtual agents using voice input. These virtual agents are generally platform-specific and work with devices using a particular operating system. For example, available from Apple Computer Inc. The Assistant is configured to work with devices running the iOS mobile operating system, and the EVA Assistant, available from Bulletproof, is configured to work with devices running the Android mobile operating system, available from Google.

Contents of the invention

Some embodiments are directed to improving the functionality of virtual agents by facilitating communication between virtual agents associated with different users in a networked system. Improved functionality may include, but is not limited to, functionality such as finding out another user's location, coordinating meeting locations, accessing another user's calendar, scheduling meetings, and sharing contact information.

Some embodiments are directed to a platform-independent virtual agent operable on at least two devices running different operating systems, thereby allowing communication between the at least two devices to perform at least one coordinated action.

Some embodiments are directed to a method of communicating between virtual agents, each virtual agent associated with a user, the method comprising: receiving input from a first user; determining, by the first virtual agent associated with the first user, whether to invoke and communicate with the first user a communication session with a second virtual agent associated with the two users; and sending an outgoing communication to the second virtual agent in response to determining to invoke the communication session.

Other embodiments are directed to a computer-readable medium encoding a plurality of instructions that, when executed by at least one computer, perform a method of communicating between virtual agents, each virtual agent associated with a user, the method comprising: from a first A user receives an input; determines, by a first virtual agent associated with the first user, whether to invoke a communication session with a second virtual agent associated with a second user; and sends an outgoing message to the second virtual agent in response to determining to invoke the communication session. Communications, wherein the outgoing communication includes a request for the second virtual agent to perform at least one action on behalf of the first user.

Other embodiments are directed to an electronic device comprising: a communication interface configured to receive input from a first user; and a first virtual agent configured to: determine whether to invoke a communication session with a second virtual agent associated with a second user; and respond An outgoing communication is sent to the second virtual agent upon determining to invoke the communication session, wherein the outgoing communication includes a request for the second virtual agent to perform at least one action for the first user.

Other embodiments are directed to methods of managing a task by a first virtual agent associated with a first user without receiving input from the first user, the method comprising: determining, by the first virtual agent, whether a condition associated with the task has occurred ; and sending an alert to the first user, or sending an outgoing communication to a second virtual agent associated with the second user, in response to determining that the condition has occurred.

Other embodiments are directed to a computer readable medium encoding a plurality of instructions that, when executed by at least one computer, perform tasks managed by a first virtual agent associated with a first user without receiving input from the first user A method comprising: determining, by a first virtual agent, whether a condition associated with the task has occurred; and sending a reminder to the first user in response to determining that the condition has occurred, or sending a reminder to a second virtual agent associated with the second user. outgoing communication.

Other embodiments are directed to an electronic device comprising: a first virtual agent associated with a first user, wherein the first virtual agent is configured to: determine whether a condition associated with a task assigned to the first virtual agent occurs; and respond to determining the condition An alert is sent to the first user as has occurred, or an outgoing communication is sent to a second virtual agent associated with the second user.

Other embodiments are directed to methods of facilitating communication between virtual agents, each virtual agent associated with a user, the method comprising: receiving, by a second virtual agent associated with a second user, a communication from a first virtual agent associated with a first user ; and performing at least one action in response to receiving the communication.

Other embodiments are directed to a computer readable medium encoding a plurality of instructions that, when executed by at least one computer, perform a method of facilitating communication between virtual agents, each virtual agent associated with a user, the method comprising: A second virtual agent associated with the second user receives a communication from a first virtual agent associated with the first user; and performs at least one action in response to receiving the communication.

Other embodiments are directed to an electronic device comprising: a communication interface configured to receive communications from a first virtual agent associated with a first user; and a second virtual agent associated with a second user, wherein the second virtual agent is configured to: respond to At least one action is performed upon receipt of the communication.

It should be appreciated that all combinations of the above concepts and additional concepts discussed in more detail below (provided such concepts are not mutually inconsistent) are contemplated as part of the inventive subject matter disclosed herein.

Description of drawings

The figures are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures is represented by a like numeral. For purposes of clarity, not every component may be labeled in every figure. In the attached picture:

1 is an exemplary system including components for facilitating communication between virtual agents for electronic devices, according to some embodiments of the invention;

2 shows an illustrative process for generating outgoing communications from a virtual agent in response to user input, according to some embodiments of the invention;

3 shows an illustrative process for providing agent-to-agent communication in the background without user interaction, according to some embodiments of the invention;

Figure 4 shows an illustrative process for managing incoming communications by a virtual agent, according to some embodiments of the invention;

5 shows an illustrative process for performing at least one action in response to receiving an incoming communication from a virtual agent, according to some embodiments of the invention; and

Figure 6 is an illustrative computer system that may be used in conjunction with some embodiments of the invention.

Detailed ways

The inventors have recognized and appreciated that while conventional virtual agents are useful for assisting users in various tasks such as initiating calls or searching for content on the World Wide Web, the functionality of conventional virtual agents can be achieved by having a virtual agent associated with a user and associated It can be improved by interacting with virtual agents associated with other users to perform collaborative actions, such as finding out another user's location, scheduling a meeting with another user, and sharing contact information or other data between users. To this end, some embodiments of the present invention serve to improve the functionality of a virtual agent by enabling agent-to-agent communication to perform one or more actions.

An exemplary system 100 used in accordance with some embodiments of the invention is illustrated in FIG. 1 . System 100 includes electronic device 102, which may be implemented as, but not limited to, a cell phone, smartphone, tablet computer, and laptop computer. System 100 may include any number of electronic devices. System 100 also includes cell phone 130 and tablet computer 140 , for example. While various components of electronic device 102 are described below, it should be understood that other devices in system 100 including cell phone 130 and/or tablet computer 140 may include at least some similar components to those described in connection with electronic device 102 .

Electronic device 102 may be configured to receive input from a user to perform one or more functions. To receive user input, the electronic device 102 may include an input interface 104 including, but not limited to, a keyboard interface (eg, a QWERTY keyboard), a touch screen interface, and a voice interface such as a microphone. In embodiments including a voice interface, received voice input may be sent to an automatic speech recognition (ASR) engine (not shown) to recognize the content of the received voice input. The ASR engine may be included as part of the electronic device 102 and/or the ASR engine may be located remotely from the electronic device 102 and may communicate with the electronic device 102 using any suitable communication medium (eg, one or more networks). In some embodiments, the electronic device 102 may be associated with one or more local ASR engines and one or more remotely located ASR engines, whereby at least some of the received speech input is processed by the one or more local ASR engines, And at least some of the received speech input is processed by the one or more remotely located ASR engines. Examples of such distributed ASR systems are known to those skilled in the art. Voice input for recognition sent to the one or more remote ASR engines using one or more communication media (e.g., one or more networks) connecting the electronic device 102 and the ASR engine may utilize the same or different communication media Returns to the electronic device 102 from the remotely located ASR engine.

In some embodiments, the ASR engine may be configured to recognize speech input using one or more fixed or dynamically generated grammars and/or vocabularies. Alternatively, some embodiments may allow free-form input to enable the user to speak to the virtual agent in a more natural manner and the ASR engine may utilize, for example, a Natural Language Understanding (NLU) model to recognize whatever the user is saying. In embodiments that include NLU model-based speech recognition, the virtual agent may attempt to infer the action the user intends to perform based on the NLU results. It should be appreciated that the particular manner in which voice input recognition is implemented does not limit embodiments of the invention, as any suitable voice recognition implementation may be used.

The electronic device 102 also includes a user interface 106 to enable a user to interact with the device to perform various actions. For example, in some embodiments, user interface 106 may display a plurality of icons associated with a plurality of applications and a user may interact with one or more of these icons to launch the associated application. Any suitable user interface may be used, as embodiments of the invention are not limited in this respect.

Electronic device 102 may include one or more applications 108 configured to execute on the device. Some applications may be native applications that execute primarily on the electronic device 102 . For example, a scheduling application (eg, a calendar application) may execute on an electronic device to enable a user to store scheduling information regarding events and/or appointments. Data associated with such applications may be stored locally on the electronic device or at least a portion of the data may be stored remotely, as discussed in more detail below. Other applications (eg, web-based applications) may be configured to interact with one or more computers via one or more networks. For example, a web-based email application may be hosted by one or more web servers, so that when a user wants to check email using the email application, a request will be sent from the electronic device to one or more remote servers hosting the email application. The located server.

In some embodiments, electronic device 102 may include a location sensor 110 configured to determine the location of the electronic device. For example, the location sensor may be a Global Positioning System (GPS) sensor configured to determine location coordinates of the device. Some embodiments may include an application that interacts with the location sensor 110 to display the location of the electronic device 102 on a map. Position sensor 110 may be implemented in any suitable manner, and embodiments of the invention are not limited in this regard.

As discussed briefly above, electronic device 102 may store user data 112 that describes information specific to a particular user or electronic device. For example, user data 112 may include, but is not limited to, contact information, calendar information, user preference information, behavior history information, and location information. In some embodiments, user data 112 may also include media content information, such as music or picture data associated with a particular user account or device. In some embodiments, at least some user data 112 may be stored remotely from electronic device 102 at a location accessible to electronic device 102 via at least one network. For example, at least some user data 112 may be stored in network storage 150 accessible by electronic device 102 via network 160 . Data stored remotely from an electronic device is often referred to as data stored "in the cloud," and such data is often accessible by the same user from multiple devices. The particular manner in which user data 112 is stored is not a limiting aspect of embodiments of the present invention, as user data 112 may be stored in any suitable location accessible by electronic device 102 .

Electronic device 102 also includes communication interface 120 configured to send and receive communications to and from other electronic devices (eg, cell phone 130 and tablet computer 140 ) and network storage 150 via network 160 . Communications may be sent over network 160 using any suitable protocol and any suitable message format, as embodiments of the invention are not limited by the particular network implementation details employed.

In some embodiments, the electronic device 102 also includes a virtual agent 114 . Users can interact with the virtual agent 114 to perform functions such as controlling electronic devices (eg, sending messages, making calls, opening applications), or accessing content and applications (eg, web searching, posting information on websites). In some embodiments, virtual agent 114 may be implemented as an application that communicates with other applications 108 and user data 112 in order to perform the functions described above, as well as other functions as described in more detail below. Not all devices in system 100 may be associated with a virtual agent and embodiments of the invention are not limited in this regard. As discussed in more detail below, electronic devices not associated with a virtual agent may receive communications in conventional formats (e.g., text messages, emails, phone calls, etc.), while electronic devices associated with a virtual agent may be able to utilize Increased communication functionality and enhanced user experience of some embodiments of the invention.

A virtual agent can be associated with a particular user and/or device in any suitable manner. In some embodiments, the virtual agent may be an application executing on the user's electronic device, as described above. In such embodiments, a virtual agent is associated with a user of the device on which the virtual agent is installed. In other embodiments, the virtual agent may not be installed on the user's electronic device, but may execute primarily on a computer at a remote location accessible to the user's electronic device through one or more networks. In embodiments where the virtual agent is located remotely from the user's electronic device, the virtual agent may include at least one identifier identifying one or more electronic devices associated with the user and/or at least one identifier identifying the user. Based on the one or more identifiers, the virtual agent may be considered "associated with" the user, as that term is used herein.

In some embodiments, virtual agent 114 may be configured to perform one or more actions based at least in part on user data, such as the user's contact information or scheduling information. For example, instead of sending or receiving communications to other devices itself (e.g., like email or text messaging applications), the virtual agent can respond to interpreting user input or for the user when a condition occurs that is associated with the task that the virtual agent is configured to perform. Instructs other applications to send one or more communications to another device.

In some embodiments, to facilitate interaction with user data, virtual agent 114 may be configured to aggregate user data for a particular user associated with the virtual agent. For example, a user's scheduling information may exist in multiple sources, including web-based calendars, native application calendars, and task lists. Virtual agent 114 may be configured to aggregate scheduling information across these multiple sources (and possibly other sources) and the aggregated user data may be stored for query by the virtual agent. By aggregating user data from multiple sources, a virtual agent may be able to quickly determine a more comprehensive view of a user's user data without having to individually query different sources. However, it should be understood that not all embodiments require aggregation of user data. More specifically, in some embodiments, a virtual agent may query multiple sources during a particular task assigned to a virtual agent (eg, managing scheduling information). In still other embodiments, only some user data may be aggregated while other user data is not aggregated. Furthermore, other user data than schedule data may alternatively be aggregated, and schedule data is provided only as an example of the types of data that may be aggregated.

User data may be aggregated and stored in any suitable manner. For example, at least some of the aggregated user data may be stored locally on the user's electronic device and/or the aggregated user data may be stored at least in part on one or more networks accessible by the user's virtual agent via one or more networks on the connected data store. In some embodiments, user data storage "in the cloud" is preferred because the user data can then be accessed by multiple electronic devices used by the user and/or by another user's virtual proxy, assuming that other user's virtual Proxy is granted access to said user data. Data aggregated by a virtual agent for a particular user may be updated periodically and/or user data may be updated in response to user requests. The particular manner in which user data is updated is not a limitation of embodiments of the invention.

In some embodiments, a virtual agent associated with one user may invoke a communication session with a virtual agent associated with another user to perform one or more actions. A communication session between virtual agents may be invoked in response to user input to perform one or more actions in the foreground, and/or a communication session between virtual agents may be invoked automatically without user input so as to be performed in the background without User input to perform one or more actions. In this disclosure, a user associated with a virtual agent invoking a communication session is referred to as a "sender," and a user associated with a virtual agent receiving a communication from the sender's virtual agent is referred to as a "receiver." Additionally, the sender's electronic device is often referred to herein as a "sending device," and the recipient's electronic device is often referred to herein as a "receiving device." However, it should be recognized that any electronic device can send and receive communications and this term does not limit the functionality of any particular device.

An exemplary process for invoking an agent communication session in response to user input is illustrated in FIG. 2 . In act 210, user input is received from a user of the electronic device. As discussed above, in some embodiments that include a voice interface, user input may include voice input that includes a request for the virtual agent to perform one or more actions, where the actions include at least one action for another user. The participation of one other virtual agent. For example, the user (ie, the sender) may ask the virtual agent "Where is John?". In response to this question, the virtual agent can determine that the sender wants information about the location of another user named John (i.e., the recipient), and the virtual agent can communicate with user data (e.g., stored contacts, schedules, etc.) information, etc.) to try to determine the contact information of the intended recipient user John.

In response to receiving user input, the process proceeds to act 210, where it is determined whether a communication session with another virtual agent should be invoked. Such a determination may be based at least in part on received user input and may be made in any suitable manner. In some embodiments, the determination of whether to communicate with another virtual agent is made by evaluating user input to identify one or more trigger words for invoking a communication session with a virtual agent associated with another user . For example, trigger words such as "where," "at home," and "on the way" used in conjunction with a person's name may indicate a user's intent to invoke a communication session related to location information. In response to receiving user input corresponding to the example discussed above (i.e., "Where is John"), the virtual agent for the sender may recognize the trigger word "Where" followed by the person name "John" and may determine to send Fang wants to know John's location information. The virtual agent can also incorporate other contextual data, such as time, location, user history, or some combination, to determine intent. If there are multiple listings for "John" in the contacts, the agent can see which "John" the user communicates with most often or usually from that location or time. In another example, the sender can say "tell John I'm on my way and I should be there in twenty minutes." Responsive to receiving this input, the virtual agent can identify a trigger used in conjunction with a user named John The word "on the way" and may determine that the sender wants to inform John of the user's current location and estimated time of arrival.

In some embodiments, the information may be sent to the user's (ie, recipient's) electronic device by sending a message (eg, a text message) including this information to the recipient's electronic device. For example, in the example above, if John is not associated with a virtual agent (e.g., because his electronic device does not include a virtual agent application), John's electronic device may receive from the sender a message that includes "I am on the road and I am on the 2nd." Should be there in ten minutes." text message. However, if John's electronic device is associated with a virtual agent, the incoming communication may be interpreted by John's virtual agent to provide a richer user experience, as discussed in more detail below. Because the sender may not know in advance whether the recipient has a compatible virtual agent executing on the recipient's device, the sender's device can provide outgoing communications to the recipient's device to determine whether the second agent is running on the recipient's device before sending the message. exists on the receiver's device. Based on the recipient device's response, the first agent can interpret the appropriate type of message to send. If the recipient is not associated with a virtual agent, at least some of the content of the received communication from the sender may still be displayed on the recipient's device (eg, as a text message). Thus, some embodiments of the invention allow communication between electronic devices with or without virtual proxies associated with users of such devices (with or without virtual proxies), users not associated with virtual proxies receiving conventional type messages (e.g., text messages, instant messages, email messages, web-based applications, etc.), users associated with virtual agents receive content in a manner that can be interpreted by their virtual agents to provide a rich user experience, Including providing responsive communication with sender virtual agents.

While the above examples describe communication regarding location information, other types of information may also be requested and/or shared using agent-to-agent communication in accordance with embodiments of the present invention. For example, other types of information that may be requested or transmitted include, but are not limited to, scheduling information, user content information (e.g., contact information or other user data), and any combination of location information, scheduling information, and user content information . In some embodiments, each type of information may be associated with a trigger word identified by the virtual agent to inform the virtual agent of the user's intent. For example, trigger words such as "Schedule," "When," and "Available" when used in conjunction with the recipient user's name may indicate that the action to be performed is with respect to scheduling information, such as "Share" or "Contact" The trigger word for may indicate that the action to be performed is about requesting or providing user content information. It should be appreciated that these trigger words are provided for example only and that any suitable word, phrase, or contextual data (e.g., location, time, behavior history, etc.) party's intentions.

Returning to the process of FIG. 2, if in act 220 it is determined that the user does not intend to invoke a communication session with another virtual agent, then the process proceeds to act 222 where an action is performed based on user input. For example, if the user input is "Where is the movie theater?", the user's virtual agent may determine that although the input includes the trigger word "Where is," the word "Movie theater" does not refer to another user. Thus, the virtual agent may determine that a communication session with another virtual agent is not an appropriate action to perform, and instead the virtual agent may perform another action such as a web search for movie theaters near the user's location, or some other suitable action. action.

If in act 220 it is determined that a communication session with another user's virtual agent should be invoked, then the process proceeds to act 230 where the content of the outgoing communication to the recipient's virtual agent is determined. A virtual agent may determine the content of outgoing communications in any suitable manner. In one example, if the user input is "Where is John?", the virtual agent can determine that the outgoing communication should include a request for recipient John's location, and this request can be sent to a user using the virtual agent associated with John. Electronic devices, as discussed in more detail below.

In some cases, the content of outgoing communications may include requests containing more than one type of information. For example, if the user input is "When is John expected to arrive at the race?", the sender's virtual agent can understand that in order to determine John's expected arrival time for the race, the sender's virtual agent will need to know John's location, "race" and knowing how long John has to travel between his current position and the race position based on, for example, John's current speed. In this example, the sender's virtual agent may attempt to determine the sender's intent by identifying what the user is likely to mean by one or more words in the user input. For example, the virtual agent may attempt to identify John, the intended recipient, by interacting with the sender's user data (eg, the sender's contact information). If the virtual agent cannot identify the intended recipient, the virtual agent can prompt the user to provide more information. The virtual agent may also attempt to discover the intended meaning of the "match" based at least in part on the sender's user data. For example, the virtual agent may interact with the sender's scheduling information to determine whether there is a scheduled event corresponding to a type of game (eg, baseball game, football game, etc.). Based on this investigation, the virtual agent can determine the identity of the "match" and the corresponding location of the game may also be inferred by the virtual agent based on information in the scheduled event.

In some embodiments, the sender's virtual agent may not be able to determine the meaning of certain words in the user input, but the recipient's virtual agent may be able to. In this case, the outgoing communication may include an indication that the sender's virtual agent cannot interpret certain words and a request to the recipient's virtual agent to see if the recipient's virtual agent can interpret those words. In some embodiments, all user input may be sent to the recipient virtual agent and the recipient's virtual agent may provide confirmation of the sender's virtual agent's interpretation (eg, based on the recipient's user data). In other embodiments, only a portion of the user input may be sent to the recipient's virtual agent and embodiments of the invention are not limited by the amount of data transferred between virtual agents.

If contact information for the intended recipient is identified based on the sender's user data, the virtual agent can generate the outgoing communication based at least in part on the identified contact information. In some embodiments, if the virtual agent does not uniquely identify the intended recipient from the sender's user data, the virtual agent may prompt the user to provide more information to resolve the ambiguity.

After determining suitable content to include in the outgoing communication, the process of FIG. 2 proceeds to act 240, where the outgoing communication is sent to the receiving user's electronic device. In some embodiments, user data may include preference information for sending communications to one or more contacts in the user's contact list and outgoing communications may be sent to the recipient's user using the preferred communication method for the particular user equipment. In the above example, the sender's user data may indicate that the sender generally prefers to send communications to the intended recipient "John" using text messages. Thus, in response to determining that the intended recipient is John, the sender's virtual agent can generate and send the outgoing communication to John's electronic device as a text message. Upon receiving the incoming text message, and assuming the recipient's electronic device is associated with the virtual agent, the virtual agent can interpret the information in the text message to perform at least one action, as discussed in more detail below. If John's device is not associated with a virtual agent, the device can display the incoming text message in a conventional format.

In some embodiments, the virtual agent may learn user preferences of users associated with the virtual agent. A virtual agent may learn user preferences in any suitable manner. For example, a virtual agent can learn user preferences by tracking one or more behavioral variables that indicate when a user prefers to send a particular type of communication. Behavioral variables that may be tracked for learning user preferences may include, but are not limited to, frequency of use, time of day, and day of the week. Additionally, user preferences may be specific to a particular intended recipient. In some embodiments, user preferences may be stored as user data associated with a particular user, although the particular manner of storing user preferences is not a limiting aspect of embodiments of the invention.

After sending the outgoing communication in act 240, the process proceeds to act 250, where a response communication is optionally received by the sending electronic device from the electronic device to which the outgoing communication was sent. The response communication may include any suitable information corresponding to the outgoing communication sent to the receiving device. For example, a responsive communication may include an acknowledgment that the outgoing communication was received by the receiving electronic device. The responsive communication may also include information in response to one or more requests included in the outgoing communication (eg, the location of the receiving user in response to a request for the location of the receiving user). The responsive communication may also include one or more requests from the receiving user's virtual agent to provide additional information prior to performing one or more actions specified in the request in the outgoing communication. Communication can continue between the sender's and receiver's virtual agents until an action has been performed or one of the users cancels the communication.

While communication between virtual agents may not require interaction from one or both users to perform an action (e.g., schedule an appointment based on the user's calendar), in some embodiments one or both users may be prompted before actually performing the action Confirm the action. For example, in some embodiments, providing information that may be considered private information (eg, a user's location) may require a user's confirmation before providing that information to another user. Constraints on sharing information between virtual assistants may be implemented in any suitable manner and embodiments of the invention are not limited in this regard.

In some embodiments, one or more tasks may be delegated to a virtual agent such that communications between one user's virtual agent and one or more other user's virtual agents occur in the background, with or without input from a virtual agent associated with the virtual agent. Intermediate input from the user. An exemplary process for managing tasks is illustrated in FIG. 3 without requiring user input to initiate the process. In act 310, the agent monitors the user data to determine whether a condition associated with the specified task has occurred.

In one illustrative example, the task assigned to the virtual agent may be to manage user events and appointments included in the user's scheduling information, and the condition the virtual agent may be configured to monitor may be whether the user is likely to respond to the scheduled Event late. The virtual agent can make this determination in any suitable manner. For example, a virtual agent associated with a first user may interact with user data for the first user to determine that the first user has a lunch meeting with the second user scheduled at a particular restaurant at 12:00pm. The virtual agent may also determine the location of the first user using, for example, a location sensor included in the first user's electronic device. When the time is close to the scheduled event (e.g., 15 minutes before the scheduled event), the virtual agent can compare the first user's location to the restaurant's location to determine whether the first user is likely to arrive at the restaurant on time for the scheduled meeting. This determination may be made in any suitable manner and embodiments of the invention are not limited to the particular manner in which the virtual agent determines the estimated travel time between two locations.

In response to determining that a condition has occurred (eg, it is determined that the user is unlikely to attend the scheduled event), the process proceeds to act 312 where it is determined whether the user should be reminded of the occurrence of the condition. The determination of whether to alert the user may be made in any suitable manner, including basing the determination at least in part on the type of task entrusted to the virtual agent. In the example above, when the virtual agent determines that the user will be late for a scheduled event, the virtual agent may alert the user that they may be late for the meeting and the process proceeds to act 314, where the user is prompted whether they should report to the second user's device An outgoing communication is sent to inform them that the first user will be late for the scheduled event. As described further below, other tasks delegated to the virtual agent may be performed without interaction or confirmation from the user and the embodiments are not limited in this regard.

In response to the user's response to the reminder in act 314, or if it is determined in act 312 that the type of task entrusted to the virtual agent does not require user reminders, the process proceeds to act 316, where it is determined whether there is at least one other user who should be reminded of the condition happened. If in act 316 it is determined that the outgoing communication should be sent to at least one other user, then the process proceeds to act 318 where the communication is sent to the at least one other agent.

In the above example, in response to determining that the first user will be late for a scheduled meeting, the virtual agent may send an outgoing communication to the second user's electronic device to notify the second user that the first user will be late. In some embodiments, the outgoing communication may also include other information including, but not limited to, the current location of the first user, and an estimated time of arrival regarding the first user's arrival at the scheduled meeting. In some embodiments, the first user's virtual agent and the second user's virtual agent may continue to communicate information about the first user's location and/or the second user's location to provide an estimated arrival of the users to the scheduled meeting Time updates in real time. For example, if the first user's virtual agent determines that the first user is likely to be thirty minutes late for a scheduled meeting, when the first user arrives at the scheduled meeting location and/or shortly before the first user's expected arrival , the virtual agent of the first user may send a follow-through communication to the virtual agent of the second user.

Once received, a virtual agent associated with the second user can interpret the information in the outgoing communication to alert the second user of the occurrence of the condition. As described above, the received communication may include additional information such as a link to a map application that identifies the first user's current location. In response to receiving a communication from the first user, the second user's virtual agent can interpret the communication to display a map on the second user's electronic device showing the first user's current location and/or estimated time of arrival. As discussed above, if the second user's electronic device does not include a compatible virtual agent capable of interpreting received communications to display, for example, a map of the first user's location, the second user can still be alerted using conventional communication applications that Communication applications such as text-based applications (eg, email or text messaging applications).

In some embodiments, a virtual agent may be tasked with monitoring events that do not require the user to be alerted in response to the occurrence of a particular condition. When a user's electronic device is placed in a particular mode (e.g., silent mode or vibrate mode) or if the electronic device is turned off, a virtual agent associated with the user can be assigned to monitor incoming and outgoing communications to the device. The sender of the incoming message sends a task for the response communication that the user is currently unavailable. In some embodiments of the invention, the response communication may include additional information that may be interpreted by a virtual agent associated with the user who sent the incoming communication. For example, the response message may include information describing when the receiving user will be available or the response message may include information that the communication is to be sent using a different format (eg, a text message rather than a phone call).

In some embodiments, the virtual assistant may alter the behavior of the electronic device based on interactions with user data of a user of the electronic device. For example, the virtual assistant may determine based on the user's scheduling information that the user is scheduled to be in a meeting from 1-2 pm. When the virtual agent determines that the user is in the meeting (for example, because the time is 1 pm), the virtual agent can instruct the electronic device to automatically go into silent or vibrate mode, thereby preventing the electronic device from disturbing others in the meeting. The virtual agent can also determine when the meeting is over (for example, because the time is 2 pm), the virtual agent can instruct the electronic device to exit silent/vibrate mode. In some embodiments, prior to exiting silent/vibrate mode, the user may be prompted to confirm whether exiting silent/vibrate mode is desired.

An exemplary process for monitoring incoming communications of a user's electronic device according to some embodiments of the invention is illustrated in FIG. 4 . In act 410, an incoming communication, such as a text message or a phone call, is received by the user's electronic device. The process then proceeds to act 412, where it is determined by the virtual agent associated with the receiving electronic device whether the user is currently available. The determination of whether a user is currently available may be made in any suitable manner. For example, if the electronic device is currently in a particular mode, such as silent mode or vibrate mode, as discussed above, the virtual agent can determine that the user is unavailable and the process can proceed to act 414, where an electronic device sending an incoming communication Send outgoing communications.

Rather than merely indicating that a user is unavailable, in some embodiments, an outgoing communication may include additional information to inform the user sending the incoming message when an unavailable user is available for communication. For example, in response to determining that the user is unavailable to receive incoming communications, the user's virtual agent may interact with the user's user data (eg, scheduling information) to determine when the user is next available for communication. After consulting the user data, the virtual agent may include in the outgoing message information indicating one or more time preferences for rescheduling the communication. By suggesting a rescheduled communication, the user sending the incoming communication may be able to choose a time that is convenient for both users. After sending the outgoing communication, the process proceeds to action 316, where the virtual agents of both users may send one or more further communications to negotiate a mutually acceptable time for both users based on their scheduling information.

In some embodiments, a user can configure a virtual agent to respond to certain incoming messages differently than other incoming messages. For example, if an incoming message is from a user identified by the virtual agent as one of the receiving user's contacts, the virtual agent may be instructed to communicate with the sending user's virtual agent to reschedule the communication at a mutually acceptable time . However, if the incoming message is from a user who is not in the receiving user's contacts, the virtual agent may be instructed not to reschedule the communication. Preferences for modifying a virtual agent's communication behavior according to some embodiments may be implemented in any suitable manner and embodiments of the invention are not limited in this respect.

In some embodiments, when an electronic device receives an incoming communication from a virtual agent, the virtual agent associated with the user of the receiving device may interpret information in the incoming communication to perform one or more actions. An exemplary process for processing incoming communications received by an electronic device according to some embodiments of the invention is illustrated in FIG. 5 . In act 510, a communication from the virtual agent of the first user is received by the electronic device of the second user. Communications may be received in any format in which an electronic device is capable of receiving incoming communications. For example, communications may be formatted as, but are not limited to, text-based messages, instant messages, phone calls, or video chat sessions. If the receiving device is associated with a virtual agent, the virtual agent is configured to perform at least one action based at least in part on the content of the information in the received communication.

A virtual agent may interpret received communications in any suitable manner to determine the content of the communications, and embodiments of the invention are not limited in this regard. For example, when an incoming message includes text, the virtual agent can determine whether any text corresponds to one or more trigger words indicative of the type of request involved in the communication. The triggering terms may be the same as or different from those discussed above in connection with invoking an agent communication session and determining the content of an outgoing communication. By determining whether an incoming communication includes a particular trigger word, the virtual agent can determine how to proceed with the request in the incoming communication. For example, if a trigger word for location information is included in the incoming message, the virtual agent may perform one or more actions with respect to the location-based request, including prompting the user for confirmation of sharing the user's location as described below . Other types of requests include, but are not limited to, scheduling requests, user content exchange requests, and requests that include multiple types of information (eg, location information and scheduling information).

It should be appreciated that identifying the type of incoming communication based on triggering words in the incoming communication is only one way in which a virtual agent may determine information about the type of content in an incoming communication and that other ways are also contemplated. For example, in one implementation, communications sent between virtual agents may be associated with metadata providing an indication of the type of content in the communication and the virtual agent may be able to determine the type of content based at least in part on the metadata associated with the communication.

In response to determining the content of the incoming communication, the process proceeds to act 514, where it is determined whether user interaction is required for one or more requests in the incoming communication. If it is determined that user interaction is required before acting on the request in the incoming communication, the process proceeds to act 516 where the user is prompted to provide user input to continue. An example of where user input may be required is if the incoming communication includes a request for user location information. For example, if the incoming communication includes "Where are you?", the virtual agent can determine that the user sending the incoming message wants to know the location of the receiving user. Some users may consider their location to be private information that generally should not be shared with other users. Thus, in response to determining that the incoming communication includes a request for the location of the user, the virtual agent of the receiving user may display to the user on the user interface of the electronic device an indication that the sending user wants to know the location of the receiving user and confirms that this information can be shared with the sending user . Alternatively, or in addition to displaying the prompt on the user interface, the electronic device may provide the prompt in some other form including voice output.

Other incoming communications besides requests for location information may also require user interaction. For example, an incoming communication from a user's virtual agent may include a request to add contact information to the receiving user's contacts and in response to determining that the request is to modify the receiving user's user data, the receiving user's virtual agent may prompt the user to indicate The sending user wants to add contact information to the receiving user's contacts. In response to receiving the prompt, the receiving user may confirm or dismiss the request to add contact information included in the incoming communication.

In some embodiments, a user may be able to set permissions and privacy permissions that a user's virtual agent may consult to determine whether user interaction is required for a particular type of request and/or what information may be shared with the particular user requesting that information or shared by all users. In some embodiments, at least some permissions or privacy permissions may be set by the virtual agent's application designer and the user may not be able to change these permissions. For example, an application designer of a virtual agent may determine that user prompts are always required for sharing location information, but permission to display user prompts for sharing user content data is user configurable. It should be appreciated that these examples are provided for illustration only and do not limit embodiments of the invention.

Returning to Figure 5, if in act 514 it is determined that no user interaction is required or user input is received in response to a prompt given in act 516, the process proceeds to act 518 where one or more actions are performed by the virtual agent to Respond to requests included in incoming communications. For example, if the incoming communication includes "Where are you?" and the receiving user has given permission to share the user's location, the virtual agent can query the electronic device's location sensors to determine the user's location. Next, the process may proceed to act 520, where the response communication is sent to the virtual agent that sent the incoming communication. In the request for the user's location, the responsive communication may include, for example, a link to a web page displaying a map application that received the user's location. In response to receiving the response communication, the agent that issued the request for location information may display a map showing the location of the other user on the receiving user's device. This can be accomplished, for example, by the receiving user's virtual agent instructing the web browser application to navigate to a link in the responsive communication.

In some embodiments, an incoming communication may include multiple requests and the receiving user's virtual agent may perform multiple actions in accordance with the requests. For example, an incoming communication may include "Tell Dave that the meeting will be postponed and let me know when he reads the information." In response to this incoming communication, the virtual agent of the receiving user (i.e., Dave) is instructed to perform two actions . The first action is to indicate to the receiving user that the meeting scheduled will be postponed, and the second action is to monitor the receiving user's behavior to determine when the receiving user has read the message that the meeting has been postponed. In response to determining that the receiving user has accessed the message, the receiving user's virtual agent may send a response message to the requesting virtual agent to fulfill a second request in the incoming communication. Thus, in this example, a virtual agent of a first user may control the behavior of a virtual agent of a second user to perform an action for the first user (in this case, monitoring the behavior of the receiving user).

Another example of handling an incoming communication that includes multiple requests is an incoming communication such as "Has my wife left the office yet? Please let me know when she will be home and let me know when she is five minutes away" . This incoming communication includes three actions that the virtual agent associated with the user's wife must perform. In response to receiving this incoming communication, the wife's virtual agent will determine the wife's location (eg, by querying a location sensor on the wife's electronic device) to determine if she is in a location corresponding to her office. Assuming the user's wife agrees to share her location, the wife's virtual agent may notify the user's virtual agent that his wife has left the office and is on his way home. The user's virtual agent can then present this information on the user's electronic device. In some embodiments, this response message may also include information about the wife's current location, which may be used by the user's virtual agent to display her location on a map.

The second action to perform is to estimate when the wife will be home. The wife's current location may also be used to determine an estimated time of arrival when she is expected to arrive home. This determination may be made in any suitable manner and may be made under the direction of the wife's virtual agent or the user's virtual agent. For example, the determination may include various factors including, but not limited to, how long it typically takes the wife to drive home from her current location, current traffic conditions, and the speed limit of the road between the wife's current location and home.

The third action in the incoming communication is to have the wife's virtual agent notify the user's virtual agent when the wife is away from home for five minutes. In some embodiments, the wife's virtual agent may provide the user's virtual agent with periodic updates describing the wife's current location. In other embodiments, the wife's virtual agent may simply monitor the wife's travel and estimate when she is five minutes away from home. When this condition is met, the wife's virtual agent may then send a communication to the user's agent that she will be home soon.

In some embodiments, agent-to-agent communication according to some embodiments of the invention may be used to facilitate tasks that require coordination between multiple virtual agents of different users. For example, a user may want to have dinner with his friends Megan and Ben at a time that is convenient for all three users. Rather than communicating with separate requests to each of Megan and Ben, the user may instruct the user's virtual agent as follows: "If Megan and Ben are free on Saturday night, please arrange dinner with them." In response, the user may determine Megan and Ben's contact information based on the user's contact information and send an outgoing communication to Megan and Ben's virtual agent to determine based on each of their stored user data whether they were on Saturday night and When will there be time for dinner. In an outgoing communication, the user's virtual agent may include one or more suggested times based on the user's scheduling information that the virtual agent may access prior to sending the outgoing communication. In response to sending an outgoing communication to Megan and Ben's virtual agents, the user's virtual agents may receive a response communication from Megan and Ben's virtual agents indicating their availability (available to attend) and the virtual agents may proceed to negotiate mutual availability to all users. accepted time. After determining an acceptable time, each user's virtual agent can update the corresponding user's scheduling information and send the user a notification that dinner has been scheduled.

As should be clear from the above description, embodiments of the present invention that provide agent-to-agent communication and permissions and privacy preferences associated with accessing user data allow for functionality that would not be expected and often impossible with conventional virtual agents. The examples provided herein are discussed merely as examples of functionality provided by embodiments of the present invention, and any other functionality that may benefit from coordination between at least two virtual agents is also possible.

An illustrative implementation of a client computer system 600 that may be used in conjunction with any of the embodiments of the invention described herein is shown in FIG. 6 . Computer system 600 may include one or more processors 610 and one or more computer-readable non-transitory storage media (eg, memory 620 and one or more non-volatile storage media 630 ). Processor 610 may control the writing of data to and reading data from memory 620 and nonvolatile storage device 630 in any suitable manner, as the aspects of the invention described herein are not limited in this respect. To perform any of the functions described herein, processor 610 may execute one or more instructions stored in one or more computer-readable storage media (e.g., memory 620), which may serve as storage for A non-transitory computer-readable storage medium for instructions executed by the processor 610 .

It should be appreciated that client computer 600 is not limited to any particular type of computer and may include, but is not limited to, handheld computers, laptop computers, tablet computers, desktop computers, smartphones, and any other type of web-enabled A computing device that renders web applications in a browser.

The above-described embodiments of the present invention can be implemented in any number of ways. For example, embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. It should be appreciated that any component or collection of components that perform the functions described above can be generally considered to be one or more controllers that control the functions discussed above. The one or more controllers can be implemented in a variety of ways, such as with dedicated hardware, or with general purpose hardware (eg, one or more processors) programmed with microcode or software to perform the functions described above.

In this regard, it should be appreciated that an implementation of an embodiment of the invention includes at least one non-transitory computer-readable storage medium (e.g., computer memory, floppy disk, compact disk, belt, etc.), when the instructions are executed on the processor, perform the above-discussed functions of the embodiments of the present invention. The computer readable storage medium may be transportable, such that the program stored thereon can be loaded onto any computer resource to implement the aspects of the present invention discussed herein. Furthermore, it should be appreciated that references to computer programs which when executed perform the functions discussed above are not limited to application programs running on a host computer. Rather, the term "computer program" is used herein in a generic sense to refer to any type of computer code (e.g., software or microprocessor) that can be used to program a processor to implement the above-discussed aspects of the present invention. code).

The various aspects of the invention may be used alone or in combination or in various arrangements not specifically discussed in the previously described embodiments, and thus their application is not limited to those set forth in the preceding description or illustrated in the drawings. details and arrangements. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

Furthermore, various embodiments of the invention may be implemented as one or more methods, examples of which are provided. Acts performed as part of a method may be ordered in any suitable manner. Thus, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts concurrently, although shown as sequential acts in illustrative embodiments.

The use of ordinal numerals such as "first," "second," "third," etc., in a claim modifying a claim element does not, in itself, imply any priority, priority, or order, or method, of one claim element over another The chronological order in which the actions are executed. These words are used merely as a marker to distinguish one claim element bearing a certain name from another element bearing the same name, but for an ordinal number.

Phrases and terms used herein are for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "involving" and variations thereof is intended to cover the items listed thereafter as well as additional items.

Having described several embodiments of the invention in detail, various modifications and improvements will readily occur to those skilled in the art. Such modifications and improvements are intended to fall within the spirit and scope of the invention. Accordingly, the above description is by way of example only, and is not intended to be limiting. The present invention is to be limited only by the claims and the equivalents thereof.

Claims (15)

1. A method of communicating between virtual agents, wherein each virtual agent is associated with a user, the method comprising: receiving voice input from a first user; processing the speech input to determine a speech recognition result; determining, by a first virtual agent associated with the first user, whether to invoke a communication session with a second virtual agent associated with the second user, wherein the determination is based at least in part on whether the speech recognition result includes one or more trigger words and information identifying the second user; and An outgoing communication is sent to a second virtual agent in response to determining to invoke the communication session. 2. The method of claim 1, further comprising: A response to the outgoing communication is received from the second virtual agent. 3. The method of claim 1, wherein the input from the first user comprises speech input. 4. The method of claim 1, wherein determining whether to invoke the communication session comprises determining whether input from the first user includes relevant contextual data. 5. The method of claim 1, further comprising: aggregating user data for the first user, wherein the user data is aggregated from multiple sources of information; store aggregated user data; and At least a portion of the stored aggregated user data is accessed by the first virtual agent. 6. The method of claim 5, wherein at least some content of the outgoing communication is determined based at least in part on the aggregated user data. 7. The method of claim 1, further comprising: identifying the second user based at least in part on information in the input received from the first user; determining whether the second user is associated with the virtual agent; and Outgoing communications are sent only when it is determined that the second user is associated with the virtual agent. 8. The method of claim 7, wherein identifying the second user comprises: A second user is identified based at least in part on the user data for the first user. 9. The method of claim 8, wherein the user data includes a list of contacts associated with the first user. 10. The method of claim 1, further comprising: determining a preferred mode of communication for the second user; and Outgoing communications are sent using the preferred communication mode. 11. The method of claim 1, wherein the response to the outgoing communication received from the second virtual agent includes a request to coordinate a new appointment; and wherein the method further comprises: At least one additional outgoing message is sent to the second virtual agent to coordinate the new appointment, wherein the at least one additional outgoing message is sent to the second virtual agent without input from the first user. 12. The method of claim 2, wherein at least one action comprises sending a response communication to the first virtual agent. 13. A computer readable medium encoded with a plurality of instructions which, when executed by at least one computer, perform a method of communicating between virtual agents, wherein each virtual agent is associated with a user, the method comprising: receiving voice input from a first user; processing the speech input to determine a speech recognition result; determining, by a first virtual agent associated with the first user, whether to invoke a communication session with a second virtual agent associated with the second user, wherein the determination is based at least in part on whether the speech recognition result includes one or more trigger words and information identifying the second user; and An outgoing communication is sent to the second virtual agent in response to determining to invoke the communication session, wherein the outgoing communication includes a request for the second virtual agent to perform at least one action for the first user. 14. An electronic device comprising: a communication interface configured to receive speech input from a first user; a speech recognition engine configured to process the speech input to determine a speech recognition result; and The first virtual agent is configured as: determining whether to invoke a communication session with a second virtual agent associated with a second user, wherein the determination is based at least in part on whether the speech recognition result includes one or more trigger words and information identifying the second user; and An outgoing communication is sent to the second virtual agent in response to determining to invoke the communication session, wherein the outgoing communication includes a request for the second virtual agent to perform at least one action for the first user. 15. The electronic device of claim 14, wherein the communication interface includes at least one voice interface configured to recognize voice input.