CN111880425A - Intelligent assistant for home automation - Google Patents

Abstract

The invention relates to an intelligent assistant for home automation. In one exemplary process, a user can speak input in natural language form to a user device to control one or more electronic devices. The user device may transmit the user speech to the server for conversion to a text representation. The server may identify the one or more electronic devices and appropriate commands to be executed by the one or more electronic devices based on the textual representations. The identified one or more devices and commands to be executed may be transmitted back to the user device, which may forward the commands to the appropriate one or more electronic devices for execution. In response to receiving the command, the one or more electronic devices may execute the command and transmit their current state to the user device.

Description

Smart Assistants for Home Automation

This application is a divisional application of a Chinese invention patent application with an application date of March 31, 2015, an application number of 201580028666.0, and an invention name of "Intelligent Assistant for Home Automation".

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application claims U.S. Provisional Application Serial No. 62/005,893, filed on May 30, 2014, entitled "INTELLIGENT ASSISTANT FOR HOME AUTOMATION" and filed on September 30, 2014, entitled "INTELLIGENT ASSISTANT FOR HOME AUTOMATION" Priority to US non-provisional application Serial No. 14/503,105, which is hereby incorporated by reference in its entirety for all purposes.

technical field

The present application relates generally to natural language processing, and more particularly to controlling electronic devices using virtual assistants with natural language processing.

Background technique

Home electronic devices that can be controlled remotely using software applications running on computing devices such as mobile phones, tablets, laptops, desktop computers, etc., have become increasingly popular. For example, many manufacturers produce light bulbs that can be controlled by a software application running on a mobile phone to adjust the brightness and/or color of the light bulb. Other devices with similar controls such as door locks, thermostats, etc. may also be used.

While these devices may provide users with a higher level of control and convenience, as the number and types of remotely controlled devices in the home increase, managing these devices has become very difficult. For example, a typical home may include 40 to 50 light bulbs placed in various rooms of the home. Using conventional software applications, each light bulb is given a unique identifier, and a user attempting to control one of these devices must select the appropriate identifier from a list of available devices within the graphical user interface. Remembering the correct identifier for a particular light bulb and looking it up from a list of 40 to 50 identifiers can be a difficult and time consuming process. Different manufacturers often provide different software applications that must be used to control their respective devices, which also adds to the difficulty of managing and controlling a large number of remotely controlled devices. So the user has to locate and open one software application to turn on/off the corresponding light bulb, and then has to locate and open another software application to set the temperature of their thermostat.

SUMMARY OF THE INVENTION

The present invention provides systems and processes for controlling electronic devices using virtual assistants. In one example process, a user can speak input to the user device in natural language to control one or more electronic devices. The user device may transmit the user speech to the server for conversion to a textual representation. The server may identify one or more electronic devices and appropriate commands to be executed by the one or more electronic devices based on the textual representation. The identified device or devices and the command to be executed can be transmitted back to the user device, which can forward the command to the appropriate electronic device or devices for execution. In response to receiving the command, one or more electronic devices can execute the command and transmit its current state to the user device.

Description of drawings

1 illustrates an exemplary environment in which a virtual assistant may be used to control an electronic device, according to various embodiments.

2 illustrates an example environment in which a virtual assistant may be used to remotely control an electronic device, according to various embodiments.

Figure 3 illustrates an exemplary user equipment according to various embodiments.

4 illustrates a visual representation of multiple items for storing information associated with an electronic device, according to various embodiments.

5 illustrates an exemplary process for controlling an electronic device using a virtual assistant implemented using a client-server model, according to various embodiments.

6 illustrates an exemplary process for remotely controlling an electronic device using a virtual assistant implemented using a client-server model, according to various embodiments.

7 illustrates an exemplary process for controlling an electronic device using a virtual assistant on a stand-alone user device, according to various embodiments.

8 illustrates an exemplary process for storing the state of an electronic device as a configuration, according to various embodiments.

9 illustrates an exemplary process for setting a state of an electronic device using a previously stored configuration, according to various embodiments.

10 shows a functional block diagram of an electronic device configured to control the electronic device, according to various embodiments.

11 illustrates a functional block diagram of an electronic device configured to store a state of the electronic device as a configuration, according to various embodiments.

12 illustrates a functional block diagram of an electronic device configured to store a state of the electronic device as a configuration, according to various embodiments.

13 shows a functional block diagram of an electronic device configured to control the electronic device in accordance with various embodiments.

14 illustrates a functional block diagram of an electronic device configured to store a state of the electronic device as a configuration, according to various embodiments.

15 illustrates a functional block diagram of an electronic device configured to store the state of the electronic device as a configuration, according to various embodiments.

Detailed ways

In the following description of the examples, reference will be made to the accompanying drawings, in which specific examples that may be implemented are shown by way of illustration. It is to be understood that other examples may be utilized and structural changes may be made without departing from the scope of the various examples.

Intelligent automated assistants (or virtual assistants) provide an intuitive interface between users and electronic devices. These assistants may allow users to interact with devices or systems using natural language in the form of speech and/or text. For example, a user may access services of an electronic device by providing voice user input in natural language to a virtual assistant associated with the electronic device. Virtual assistants can perform natural language processing on speech user input to infer user intent and implement user intent into tasks. The task can then be performed by performing one or more functions of the electronic device, and the relevant output can be returned to the user in natural language.

This involves the use of virtual assistants to control the systems and processes of electronic devices. In one example process, a user can speak input to the user device in natural language to control one or more electronic devices. The user device may transmit the user speech to the server for conversion to a textual representation. The server may identify one or more electronic devices and appropriate commands to be executed by the one or more electronic devices based on the textual representation. The identified device or devices and the command to be executed can be transmitted back to the user device, which can forward the command to the appropriate electronic device or devices for execution. In response to receiving the command, one or more electronic devices can execute the command and transmit its current state to the user device.

System Overview

FIG. 1 illustrates an exemplary system 100 for implementing a virtual assistant to control electronic devices, according to various embodiments. The terms "virtual assistant", "digital assistant", "intelligent automated assistant" or "automated digital assistant" may refer to interpreting natural language input in voice and/or text form to infer user intent, and based on the inferred user intent Any information processing system that performs an action. For example, to perform inferred user intent, the system may perform one or more of the following: identify a task flow by designing steps and parameters to implement the inferred user intent; The specific requirements of the input are input into the task flow; the task flow is executed by invoking procedures, methods, services, APIs, etc.;

The virtual assistant may be capable of accepting user requests, at least in part, in the form of natural language commands, requests, statements, narrations, and/or queries. Typically, user requests either seek informational responses from a digital assistant or a virtual assistant to perform a task. A satisfactory response to a user request may include providing the requested informative answer, performing the requested task, or a combination of the two. For example, a user may ask the virtual assistant a question such as "Where am I now?" Based on the user's current location, the virtual assistant can answer "You're in Central Park." The user can also request a task, such as "Please remind me to call mom at 4 pm today." In response, the virtual assistant can confirm the request and then create the appropriate reminder item in the user's electronic timesheet. During the performance of a requested task, the virtual assistant may interact with the user in an ongoing conversation involving multiple exchanges of information, sometimes over long periods of time. There are many other methods of interacting with virtual assistants to request information or perform various tasks. In addition to providing verbal responses and taking programmed actions, virtual assistants can also provide responses in other forms of video or audio (eg, as text, alerts, music, video, animation, etc.).

Examples of virtual assistants are described in Applicant's US Utility Model Application Serial No. 12/987,982, filed January 10, 2011, entitled "Intelligent AutomatedAssistant," the entire disclosure of which is incorporated by reference This article.

As shown in FIG. 1, in some embodiments, the virtual assistant may be implemented according to a client-server model. The virtual assistant may include a client-side portion that executes on the user device 102 and a server-side portion that executes on the server system 110 . User equipment 102 may include any electronic device, such as a mobile phone, tablet computer, portable media player, desktop computer, laptop computer, PDA, television, television set-top box, wearable electronic device, etc., and may be accessed through one or more The network 108 communicates with the server system 110, which may include the Internet, an intranet, or any other wired or wireless public or private network. The client-side portion executing on the user device 102 may provide client-side functionality, such as user-oriented input and output processing and communication with the server system 110 . Server system 110 may provide server-side functionality for any number of clients residing on respective user devices 102 .

Server system 110 may include one or more virtual assistant servers 114, which may include client-facing I/O interfaces 122, one or more processing modules 118, data and model storage 120, and I/O interfaces 116 to external services. Client-facing I/O interface 122 may facilitate client-facing input and output processing of virtual assistant server 114 . One or more processing modules 118 may utilize the data and model store 120 to determine the user's intent based on natural language input and perform task performance based on the inferred user intent. Additionally, data and model store 120 may store unique identifiers, states, types associated with one or more electronic devices (eg, electronic devices 128 , 130 and 132 ) that can be controlled by user device 102 and/or server system 110 , location and any other relevant information. In some embodiments, the virtual assistant server 114 may communicate with external services 124, such as telephony services, calendar services, information services, messaging services, navigation services, etc., over the network 108 for task completion or information acquisition. I/O interfaces 116 to external services may facilitate such communications.

Server system 110 may be implemented on one or more stand-alone data processing devices of a computer or on a distributed network. In some embodiments, server system 110 may employ various virtual appliances and/or services of third-party service providers (eg, third-party cloud service providers) to provide potential computing resources and/or infrastructure resources of server system 110 .

User equipment 102 may also be coupled to electronic devices 128 , 130 , and 132 via one or more networks 126 . Electronics 128, 130, and 132 may include any type of remote control electronics, such as light bulbs (eg, with binary on/off status, digital dimmable status, color status, etc.), garage doors (eg, with binary open/closed status), Door locks (e.g. with binary lock/unlock states), thermostats (e.g. with one or more digital temperature states such as high, low, time-based, etc.), electrical outlets (e.g. with binary on/off states) , switches (eg with binary on/off states), etc. Network 126 may include a WiFi network or any other wired or wireless public or private local network. Additionally or alternatively, user equipment 102 may be coupled to communicate directly with electronic device 128, 130 or 132 using, for example, Bluetooth, BTLE, line-of-sight, peer-to-peer, or another radio-based communication or other wireless communication. Thus, in the illustrated embodiment, user equipment 102 may be located near electronic devices 128, 130, and 132 such that the user equipment and these electronic devices may communicate directly or through the same local network. For example, user device 102 and electronic devices 128, 130, and 132 may be located within the same home or building, and network 126 may include the home or building's WiFi network. As discussed in greater detail below with reference to FIGS. 5, 8, and 9, user device 102 may issue commands to control any of electronic devices 128, 130, and 132 in response to natural language speech input provided to user device 102 by the user. One.

Although only three electronic devices 128, 130, and 132 are shown, it should be understood that system 100 may include any number of electronic devices. Additionally, although the functionality of the virtual assistant is shown in FIG. 1 as including a client-side portion and a server-side portion, in some embodiments, the functionality of the virtual assistant may be implemented as a stand-alone application installed on a user device. Furthermore, the functional division between the client portion and the server portion of the virtual assistant may vary in different embodiments. For example, in some embodiments, the client executing on the user device 102 may be a thin client that provides only user-facing input and output processing functions and delegates all other functions of the virtual assistant to the backend server.

FIG. 2 illustrates another exemplary system 200 for implementing a virtual assistant to remotely control electronic devices in accordance with various embodiments. Similar to system 100 , system 200 may include user equipment 102 , server system 110 , and external services 124 communicatively coupled together through network 108 . However, in contrast to system 100, user device 102 may not be coupled to electronic devices 128, 130, and 132. Alternatively, system 200 may include second user equipment 134 coupled to communicate with user equipment 102 and/or server system 110 via network 108 and coupled to communicate with electronic devices 128 , 130 via network 126 Communicate with 132. This configuration may represent a situation where the user and user device 102 are remote from electronic devices 128, 130 and 132 (eg, user and user device 102 are in the user's office and electronic devices 128, 130 and 132 are in the user's home).

The second user equipment 134 may comprise any type of electronic device, such as a mobile phone, tablet computer, portable media player, desktop computer, laptop computer, PDA, television, television set-top box, wearable electronic device, etc., and may be used by Configured to receive commands from user device 102 and/or server system 110 and issue commands to electronic devices 128 , 130 and 132 . As discussed in more detail below with reference to FIG. 6 , the second user device 134 may issue commands to control any of the electronic devices 128 , 130 and 132 in response to natural language speech input provided by the user to the user device 102 .

User equipment

Figure 3 is a block diagram of the user equipment 102 (or the second user equipment 134) according to various embodiments. As shown, user equipment 102 may include a memory interface 302 , one or more processors 304 , and a peripherals interface 306 . The various components in user equipment 102 may be coupled together by one or more communication buses or signal lines. User device 102 may also include various sensors, subsystems, and peripherals coupled to peripherals interface 306 . Sensors, subsystems, and peripherals collect information and/or facilitate various functions of the user device 102 .

For example, user device 102 may include motion sensor 310, light sensor 312, and proximity sensor 314 coupled to peripherals interface 306 to facilitate orientation, lighting, and proximity sensing functions. One or more other sensors 316, such as positioning systems (eg, GPS receivers), temperature sensors, biometric sensors, gyroscopes, compasses, accelerometers, etc., are also connected to the peripherals interface 306 to facilitate related functions.

In some embodiments, camera subsystem 320 and optical sensor 322 may be utilized to facilitate camera functions, such as taking pictures and recording video clips. Communication functions may be facilitated by one or more wired and/or wireless communication subsystems 324, which may include various communication ports, radio frequency receivers and transmitters, and/or optical (eg, infrared) Receiver and transmitter. Audio subsystem 326 may be coupled to speaker 328 and microphone 330 to facilitate voice-enabled functions, such as voice recognition, voice reproduction, digital recording, and telephony functions.

In some embodiments, user device 102 may also include an I/O subsystem 340 coupled to peripherals interface 306 . I/O subsystem 340 may include touch screen controller 342 and/or other input controllers 344 . Touch screen controller 342 may be coupled to touch screen 346 . The touch screen 346 and touch screen controller 342 may, for example, detect contacts and movements or their discontinuities using any of a variety of touch sensitivity techniques such as capacitive, resistive, infrared, and surface acoustic wave techniques , proximity sensor arrays, etc. Other input controllers 344 may be coupled to other input/control devices 348, such as one or more buttons, rocker switches, thumb wheels, infrared ports, USB ports, and/or a pointing device (eg, a stylus).

In some embodiments, user equipment 102 may also include a memory interface 302 coupled to memory 350 . Memory 350 may include any electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, portable computer disk (magnetic), random access memory (RAM) (magnetic), read only memory (ROM) (magnetic), Erasable Programmable Read Only Memory (EPROM) (magnetic), portable optical discs such as CD, CD-R, CD-RW, DVD, DVD-R or DVD-RW, or flash memory such as compact flash cards, secure digital cards , USB storage devices, memory sticks, etc. In some embodiments, the non-transitory computer-readable storage medium of memory 350 may be used to store storage media for use by or in conjunction with an instruction execution system, apparatus, or device (eg, for performing the execution of the instructions described below). some or all of process 500, 600, 700, 800, or 900) instructions that execute a system, apparatus, or device, such as a computer-based system, a system including a processor, or executable from the instructions Other systems that fetch instructions and execute them. In other embodiments, instructions (eg, for performing process 500, 600, 700, 800, or 900 described below) may be stored on a non-transitory computer-readable storage medium of server system 110, or may be stored in memory 350 is divided between the non-transitory computer-readable storage medium of the server system 110 and the non-transitory computer-readable storage medium of the server system 110 . In the context of this document, a "non-transitory computer-readable storage medium" can be any medium that can contain or store a program for use by or in connection with the instruction execution system, apparatus, and apparatus.

In some embodiments, memory 350 may store operating system 352 , communication module 354 , graphical user interface module 356 , sensor processing module 358 , telephony module 360 , and application programs 362 . Operating system 352 may include instructions for handling underlying system services and for performing hardware-related tasks. Communication module 354 may facilitate communication with one or more additional devices, one or more computers, and/or one or more servers. Graphical user interface module 356 may facilitate graphical user interface processing. Sensor processing module 358 may facilitate sensor-related processing and functions. The telephony module 360 may facilitate telephony-related processes and functions. The application module 362 may facilitate various functions of the user application, such as electronic messaging, web browsing, media processing, navigation, imaging, and/or other processes and functions.

Memory 350 may also store client-side virtual assistant instructions (eg, in virtual assistant client module 364) and various user data 366 (eg, user-specific vocabulary data, preference data, and/or other data such as the user's electronic address book, to-do list, shopping list, etc.) to provide client-side functionality for virtual assistants.

In various embodiments, virtual assistant client module 364 may be capable of accepting voice input (eg, voice input), text input through various user interfaces of user device 102 (eg, I/O subsystem 340 , audio subsystem 326 , etc.) , touch input and/or gesture input. The virtual assistant client module 364 may also be capable of providing output in the form of audio (eg, voice output), video, and/or haptics. For example, the output may be provided as speech, sound, alarm, text message, menu, graphics, video, animation, vibration, and/or a combination of two or more of the above. During operation, the virtual assistant client module 364 may use the communication subsystem 324 to communicate with the virtual assistant server.

In some embodiments, virtual assistant client module 364 may utilize various sensors, subsystems, and peripherals to gather additional information from the surrounding environment of user device 102 to establish association with the user, current user interaction, and/or current user input context. In some embodiments, the virtual assistant client module 364 may provide contextual information, or a subset thereof, along with user input to the virtual assistant server to help infer the user's intent. The virtual assistant may also use contextual information to determine how to prepare and deliver output to the user.

In some embodiments, contextual information accompanying user input may include sensor information such as lighting, ambient noise, ambient temperature, images or video of the surrounding environment, distance to another object, and the like. The contextual information may also include information related to the physical state of the user device 102 (eg, device orientation, device location, device temperature, power level, speed, acceleration, motion mode, cellular signal strength, etc.) or the software state of the user device 102 (eg, running process). , installed programs, past and current network activity, background services, error logs, resource usage, etc.) Any of these types of contextual information may be provided to virtual assistant server 114 as contextual information associated with user input.

In some embodiments, virtual assistant client module 364 may selectively provide information (eg, user data 366 ) stored on user device 102 in response to a request from virtual assistant server 114 . The virtual assistant client module 364 may also elicit additional input from the user via a natural language dialog or other user interface when requested by the virtual assistant server 114 . The virtual assistant client module 364 may communicate the additional input to the virtual assistant server 114 to assist the virtual assistant server 114 in making intent inference and/or satisfying the user intent expressed in the user request.

Memory 350 may also store electronic device data 370 , which may include information related to one or more of electronic devices (eg, electronic devices 128 , 130 , and 132 ) that can be controlled by user device 102 and/or server system 110 The unique identifier, status, type, location and any other relevant information of the link. FIG. 4 shows a visual representation of entries that may be stored in electronic device data 370 for seven different electronic devices. As shown, each entry includes the unique name, type, and status of the electronic device. The data and model store 120 of the virtual assistant server 114 may include similar or identical entries for electronic devices, which may be kept separate from the electronic device entries of the electronic device data 370 of the memory 350 .

Referring back to FIG. 3 , memory 350 may also include entries (eg, in daemon module 368 ) for creating and updating electronic device entries in electronic device data 370 , for communicating with electronic devices of system 100 , and for communicating with server system 110 communication instructions. For example, to add an electronic device to system 100, a software application associated with the electronic device may communicate with processor 304 executing daemon module 368 to provide user device 102 with the electronic device's unique name, type, status, location, etc. . The software application may allow the user to enter the unique name in any desired manner. For example, a drop-down box with a common name and/or a free-form text field can be provided in the application to allow the user to name a specific device. The type, state and/or location of the electronic device may be predetermined or determined by the software application by communicating with the electronic device. The processor 304 executing the daemon module 368 may store this information as an entry in the electronic device data 370 and may also transmit the information to the server system 110 for storage in the data and model storage 120 . Additionally, when executed by processor 304, daemon module 368 may receive commands from server system 110 to provide to electronic devices 128, 130, and 132 via a network, and may transmit the commands to the appropriate electronic device. The processor 304 executing the daemon module 368 may also receive status updates from the electronic devices 128, 130 and 132, update corresponding entries in the electronic device data 370 to reflect the updated status of the devices, and transmit the status updates to the server system 110 to The server system 110 is allowed to update the corresponding entries in the data and model store 120 to reflect the updated status of the device.

Additionally, daemon module 368 may include instructions for managing access to electronic device data 370 by other devices and software applications. For example, when executed by processor 304 , daemon module 368 may grant server system 110 access to all electronic device data 370 , but may restrict other devices or software applications from accessing only a portion of electronic device data 370 . This may be useful when the user equipment 102 is used to control electronic equipment made by different manufacturers. In these cases, a device or software application from each manufacturer can use the API to communicate with the daemon module 368, and the daemon module 368 can restrict the device or software application to access only the electronic device data 370 that is related to their respective The part corresponding to the electronic device. For example, company X may manufacture light bulbs that can be controlled by user device 102 , and company Y may manufacture thermostats that can be controlled by user device 102 . The daemon module 368 may facilitate communication between the user device 102 and each of the light bulb and thermostat, allowing the user device 102 to issue commands to the electronic device and receive status information associated with the electronic device for updating the electronic device Data 370. However, the daemon module 368 may limit the access of the light bulb (and related software applications running on the user device 102) to the information in the electronic device data 370 only to those related to the light bulb (and any other electronic devices that may be manufactured by company X). link information. Similarly, the daemon module 368 may limit the access of the thermostat (and related software applications running on the user device 102 ) to the information in the electronic device data 370 only to those of the thermostat (and any other electronic devices that may be manufactured by company Y). device) associated information. However, daemon module 368 may grant server system 110 access to all information in electronic device data 370 .

In various embodiments, memory 350 may include additional instructions or fewer instructions. Furthermore, the various functions of the user equipment 102 may be performed in hardware and/or firmware, including in one or more signal processing and/or application specific integrated circuits.

Local control of electronic equipment

FIG. 5 illustrates an exemplary process 500 for controlling an electronic device using a virtual assistant. In some embodiments, process 500 may be performed using a system similar to or the same as system 100 shown in FIG. 1 . In these embodiments, the blocks of process 500 may be performed by both user device 102 and server system 110 . Specifically, the blocks on the left side of FIG. 5 may be performed by the user equipment 102 , while the blocks on the right side of FIG. 5 may be performed by the server system 110 .

At block 502, audio input including a user's speech may be received at a user device. In some embodiments, a user device (eg, user device 102 ) may receive audio input including the user's speech via a microphone (eg, microphone 330 ). The microphone may convert the audio input to an analog or digital representation and provide the audio data to one or more processors (eg, processor 304).

At block 504, data corresponding to the audio input received at block 502 may be transmitted to one or more servers for processing. For example, user device 102 may transmit data corresponding to the audio input to virtual assistant server 114 of server system 110 via network 108 .

At block 506, data corresponding to the audio input transmitted by the user device at block 504 may be received by one or more servers. For example, virtual assistant server 114 of server system 110 may receive data corresponding to audio input transmitted by user device 102 via network 108 .

At block 508, speech-to-text conversion may be performed on the data corresponding to the audio input to convert the user's speech into a textual representation of the user's speech. The user speech may be converted using any known speech-to-text conversion method.

At block 510 , one or more electronic devices may be identified based at least in part on the textual representation generated at block 508 . In some embodiments, block 510 may include processing the textual representation entered by the user to determine user intent to issue commands to one or more electronic devices. As noted above, server system 110 may include one or more data and model stores 120 that may store unique identifiers, states, types, locations, and any associated electronic devices that may be controlled using system 100 . other relevant information. Accordingly, block 510 may include identifying one or more of the electronic devices having associated information stored in the data and model store 120 .

One or more electronic devices may be identified in any number of ways. In some embodiments, one or more electronic devices may be identified by parsing the textual representation to identify any of a set of nouns supported by the system 100 that correspond to electronic devices. For example, the set of nouns may include the unique name of the electronic device stored in the electronic device data 370 and the data and model storage 120, possible types of electronic device and their synonyms (eg, garage door, thermostat, light, tunable light) , switches, color-changing lights, bulbs, lamps, locks, sockets, sockets, etc.), possible device status categories (e.g. volume, temperature, brightness, color, etc.), etc.

To illustrate, using the exemplary electronic device entry of FIG. 4, the set of nouns may include the unique names of seven electronic devices (eg, "garage door", "upstairs thermostat", "downstairs thermostat", "living room light 1" ", "Living Room Light 2", "Front Door" and "Toaster Socket"), possible types of electronic devices and their synonyms (e.g. "garage door", "thermostat", "illuminator", "light bulb", " Lights, Locks, Outlets, Sockets, etc.), and possibly a category of device status (e.g. temperature). Thus, if the textual representation of the user speech generated at block 508 includes "lock the front door," the textual representation may be processed using the processing module 118 and the data and model store 120 to search for any of the set of nouns. As a result of the search, it may be determined that the text representation includes the unique name "Front Door," and the electronic device identified at block 510 may include the device. It should be understood that more than one electronic device may be identified at block 510 depending on the textual representation of the user's speech. For example, if the textual representation of the user's speech generated at block 508 instead includes "turn off all the light bulbs," the textual representation may be processed using the processing module 118 and the data and model store 120 to search for any of the set of nouns . As a result of the search, it was determined that the textual representation included the possible device type "light bulb", a synonym for "light bulb", and that the instruction from the user was to turn off all these types of devices. Thus, the electronic devices identified at block 510 may include "Living Room Light 1" and "Living Room Light 2."

In some embodiments, it may be difficult to identify an appropriate electronic device using only the aforementioned set of nouns. For example, a text representation including "Turn On Lights" may result in a type match to both "Living Room Light 1" and "Living Room Light 2." In these embodiments, block 510 may also include using the contextual information received from the user device 102 (eg, received as part of the data corresponding to the audio input at block 506 ) to eliminate potential matches between electronic devices ambiguity. Any type of contextual information may be used, such as sensor information (eg, lighting, ambient noise, ambient temperature, image or video of the surrounding environment, distance to another object, etc.), physical state of the user device 102 (eg, device orientation, device orientation, etc.) location, device temperature, power level, speed, acceleration, motion mode, cellular signal strength, etc.), software status of the user device 102 (eg, running processes, past and present network activity, background services, error logs, resource usage, etc.) ) and other related information. For example, continuing with the "turn on lights" embodiment provided above, when user equipment 102 receives audio input at block 502, the location and orientation of user equipment 102 may be provided to server system 110 at blocks 504 and 506 . This contextual information may be used to determine the location of the user equipment 102 and/or the direction in which the user equipment 102 was pointed when the user's speech was received. When compared to the known positions of "Living Room Light 1" and "Living Room Light 2" stored in the data and model memory 120, the closest illuminant or illuminant pointed by the user device 102 may be selected at block 510 as the Electronic equipment. Other types of contextual information may be used in a similar manner at block 510 to disambiguate between potential matching electronic devices by identifying contextual information that makes one or more potential matching electronic devices more or less likely. Has been referenced by the textual representation of the user's speech.

In other embodiments, words associated with states in the user-entered text representation may additionally or alternatively be used to disambiguate between potentially matching electronic devices or to identify appropriate electronic devices. For example, a textual representation including "go it to 68" may not use the set of nouns described above to produce any matches. Accordingly, in these embodiments, block 510 may also include parsing the textual representation to identify any of a set of words associated with a state corresponding to an electronic device supported by system 100 . For example, the set of words associated with states may include possible states of the electronic device and their synonyms (eg, open, closed, closed, off, open, closed, active, passive, locked, locked, color, etc. ), the type of state value (e.g. binary, floating point, etc.), queries for device states, adjectives associated with a particular type of state (e.g. warmer, cooler, brighter, darker, color, etc.), etc. . When used to parse "turn it to 68", it can be determined that the text representation includes the floating point value "68". When compared to the entries shown in Figure 4, it can be determined that only "Upstairs Thermostat" and "Downstairs Thermostat" accepted the status float value. To disambiguate between the two thermostats, contextual information (eg, location of user device 102 ) may be used to select the thermostat that is closer to user device 102 as the electronic device identified at block 510 . Similarly, if the textual representation instead includes "make it brighter," the set of words associated with the state can be used to determine that the textual representation includes the word "brighter," which is a description that has "lighter" ” An adjective for the state of a device of the type. To disambiguate between the two lights, contextual information (eg, location of user device 102 ) may be used to select the light closer to user device 102 as the electronic device identified at block 510 .

At block 512, a command to be executed by each of the one or more electronic devices identified at block 510 may be identified. The command to execute can be identified in any number of ways. In some embodiments, a textual representation of the user's speech may be parsed to identify any of a set of words associated with a state (eg, possible states of an electronic device and their synonyms, type of state value, query for device state , adjectives associated with a particular type of state), identifying the command to execute. The identified state or operation may then be used to identify a command to be executed by each of the one or more electronic devices identified at block 510 . For example, if the textual representation of the user's speech includes "lock the front door," it may be determined that the textual representation includes the "locked" state from the state set. Accordingly, a command to transition to the "locked" state may be generated and identified at block 512 as a command to be executed by the electronic device "front door" identified at block 510 . Although a single command for a single electronic device is identified in the above-described embodiments, it should be understood that if multiple electronic devices are identified at block 510, multiple commands may be identified. For example, the text representation "turn on all lights" may cause a command to transition to the "on" state for each of "living room light 1" and "living room light 2" to be identified at block 512 .

In some embodiments, when the textual representation of the user's speech includes one of the adjectives associated with a particular type of state (eg, warmer, cooler, brighter, darker, etc.), block 512 may include identifying the The state of the electronic device identified at block 510 sets the command to a value relative to its current value. For example, if the textual representation includes "make it warm," the command identified at block 514 may be a command to increase the temperature of the thermostat identified at block 510 by a predetermined amount. The command may be a command that changes the state value by an amount relative to the current value of the electronic device, rather than transitioning the state value to a specific value (eg, as determined using the thermostat state stored in the data and model memory 120 ) command, because the actual state of the electronic device may differ from the state stored in the data and model memory 120 .

In some embodiments, when the textual representation of the user's speech includes a query for device status, block 512 may include identifying instructions for causing the user device 102 to query the identified electronic device. For example, if the textual representation includes "Is the garage door closed?", the command identified at block 514 may be a command to query the status of the electronic device "garage door."

At block 514, the identification of each of the one or more electronic devices identified at block 510 and the command to be executed by the one or more electronic devices identified at block 512 may be transmitted to the user equipment. For example, server system 110 may transmit a unique identifier associated with each electronic device identified at block 510 and commands to be executed by one or more electronic devices to user device 102 via network 108 .

In some embodiments, the textual representation of the user's speech generated at block 508 may also be transmitted to the user device at block 514 . In these embodiments, blocks 510 and 514 may also be performed on the user equipment. The transmitted text representation may be used by the user device to identify one or more electronic devices and/or identify one or more commands in the electronic device data 370 to be executed by the one or more electronic devices. This may be desirable, for example, where electronic device data 370 in the user device is updated compared to data on one or more servers. In this case, the user equipment may identify the electronic equipment and/or commands not included in the data of one or more servers.

In other embodiments, the textual representation may be parsed in one or more servers at block 510 and/or block 512 to determine which electronic device(s) may be suitable for identifying and/or identifying to be processed by the one or more electronic devices. A keyword or term for one or more commands performed by an electronic device. In these embodiments, the parsed keywords or terms may also be transmitted to the user device at block 514 . The transmitted keywords or terms may be used by the user device to identify one or more electronic devices and/or identify one or more commands in the electronic device data 370 to be executed by the one or more electronic devices. This may be desirable, for example, where electronic device data 370 in the user device is updated compared to data on one or more servers. In this case, the user equipment may identify the electronic equipment and/or commands not included in the data of one or more servers.

At block 516, the identification of each of the one or more electronic devices and the command transmitted by the one or more servers may be received by the user equipment. For example, the user device 102 may receive the unique identifier and command transmitted by the server system 110 via the network 108 at block 514 .

At block 518 , the user equipment may transmit the command received at block 516 to the electronic device associated with the identifier received at block 516 . For example, if the user device 102 receives an identifier associated with the electronic device 128 (eg, "front door") and a command to transition the electronic device to the "locked" state at block 516, the user device 102 may via the network 126 transmits the command to electronic device 128 to transition the electronic device to the "locked" state. If the user device 102 receives additional identifiers and commands at block 516 , the user device 102 may further transmit these commands to the identified electronic device at block 518 .

In some embodiments, the actual state of the electronic device may differ from the electronic device state stored in user device 102 (eg, memory 350 ) and/or server system 110 (eg, data and model memory 120 ). For example, a door lock may have been manually opened or closed without the use of a virtual assistant of system 100 . Accordingly, in some embodiments, block 518 may be performed regardless of the electronic device state stored in user device 102 and/or server system 110 . For example, a command to set the door lock state to "locked" may still be transmitted to the door lock even if the corresponding entry in the user device 102 and/or server system 110 indicates that the door has been locked. Additionally, block 518 may be performed without first querying the electronic device (eg, between blocks 516 and 518) to determine its actual status, in order to reduce the amount of time required to issue commands to the electronic device. For example, a command to set a door lock status to "locked" can be transmitted to a door lock without first querying the door lock status, thereby reducing the time required to transmit a command to a door lock versus sending a query to the door lock and The amount of time it takes to receive the door lock status.

At block 520, after each electronic device to which the command was sent at block 518 executes their respective commands, the user equipment may receive the current state of each electronic device. For example, user device 102 may receive an updated state of the electronic device after electronic device 128 executes a command to set the state of the electronic device to "locked." In this embodiment, the current state returned to the user equipment 102 may be the "locked" state. If the command was sent to more than one electronic device at block 518, block 520 may also include receiving the current status of those electronic devices. In some embodiments, block 520 may also include updating the electronic device state in the electronic device data 370 based on the received current state. For example, the user device 102 may update the electronic device 128 status in the electronic device data 370 to "locked". In some embodiments, block 520 may also include outputting an audio or visual indication of the result of the command transmitted to the electronic device at block 518 based on the transmitted command and the received current state of the electronic device. For example, if the command transmitted to the electronic device 128 is a command to transition the device state to "locked" and the current state of the electronic device 128 received at block 520 is "locked," an indication of the result may be that the electronic device 128 successfully Transition to "locked" state. Alternatively, if the current state of the electronic device received (eg, "unlocked") is different from the desired state (eg, "locked") indicated in the command, an indication of an unsuccessful transition to the "locked" state may be presented to the user . The current state may also include an error state, such as an undetermined state or an unavailable state of the device.

At block 522, the user equipment may transmit the current state of the electronic device received at block 520 to one or more servers. For example, the user device 102 may transmit the current state of the electronic device received at block 520 to the server system 110 . If the current state of more than one electronic device is received at block 520 , block 522 may also include transmitting these current states to the server system 110 .

At block 524, the one or more servers may receive the current state of the electronic device transmitted by the user equipment at block 522. For example, at block 522 , the server system 110 may receive the current state of the electronic device transmitted by the user device 102 . If the user equipment 102 has transmitted more than one current state, block 524 may also include receiving those current states. In some embodiments, block 524 may also include updating the electronic device state in the data and model store 120 based on the received current state. For example, the server system 110 may update the electronic device 128 status in the data and model store 120 to "locked".

In some embodiments, process 500 may also include generating a notification associated with the current state of one or more of the electronic devices in response to determining that the predetermined condition has been met. For example, in response to the location of the user device 102 leaving a predetermined area (eg, the area corresponding to the user's home) when the one or more electronic devices are in a predetermined state (eg, the "front door" is unlocked), via User device 102 presents a notification to the user indicating that the user forgot to lock the door. Other similar notifications may be generated in response to other predetermined conditions.

Using process 500, a virtual assistant implemented by a user device can receive natural language commands to set a state or query any number of electronic devices. The natural language commands may refer to the electronic device in any desired manner and need not include a unique identifier of the electronic device or the type of the electronic device.

Remote control of electronic equipment

6 illustrates an example process 600 for remotely controlling an electronic device using a virtual assistant. In some embodiments, process 600 may be similar to process 500 except that process 600 may be performed using a system similar to or the same as system 200 shown in FIG. 2 . For example, process 600 may be performed by a user device (eg, user device 102 ) that is located remotely from controlled electronic devices (eg, electronic devices 128 , 130 , and 132 ), and a second user device (eg, second user device 134 ) Alternatively used to control electronics. Accordingly, portions of process 600 may be performed by each of user equipment 102 , server system 110 , and second user equipment 134 . Specifically, the blocks on the left side of FIG. 6 may be performed by the user equipment 102 , the blocks in the middle of FIG. 6 may be performed by the server system 110 , and the blocks on the right side of FIG. 6 may be performed by the second user equipment 134 .

Blocks of process 600 may be similar or identical to like-numbered blocks of process 500, except that blocks 516, 518, 520, and 522 of process 600 may instead be performed by a second user equipment (eg, second user equipment 134) implement. Additionally, as a result, the identification of each of the one or more electronic devices and the command to be executed by the electronic device may alternatively be transmitted to the second user device at block 514, and alternatively, at block 524 The current state of the electronic device is received from the second user device.

In some embodiments, the process 600 may also include transmitting, by the one or more servers, the current state of the electronic device received at block 524 to the user device that received the audio input at block 502 . Additionally or alternatively, the process may include transmitting, at block 518, an indication that the command transmitted to the electronic device was successful. For example, visual output or audio output may be generated for informing the user of the success, partial success or failure of execution of a command by the electronic device. A successful determination may depend on the command transmitted to the electronic device and the current state of the device received at block 524 .

Independent control of electronic equipment

FIG. 7 illustrates an exemplary process 700 for controlling an electronic device using a virtual assistant. In some embodiments, process 700 may be similar to process 500 except that process 700 may be performed using a stand-alone user device that may perform the functions of both user device 102 and server system 110 . As a result, all blocks of process 700 may be performed by user equipment (eg, user equipment 102).

Blocks of process 700 may be similar or identical to like-numbered blocks of process 500, except that blocks 508, 510, and 512 may instead be performed by user equipment (eg, user equipment 102). Additionally, as a result, blocks corresponding to functions for communicating between user device 102 and server system 110 need not be performed (eg, blocks 504, 506, 514, 516, 522, and 524).

Configuration of storage electronics

FIG. 8 illustrates an example process 800 for storing the state of a plurality of electronic devices as a configuration using a virtual assistant. The configuration may represent a stored set of states for the plurality of electronic devices, which may be referenced in voice user input to cause the plurality of electronic devices to transition to the states defined in the configuration. For example, a user may create a "sleep" configuration in which the state of all lights is set to off, the state of the thermostat is set to 72°F, the state of all doors is set to locked, and the state of the garage door is set to closed. Thus, when the user is about to go to sleep, the user may provide a command to the user device 102 to reference the stored configuration (eg, "I'm going to sleep"), and the system 100 may set the electronic device based on the state stored in the sleep configuration status. In some embodiments, process 800 may be performed using a system similar to or the same as system 100 shown in FIG. 1 . In these embodiments, the blocks of process 800 may be performed by both user device 102 and server system 110 . Specifically, the blocks on the left side of FIG. 8 may be performed by the user equipment 102 , while the blocks on the right side may be performed by the server system 110 .

At block 802 , audio input including user speech may be received at a user device in a similar or identical manner to block 502 of process 500 . At block 804 , data corresponding to the audio input received at block 802 may be transmitted to one or more servers for processing in a similar or identical manner to block 504 of process 500 .

At block 806 , data corresponding to the audio input transmitted by the user device at block 804 may be received by one or more servers in a similar or identical manner to block 506 of process 500 . At block 808 , speech-to-text conversion may be performed on the data corresponding to the audio input, thereby converting the user's speech to a textual representation of the user's speech in a similar or identical manner to block 508 of process 500 .

At block 810, it may be determined that the textual representation of the user's speech represents the user's intent to store the state of the electronic device of system 100 as a configuration. For example, one or more processing modules 118 of server system 110 may utilize data and model store 120 to determine the user's intent based on natural language input. In some embodiments, this may include parsing the text of words that may be related to storing configurations such as "save", "storage", "name", "hold", "configuration", "scenario" and their synonyms express. For example, if the textual representation includes "store this configuration as sleep," it may be determined at block 810 that the user intends to store the state of the electronic device as a "sleep" configuration. Other textual representations such as "save this scene as work" may similarly result in a determination at block 810 that the user intends to store the state of the electronic device as a "work" configuration.

At block 812, in response to determining that the textual representation of the user's speech represents the user's intent to store the state of the electronic device of system 100 as a configuration, the one or more servers may transmit instructions to the user device to query the state of the electronic device. For example, server system 110 may transmit instructions to user device 102 via network 108 to query the status of electronic devices 128 , 130 and 132 .

At block 814, instructions for querying electronic device status transmitted by one or more servers may be received by the user device. For example, user device 102 may receive instructions from server system 110 via network 108 to query the status of electronic devices 128 , 130 , and 132 .

At block 816, the user equipment may transmit a query for their current status to each electronic device. For example, user device 102 may transmit a command via network 126 to each of electronic devices 128, 130, and 132 instructing them to return to their current state.

At block 818, the user equipment may receive the current state of each electronic device (or the current state of all electronic devices capable of transmitting its current state) in response to the query transmitted at block 516. For example, in response to the query sent to each device at block 816 , the user device 102 may receive the current state of the electronic devices 128 , 130 , and 132 via the network 126 . In some embodiments, block 818 may also include updating the electronic device state in the electronic device data 370 based on the received current state. The current state may also include an error state, such as an undetermined state or an unavailable state of the device.

At block 820, the user equipment may transmit the current state of the electronic device received at block 818 to one or more servers. For example, the user device 102 may transmit the current state of the electronic device received at block 818 to the server system 110 .

At block 822, the one or more servers may receive the current state of the electronic device transmitted by the user equipment at block 820. For example, server system 110 may receive the current state of electronic devices 128 , 130 , and 132 from user device 102 via network 126 .

At block 824, the one or more servers may store the current state of the electronic device received at block 822 as a configuration. In some embodiments, the configuration may be assigned a unique identifier, such as "sleep", "morning", "work", etc., based on the name provided in the textual representation and identified at block 810 . For example, server system 110 may store the current state of electronic devices 128 , 130 , and 132 as "sleep" in model and data store 120 in response to a user speech received at block 802 including "name this configuration sleep" configuration.

Control electronics using stored configuration

9 illustrates an exemplary process 900 of using a virtual assistant to set the state of a plurality of electronic devices using a previously stored configuration (eg, a configuration created using process 800). In some embodiments, process 900 may be performed using a system similar to or the same as system 100 shown in FIG. 1 . In these embodiments, the blocks of process 900 may be performed by both user device 102 and server system 110 . Specifically, the blocks on the left side of FIG. 9 may be performed by the user equipment 102 , while the blocks on the right side may be performed by the server system 110 .

At block 902 , audio input including user speech may be received at a user device in a similar or identical manner to block 502 of process 500 . At block 904 , data corresponding to the audio input received at block 902 may be transmitted to one or more servers for processing in a manner similar or identical to that of block 504 of process 500 .

At block 906 , data corresponding to the audio input transmitted by the user device at block 904 may be received by one or more servers in a similar or identical manner to block 506 of process 500 . At block 908 , speech-to-text conversion may be performed on the user speech of the audio input, thereby converting the user speech to a textual representation of the user speech in a similar or identical manner to block 508 of process 500 .

At block 910, it may be determined that the textual representation of the user's speech represents the user's intent to set the electronic device state of the system 100 based on the stored configuration. For example, one or more processing modules 118 of server system 110 may utilize data and model store 120 to determine the user's intent based on natural language input. In some embodiments, this may include parsing textual representations of words that may be relevant to using the stored configuration, such as associated with the stored configuration "settings," "configurations," "scenarios," and their synonyms, etc. unique identifier for . For example, if the textual representation includes "I'm going to sleep," it may be determined at block 910 that the user intends to set the state of the electronic device based on the "sleep" configuration. Other textual representations such as "night mode", "set to sleep", etc. may similarly result in a determination at block 910 that the user intends to set the state of the electronic device based on the "sleep" configuration.

At block 912 , commands may be transmitted by the one or more servers to the user device to set the electronic device state of the system 100 based on the configuration identified at block 910 . An identification associated with the command may also be transmitted to identify which devices are to execute each command. For example, server system 110 may transmit unique identifiers associated with electronic devices and commands to be executed by those electronic devices to place the electronic devices in the state specified by the stored configuration.

At block 914, the user device may receive a command to set the electronic device state of the system 100 transmitted by the server. For example, the user device 102 may receive the command transmitted by the server system 110 via the network 108 at block 912 .

At block 916, the user equipment may transmit the command received at block 914 to the electronic device associated with the command. For example, user device 102 may transmit commands via network 126 to electronic devices 128, 130, and 132 to place the electronic devices in a state specified by the stored configuration.

In some embodiments, the actual state of the electronic device may differ from the electronic device state stored in user device 102 (eg, memory 350 ) and/or server system 110 (eg, data and model memory 120 ). Thus, in some embodiments and similar to process 500, block 916 may be performed regardless of the state of the electronic device stored in user device 102 and/or server system 110, and without first querying the electronic device (eg, at block 102). 914 and 916) to determine their actual status in order to reduce the amount of time required to issue commands to the electronic device.

At block 918 , the user equipment may receive an updated status of the electronic device after execution by the electronic device of the command transmitted by the user equipment at block 916 . For example, user device 102 may receive an updated state of electronic devices 128, 130, and 132 after electronic devices 128, 130, and 132 execute a command to set their state to the state specified by the stored configuration. In some embodiments, block 920 may also include updating the electronic device state in the electronic device data 370 based on the received current state. For example, the user device 102 may update the electronic device 128 status in the electronic device data 370 to "locked". In some embodiments, similar to block 520, block 918 may also include outputting an audio or visual indication of the result of the command transmitted to the electronic device at block 916 based on the transmitted command and the received current state of the electronic device . The update status may also include error status, such as an undetermined or unavailable status of the device.

At block 920, the user equipment may transmit the electronic device update status received at block 918 to one or more servers. For example, the user device 102 may transmit the electronic device update status received at block 918 to the server system 110 .

At block 922, the one or more servers may receive the electronic device update status transmitted by the user equipment at block 920. For example, at block 920 , the server system 110 may receive the electronic device update status transmitted by the user device 102 . In some embodiments, block 922 may also include updating the electronic device state in the data and model store 120 based on the received updated state.

In some embodiments, in addition or as an alternative, the state of the electronic device may be configured using the stored configuration in response to determining that a predetermined condition has been met. For example, in response to the location of the user device 102 entering a predetermined area (eg, the area corresponding to the user's home), a command may be sent to the electronic device to transition the electronic device to the configuration that is stored (eg, a "home" configuration) ) specified state. Similarly, in response to the location of the user device 102 leaving a predetermined area (eg, the area corresponding to the user's home), it may be possible during a predetermined time window (eg, between 8 am and 9 am on weekdays) ), sending a command to the electronic device to cause the electronic device to transition to a state specified by another stored configuration (eg, a "working" configuration). Other similar predetermined conditions may be created to cause the system 100 to configure the electronic device based on the stored configuration.

Electronic equipment

According to some embodiments, Figure 10 shows a functional block diagram of an electronic device 1000 configured in accordance with the principles of various described embodiments. The functional blocks of the apparatus may be implemented by hardware, software or a combination of hardware and software implementing the principles of the various described embodiments. Those skilled in the art will understand that the functional blocks described in Figure 10 may be combined or separated into sub-blocks in order to implement the principles of the various described embodiments. Accordingly, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein.

As shown in FIG. 10, the electronic device 1000 may include a touch screen display unit 1002 configured to display a user interface and receive touch input, and a sound receiving unit 1004 configured to receive sound input. In some embodiments, the electronic device 1000 may include a speaker unit 1006 configured to generate sound. The electronic device 1000 may also include a processing unit 1008 coupled to the touch screen display unit 1002 and the sound receiving unit 1004 (and optionally to the speaker unit 1006). In some embodiments, processing unit 1008 may include audio input reception unit 1010, audio input transmission unit 1012, identification and command reception unit 1014, identification and command transmission unit 1016, status reception unit 1018, status transmission unit 1020, status update unit 1022 , an indication output unit 1024 , a second identification and command receiving unit 1026 , a second identification and command transmitting unit 1028 , a second status receiving unit 1030 , a second status transmitting unit 1032 and a notification transmitting unit 1034 .

The processing unit 1008 may be configured (eg, using the audio input receiving unit 1010) to receive audio input including the user's speech. The processing unit 1008 may also be configured (eg, using the audio input transmission unit 1012) to transmit data corresponding to the audio input to one or more servers. The processing unit 1008 may also be configured to receive (eg, using the identification and command receiving unit 1014) from one or more servers: an identification of the first electronic device determined by the one or more servers based on data corresponding to the audio input, and a first command to be executed by the first electronic device determined by the one or more servers based on data corresponding to the audio input. The first command may be transmitted to the first electronic device (eg, using the identification and command transmission unit 1016). After transmitting the first command to the first electronic device, the current state of the first electronic device may be received (eg, using the state receiving unit 1018 ) from the first electronic device. The current state of the first electronic device may be transmitted (eg, using the state transmission unit 1020 ) to one or more servers.

In some embodiments, the first electronic device includes a light bulb. In other embodiments, the first command includes a command to set the light on/off state, dimmable state or color state. In other embodiments, the current state of the first electronic device includes the on/off state, dimmable state of the light bulb after transmitting the first command to set the on/off state, dimmable state or color state of the light bulb state or color state.

In some embodiments, the first electronic device includes a switch. In other embodiments, the first command includes a command to set the on/off state of the switch. In other embodiments, the current state of the first electronic device includes the on/off state of the switch after transmitting the first command to set the on/off state of the switch.

In some embodiments, the first electronic device includes a power outlet. In other embodiments, the first command includes a command to set the active/passive state of the electrical outlet. In other embodiments, the current state of the first electronic device includes the active/passive state of the electrical outlet after transmitting the command to set the active/passive state of the electrical outlet.

In some embodiments, the first electronic device includes a door lock. In other embodiments, the first command includes a command to set the locked/unlocked state of the door lock. In other embodiments, the current state of the first electronic device includes the locked/unlocked state of the door lock after transmitting the command to set the locked/unlocked state of the door lock.

In some embodiments, the first electronic device includes a garage door. In other embodiments, the first command includes a command to set the open/closed state of the garage door. In other embodiments, the current state of the first electronic device includes the open/closed state of the garage door after transmitting the command to set the open/closed state of the garage door.

In some embodiments, the first electronic device includes a thermostat. In other embodiments, the first command includes a command for setting a temperature setpoint value for the thermostat. In other embodiments, the current state of the first electronic device includes the temperature setpoint value of the thermostat following the transmission of the first command to set the temperature setpoint value of the thermostat.

In some embodiments, the first command includes a query for the current state of the first electronic device.

In some embodiments, the first command may be transmitted (eg, using identification and command transmission unit 1016) to the first electronic device over a local wireless network.

In some embodiments, the first command may be transmitted directly to the first electronic device via Bluetooth, line-of-sight, peer-to-peer, or WiFi communication.

In some embodiments, the processing unit 1008 may be configured not to include querying the first electronic device for the first electronic device between receiving the first command from the one or more servers and transmitting the first command to the first electronic device Status of the device.

In some embodiments, the electronic device 1000 may further include a database unit 1034 for storing a state of each electronic device of a plurality of electronic devices including the first electronic device. In other embodiments, the processing unit 1008 may also be configured to update the data stored in the database unit 1034 based at least in part on the current state of the first electronic device received from the first electronic device (eg, using the state update unit 1022 ). the state of the first electronic device.

In some embodiments, the processing unit 1008 may also be configured to output a result of the first command based on the first command and the current state of the first electronic device received from the first electronic device (eg, using the indication output unit 1024 ). instruct.

In some embodiments, electronic device 1000 may include a mobile phone, desktop computer, laptop computer, tablet computer, portable media player, television, television set-top box, or wearable electronic device.

In some embodiments, the processing unit 1008 may also be configured to receive (eg, using the second identification and command receiving unit 1026 ) from one or more servers: determined by the one or more servers based on data corresponding to the audio input an identification of the second electronic device, and a second command to be executed by the second electronic device determined by the one or more servers based on data corresponding to the audio input. The processing unit 1008 may also be configured to transmit the second command (eg, using the second identification and command transmission unit 1028) to the second electronic device, and after transmitting the second command to the second electronic device, (eg, using the first The two-state receiving unit 1030) receives the current state of the second electronic device from the second electronic device. The processing unit 1008 may also be configured to transmit the current state of the second electronic device (eg, using the second state transmission unit 1032) to the one or more servers.

In some embodiments, processing unit 1008 may also be configured to transmit a notification associated with the current state of the first electronic device (eg, using notification transmission unit 1036) in response to determining that a predetermined condition has been met.

According to some embodiments, Figure 11 shows a functional block diagram of an electronic device 1100 configured in accordance with the principles of various described embodiments. The functional blocks of the apparatus may be implemented by hardware, software or a combination of hardware and software implementing the principles of the various described embodiments. Those skilled in the art will understand that the functional blocks described in Figure 11 may be combined or separated into sub-blocks in order to implement the principles of the various described embodiments. Accordingly, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein.

As shown in FIG. 11 , the electronic device 1100 may include a touch screen display unit 1102 configured to display a user interface and receive touch input, and a sound receiving unit 1104 configured to receive sound input. In some embodiments, the electronic device 1100 may include a speaker unit 1106 configured to generate sound. The electronic device 1100 may also include a processing unit 1108 coupled to the touch screen display unit 1102 and the sound receiving unit 1104 (and optionally to the speaker unit 1106). In some embodiments, processing unit 1108 may include audio input reception unit 1110 , audio input transmission unit 1112 , instruction reception unit 1114 , query transmission unit 1116 , status reception unit 1118 , and status transmission unit 1120 .

Processing unit 1108 may be configured (eg, using audio input receiving unit 1110) to receive audio input including user speech. The processing unit 1108 may also be configured to transmit data corresponding to the audio input (eg, using the audio input transmission unit 1112) to one or more servers. The processing unit 1108 may also be configured to receive (eg, using the instruction receiving unit 1114 ) instructions from one or more servers for querying the status of each of the plurality of electronic devices, wherein the status of each electronic device is determined by The one or more servers are determined based on data corresponding to the audio input. The status query may be transmitted (eg, using the query transmission unit 1116 ) to each of the plurality of electronic devices, and may receive a status query from the plurality of electronic devices (eg, using the status receiving unit 1118 ). The current state of each electronic device. The current state of each of the plurality of electronic devices may be transmitted (eg, using the state transmission unit 1120 ) to one or more servers for storage as a configuration.

According to some embodiments, Figure 12 shows a functional block diagram of an electronic device 1200 configured in accordance with the principles of various described embodiments. The functional blocks of the apparatus may be implemented by hardware, software or a combination of hardware and software implementing the principles of the various described embodiments. Those skilled in the art will understand that the functional blocks described in Figure 12 may be combined or separated into sub-blocks in order to implement the principles of the various described embodiments. Accordingly, the description herein optionally supports any possible combination or separation or further definition of the functional blocks described herein.

As shown in FIG. 12, the electronic device 1200 may include a touch screen display unit 1202 configured to display a user interface and receive touch input, and a sound receiving unit 1204 configured to receive sound input. In some embodiments, the electronic device 1200 may include a speaker unit 1206 configured to generate sound. The electronic device 1200 may also include a processing unit 1208 coupled to the touch screen display unit 1202 and the sound receiving unit 1204 (and optionally to the speaker unit 1206). In some embodiments, processing unit 1208 may include audio input reception unit 1210 , audio input transmission unit 1212 , instruction reception unit 1214 , query transmission unit 1216 , status reception unit 1218 , and status transmission unit 1220 .

In some embodiments, processing unit 1208 may also be configured to receive audio input including user speech (eg, using audio input receiving unit 1210), and (eg, using audio input transmitting unit 1212) to transmit data corresponding to the audio input to one or more servers, and (eg, using command receiving unit 1214) from one or more servers for setting up each electronic device of a plurality of electronic devices determined by the one or more servers based on the stored configuration status of multiple commands. Processing unit 1208 may also be configured (eg, using command transmission unit 1216) to transmit multiple commands to multiple electronic devices. The processing unit 1208 may also be configured to receive (eg, using the status receiving unit 1218 ) an updated status of each of the plurality of electronic devices from the plurality of electronic devices, and (eg, using the status transmitting unit 1220 ) to transfer the plurality of electronic devices. The updated status of each of the electronic devices is transmitted to one or more servers.

As shown in FIG. 13, the electronic device 1300 may include a touch screen display unit 1302 configured to display a user interface and receive touch input, and a sound receiving unit 1304 configured to receive sound input. In some embodiments, the electronic device 1300 may include a speaker unit 1306 configured to generate sound. The electronic device 1300 may also include a processing unit 1308 coupled to the touch screen display unit 1302 and the sound receiving unit 1304 (and optionally to the speaker unit 1306). In some embodiments, processing unit 1308 may include audio input reception unit 1310, speech-to-text conversion unit 1312, electronic device recognition unit 1314, command recognition unit 1316, identification and command transmission unit 1318, status reception unit 1320, status update unit 1322 , a context information receiving unit 1324 , an instruction transmission unit 1326 , a second electronic device identification unit 1328 , a second command identification unit 1330 , a second identification and command transmission unit 1332 , and a second status receiving unit 1336 .

Processing unit 1308 may be configured (eg, using audio input receiving unit 1310) to receive data corresponding to audio input including user speech. Processing unit 1308 may also be configured (eg, using speech-to-text conversion unit 1312) to perform speech-to-text conversion on data corresponding to the audio input to generate a textual representation of the user's speech. The processing unit 1308 may also be configured (eg, using the electronic device identification unit 1314) to identify the first electronic device based on the textual representation of the user's speech. The processing unit 1308 may also be configured (eg, using the command recognition unit 1316) to recognize a first command to be executed by the first electronic device based on the textual representation of the user's speech. The processing unit 1308 may also be configured to transmit the identification of the first electronic device and the first command to the user equipment (eg, using the identification and command transmission unit 1318). The processing unit 1308 may also be configured (eg, using the state receiving unit 1320) to receive the current state of the first electronic device.

In some embodiments, the first electronic device includes a light bulb. In other embodiments, the first command includes a command to set the light on/off state, dimmable state or color state. In other embodiments, the current state of the first electronic device includes the on/off state, dimmable state of the light bulb after transmitting the first command to set the on/off state, dimmable state or color state of the light bulb state or color state.

In some embodiments, the first electronic device includes an electrical outlet. In other embodiments, the first command includes a command to set the active/passive state of the electrical outlet. In other embodiments, the current state of the first electronic device includes the active/passive state of the electrical outlet after transmitting the command to set the active/passive state of the electrical outlet.

In some embodiments, the first electronic device includes a switch. In other embodiments, the first command includes a command to set the on/off state of the switch. In other embodiments, the current state of the first electronic device includes the on/off state of the electrical outlet after transmitting the command to set the on/off state of the switch.

In some embodiments, the first electronic device includes a door lock. In other embodiments, the first command includes a command to set the locked/unlocked state of the door lock. In other embodiments, the current state of the first electronic device includes the locked/unlocked state of the door lock after transmitting the command to set the locked/unlocked state of the door lock.

In some embodiments, the first electronic device includes a garage door. In other embodiments, the first command includes a command to set the open/closed state of the garage door. In other embodiments, the current state of the first electronic device includes the open/closed state of the garage door after transmitting the command to set the open/closed state of the garage door.

In some embodiments, the first electronic device includes a thermostat. In other embodiments, the first command includes a command for setting a temperature setpoint value for the thermostat. In other embodiments, the current state of the first electronic device includes the temperature setpoint value of the thermostat following the transmission of the first command to set the temperature setpoint value of the thermostat.

In some embodiments, the first command includes a query for the current state of the first electronic device.

In some embodiments, the electronic device 1300 also includes a database unit 1334 for storing the name, type and status of each electronic device of the plurality of electronic devices including the first electronic device. In other embodiments, the processing unit 1308 may also be configured to update the state of the first electronic device stored in the database unit 1334 based on the current state of the first electronic device (eg, using the state update unit 1322).

In some embodiments, the processing unit 1308 may also be configured to receive (eg, using the context information receiving unit 1324) context information within the data corresponding to the audio input from the user equipment. In other embodiments, the processing unit 1308 may also be configured (eg, using the electronic device identification unit 1314) to identify the first electronic device based on the contextual information. In other embodiments, processing unit 1308 may also be configured (eg, using command identification unit 1316) to identify the first command based on the context information. In some embodiments, the contextual information includes the orientation of the user equipment when the user equipment received the audio input or the location of the user equipment when the user equipment received the audio input.

In some embodiments, the textual representation of the user's speech does not include the name of the first electronic device. In other embodiments, the textual representation of the user's speech does not include the type of the first electronic device.

In some embodiments, electronic device 1300 includes a mobile phone, desktop computer, laptop computer, tablet computer, portable media player, television, television set-top box, or wearable electronic device.

In some embodiments, the processing unit 1308 may also be configured to transmit the first command and the identity of the first electronic device to the user equipment (eg, using the identification and command transmission unit 1318), and (eg, using the status receiving unit 1320) The current state of the first electronic device is received from the user device.

In other embodiments, the processing unit 1308 may also be configured to transmit the first command and the identity of the first electronic device to the second user equipment (eg, using the identification and command transmission unit 1318), and (eg, using the status receiving unit) 1320) Receive the current state of the first electronic device from the second user equipment.

In other embodiments, the processing unit 1308 may also be configured to transmit an indication of the result of the first command to the second user equipment (eg, using the indication transmission unit 1326) based on the first command and the current state of the first electronic device.

In other embodiments, the processing unit 1308 may also be configured (eg, using the second electronic device recognition unit 1328) to recognize the second electronic device based on the textual representation of the user's speech, (eg, using the second command recognition unit 1330) based on the The textual representation of the user's speech identifies the second command to be executed by the second electronic device, transmits (eg, using the second identification and command transmission unit 1332 ) the identification of the second electronic device and the second command, and (eg, using the second identification and command transmission unit 1332 ) The state receiving unit 1336) receives the current state of the second electronic device.

As shown in FIG. 14, the electronic device 1400 may include a touch screen display unit 1402 configured to display a user interface and receive touch input, and a sound receiving unit 1404 configured to receive sound input. In some embodiments, the electronic device 1400 may include a speaker unit 1406 configured to generate sound. The electronic device 1400 may also include a processing unit 1408 coupled to the touch screen display unit 1402 and the sound receiving unit 1404 (and optionally to the speaker unit 1406). In some embodiments, the processing unit 1408 may include an audio input receiving unit 1410, a speech-to-text conversion unit 1412, a determination unit 1414, an instruction transmission unit 1416, a status receiving unit 1418, a configuration storage unit 1420, a second audio input receiving unit 1422 , a second speech-to-text conversion unit 1424 , a second determination unit 1426 , a command transmission unit 1428 , and a second status reception unit 1430 .

The processing unit 1408 may be configured (eg, using the audio input receiving unit 1410) to receive data from the user device corresponding to the audio input including the user's speech. Processing unit 1408 may also be configured (eg, using speech-to-text conversion unit 1412) to perform speech-to-text conversion on data corresponding to the audio input to generate a textual representation of the user's speech. Processing unit 1408 may also be configured (eg, using determination unit 1414) to determine that the textual representation of the user's speech represents the user's intent to store the state of each of the plurality of electronic devices as configured. Processing unit 1408 may also be configured (eg, using instruction transmission unit 1416) to transmit instructions for querying the status of each of the plurality of electronic devices. The processing unit 1408 may also be configured (eg, using the status receiving unit 1418) to receive the current status of each of the plurality of electronic devices. Processing unit 1408 may also be configured (eg, using configuration storage unit 1420) to store the received current state of each electronic device of the plurality of electronic devices as a configuration.

As shown in FIG. 15, the electronic device 1500 may include a touch screen display unit 1502 configured to display a user interface and receive touch input, and a sound receiving unit 1504 configured to receive sound input. In some embodiments, the electronic device 1500 may include a speaker unit 1506 configured to generate sound. The electronic device 1500 may also include a processing unit 1508 coupled to the touch screen display unit 1502 and the sound receiving unit 1504 (and optionally to the speaker unit 1506). In some embodiments, the processing unit 1408 may include an audio input receiving unit 1510, a speech-to-text conversion unit 1512, a determination unit 1514, an instruction transmission unit 1516, a status receiving unit 1518, a configuration storage unit 1520, a second audio input receiving unit 1522 , a second speech-to-text conversion unit 1524 , a second determination unit 1526 , a command transmission unit 1528 , and a second status reception unit 1530 .

In some embodiments, processing unit 1508 may also be configured to receive (eg, using audio input receiving unit 1510 ) data corresponding to audio input including the user's speech, (eg, using speech-to-text conversion unit 1512 ) pairing with the audio Input corresponding data to perform a speech-to-text conversion to generate a textual representation of the user's speech, determining (eg, using determining unit 1514) that the textual representation of the user's speech represents the user's desire to change the configuration of each of the plurality of electronic devices based on the configuration. Intent of the state, to transmit a plurality of commands (eg, using the command transmission unit 1516 ) to set the state of each of the plurality of electronic devices based on the configuration, and to receive the plurality of electronic devices (eg, using the state receiving unit 1518 ) The updated status of each electronic device in .

As mentioned above, one aspect of the present technology resides in the collection and use of data from various sources to improve the delivery of heuristics or any other content to the user that may be of interest. This disclosure contemplates that, in some instances, these collected data may include personal information data that uniquely identifies or may be used to contact or locate a particular person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, home addresses, or any other identifying information.

This disclosure recognizes that the use of such personal information data in the present technology can be used to benefit users. For example, the personal information data may be used to deliver targeted content of greater interest to the user. The use of such personal information data thus enables programmatic control over the content delivered. In addition, this disclosure also contemplates other uses of personal information data that benefit the user.

This disclosure also anticipates that entities responsible for the collection, analysis, disclosure, transfer, storage or other use of such personal information data will comply with established privacy policies and/or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data. For example, personal information from users should be collected for the entity's lawful and reasonable uses and not shared or sold outside of those lawful uses. In addition, such collection should only take place after the user's informed consent. In addition, such entities shall take any necessary steps to safeguard and protect access to such personal information data and to ensure that others who have access to personal information data comply with their privacy policies and procedures. Additionally, such entities may subject themselves to third-party assessments to demonstrate compliance with generally accepted privacy policies and practices.

Regardless of the foregoing, the present disclosure also contemplates embodiments in which users selectively block the use or access to personal information data. That is, the present disclosure contemplates that hardware elements and/or software elements may be provided to prevent or prevent access to such personal information data. For example, with respect to advertising delivery services, the techniques of the present invention may be configured to allow users to opt-in or opt-out of participation in the collection of personal information data during registration for the service. As another example, the user may choose not to provide location information for the target content delivery service. As another example, the user may choose not to provide precise location information, but permit the transmission of location area information.

Thus, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, this disclosure also contemplates that the various embodiments may also be used without access to such personal information data. accomplish. That is, various embodiments of the present technology do not fail due to lack of all or a portion of such personal information data. For example, preferences may be inferred based on non-personal information data or absolute minimum personal information (eg, content requested by a device associated with a user, other non-personal information available to content delivery services, or publicly available information), thereby Content is selected and delivered to the user.

Although the embodiments have been fully described with reference to the accompanying drawings, it is to be noted that various changes and modifications will become apparent to those skilled in the art. It should be understood that such changes and modifications are considered to be included within the scope of the various embodiments as defined by the appended claims.

Claims (28)

1. A user equipment comprising: one or more processors; memory; and One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for: receiving audio input representing a user request for the electronic device to perform an action, wherein the audio input is in natural language, and wherein the audio input is ambiguous with respect to defining an electronic device for performing the action; determining one or more possible electronic device characteristics corresponding to the audio input; retrieving a data structure representing a collection of electronic devices at a predetermined location; determining one or more candidate electronic devices from the set of electronic devices based on the data structure, the one or more candidate electronic devices corresponding to the audio input; determining a user intent corresponding to the audio input based on the one or more possible electronic device characteristics and one or more actual electronic device characteristics of the one or more candidate electronic devices; and A command is provided that causes an electronic device of the one or more candidate electronic devices to perform the action corresponding to the user intent. 2. The user device of claim 1, wherein the audio input is in natural language, and wherein the audio input is ambiguous with respect to defining the action. 3. The user equipment of claim 1, wherein the one or more possible electronic device characteristics include one or more words that are associated with an electronic device that the user equipment can control is associated with, or is associated with, one or more states corresponding to the electronic device capable of being controlled by the user device. 4. The user equipment of claim 1, wherein the data structure defines one or more actual device characteristics associated with each electronic device in the set of electronic devices. 5. The user equipment of claim 4, wherein the one or more actual electronic device characteristics comprise one or more words associated with each electronic device in the set of user equipment A device is associated or associated with one or more states corresponding to each electronic device in the set of user devices. 6. The user equipment of claim 1, wherein the data structure defines a current state of each electronic device in the set of electronic devices. 7. The user equipment of claim 6, wherein the one or more programs further comprise instructions for: The user intent is determined further based on a current state of each of the one or more candidate electronic devices. 8. The electronic device of claim 6, wherein the one or more programs further comprise instructions for: receiving an updated status from the electronic device after providing the command; and The current state of the electronic device is updated based on the updated state received from the electronic device. 9. The user equipment of claim 8, wherein the one or more programs further comprise instructions for: An indication of a result of the command is output based on the updated state of the electronic device received from the electronic device. 10. The user equipment of claim 9, wherein the indication of the result of the command is an audio indication, a visual indication, or both an audio indication and a visual indication. 11. The user equipment of claim 1, wherein the one or more programs further comprise instructions for: determining a location corresponding to the audio input; and Based on the determined location, the predetermined location is determined from a plurality of predetermined locations. 12. The user equipment of claim 1, wherein determining the one or more candidate electronic devices from the set of electronic devices comprises: One or more alternative terms associated with terms in the audio input are determined, wherein the one or more candidate electronic devices are determined based on the one or more alternative terms. 13. The user equipment of claim 1, wherein determining the user intent further comprises: determining the electronic device from the one or more candidate electronic devices based on the one or more possible electronic device characteristics and the one or more actual electronic device characteristics of the one or more candidate electronic devices . 14. The user equipment of claim 1, wherein the one or more programs further comprise instructions for: A relative location of the user equipment relative to an area within the predetermined location is determined, wherein the user intent is determined based on the relative location. 15. A method for controlling an electronic device using a virtual assistant on a user device having one or more processors and memory, the method comprising: At the user equipment: receiving audio input representing a user request for the electronic device to perform an action, wherein the audio input is in natural language, and wherein the audio input is ambiguous with respect to defining an electronic device for performing the action; determining one or more possible electronic device characteristics corresponding to the audio input; retrieving a data structure representing a collection of electronic devices at a predetermined location; determining one or more candidate electronic devices from the set of electronic devices based on the data structure, the one or more candidate electronic devices corresponding to the audio input; determining a user intent corresponding to the audio input based on the one or more possible electronic device characteristics and one or more actual electronic device characteristics of the one or more candidate electronic devices; and A command is provided that causes an electronic device of the one or more candidate electronic devices to perform the action corresponding to the user intent. 16. The method of claim 15, wherein the audio input is in natural language, and wherein the audio input is ambiguous with respect to defining the action. 17. The method of claim 15, wherein the data structure defines one or more actual device characteristics associated with each electronic device in the set of electronic devices. 18. The method of claim 17, wherein the one or more actual electronic device characteristics include one or more words associated with each electronic device in the set of user devices is associated with, or is associated with, one or more states corresponding to each electronic device in the set of user devices. 19. The method of claim 15, further comprising: determining a location corresponding to the audio input; and Based on the determined location, the predetermined location is determined from a plurality of predetermined locations. 20. The method of claim 15, wherein determining the one or more candidate electronic devices from the set of electronic devices comprises: One or more alternative terms associated with terms in the audio input are determined, wherein the one or more candidate electronic devices are determined based on the one or more alternative terms. 21. The method of claim 15, further comprising: A relative location of the user equipment relative to an area within the predetermined location is determined, wherein the user intent is determined based on the relative location. 22. A non-transitory computer-readable storage medium comprising instructions for one or more programs for execution by one or more processors of a user equipment, the one or more programs A program includes instructions that, when executed by the one or more processors, cause the user equipment to perform a method of: receiving audio input representing a user request for the electronic device to perform an action, wherein the audio input is in natural language, and wherein the audio input is ambiguous with respect to defining an electronic device for performing the action; determining one or more possible electronic device characteristics corresponding to the audio input; retrieving a data structure representing a collection of electronic devices at a predetermined location; determining one or more candidate electronic devices from the set of electronic devices based on the data structure, the one or more candidate electronic devices corresponding to the audio input; determining a user intent corresponding to the audio input based on the one or more possible electronic device characteristics and one or more actual electronic device characteristics of the one or more candidate electronic devices; and A command is provided that causes an electronic device of the one or more candidate electronic devices to perform the action corresponding to the user intent. 23. The non-transitory computer-readable storage medium of claim 22, wherein the audio input is in natural language, and wherein the audio input is ambiguous with respect to defining the action. 24. The non-transitory computer-readable storage medium of claim 22, wherein the data structure defines one or more actual device characteristics associated with each electronic device in the collection of electronic devices. 25. The non-transitory computer-readable storage medium of claim 24, wherein the one or more actual electronic device characteristics include one or more words that are associated with the user device Each electronic device in the set is associated or associated with one or more states corresponding to each electronic device in the set of user equipment. 26. The non-transitory computer-readable storage medium of claim 22, wherein the one or more programs further comprise instructions for: determining a location corresponding to the audio input; and Based on the determined location, the predetermined location is determined from a plurality of predetermined locations. 27. The non-transitory computer-readable storage medium of claim 22, wherein determining the one or more candidate electronic devices from the set of electronic devices comprises: One or more alternative terms associated with terms in the audio input are determined, wherein the one or more candidate electronic devices are determined based on the one or more alternative terms. 28. The non-transitory computer-readable storage medium of claim 22, wherein the one or more programs further comprise instructions for: A relative location of the user equipment relative to an area within the predetermined location is determined, wherein the user intent is determined based on the relative location.

Images