CN106999257A - Operating room black box device, system, method and computer-readable medium - Google Patents

Abstract

A kind of multichannel logging recorder/encoder, for collecting, integrated, synchronization and record receive medical treatment from multiple hardware cells or surgical operation data are used as independent live or real-time stream.Medical treatment or surgical operation data are relevant with live or real time medical process.Exemplary hardware unit includes control interface, camera, sensor, audio frequency apparatus and patient monitoring hardware.Other examples system can include the platform based on cloud for being incorporated to encoder.

Description

Operating room black box device, system, method and computer readable medium

Cross References to Related Applications

This application claims priority to US Provisional Application No. 62/054,057, filed September 23, 2014, and US Provisional Application No. 62/138,647, filed March 26, 2015, the entire contents of which are incorporated herein by reference.

technical field

Embodiments described herein generally relate to the field of medical devices, systems and methods, and more particularly, to a medical or surgical black-box device, system, method and computer readable medium.

Background technique

Previous attempts to enable data collection in live operating room (OR) settings or patient intervention areas may have been unsuccessful. Example reasons might include: (1.) Not comprehensive. Previous attempts have included a very limited number of inputs, which may have resulted in failure to identify chains of events leading to adverse outcomes, and/or failure to verify delivery of quality improvement benefits. (2.) Not synchronized. Previous attempts have not achieved synchronization of recording of multiple video-audio feeds. (3.) Strict data analysis methods are not applicable. Previously tried using isolation metrics. These attempts fail to analyze multiple aspects of surgery simultaneously—for example, technical performance, nontechnical skills, human factors, workflow, occupational safety, communication, etc. And (4.) The value of the analysis may not be fully demonstrated. These are only examples and there may be other disadvantages of previous approaches.

Description of drawings

In the attached picture,

Figure 1 illustrates a schematic diagram of an architectural platform according to some embodiments.

Figure 2 illustrates a schematic diagram of a multi-channel recording device or encoder according to some embodiments.

FIG. 3 illustrates a schematic diagram of an example wide-angle video camera, according to some embodiments.

FIG. 4 illustrates a schematic diagram of an example microphone, according to some embodiments.

Figure 5 illustrates a schematic diagram of an example distribution amplifier and converter in accordance with some embodiments.

Figure 6 illustrates a schematic diagram of an example central signal processor in accordance with some embodiments.

Figure 7 illustrates a schematic diagram of an example touch screen monitor in accordance with some embodiments.

Figure 8 illustrates a schematic diagram of an example view according to some embodiments.

Figure 9 illustrates a schematic diagram of a polar pattern according to some embodiments.

Figure 10 illustrates a schematic diagram of an example network, according to some embodiments.

Figure 11 illustrates a schematic diagram of an example encoder, according to some embodiments.

Figure 12 illustrates a flowchart of an example method according to some embodiments.

Figure 13 illustrates a schematic diagram of an example interface, according to some embodiments.

Figure 14 illustrates a schematic diagram of an example system, according to some embodiments.

Figure 15 illustrates a schematic diagram of an example view according to some embodiments.

Figure 16 illustrates a schematic diagram of a black box recording device according to some embodiments.

detailed description

To illustrate various embodiments, reference is made to components, architectures, descriptions and definitions. Embodiments may provide a system, method, platform, device, and/or computer readable medium that provides a surgical operating room (OR), intensive care unit, trauma room, emergency department, interventional suite, endoscopy suite, obstetrics suite and/or comprehensive data collection of patient care details in medical or surgical wards, outpatient medical facilities, clinical sites, or healthcare training facilities (simulation centres). These various example environments or settings may be referred to herein as surgical or clinical settings.

Embodiments described herein may provide devices, systems, methods, platforms, and/or computer-readable media that provide comprehensive data collection of all details of patient care in one or more such settings to: identify and/or or analyze errors, adverse events, and/or adverse outcomes; provide comprehensive data that allow investigation of the chain of events from error to adverse event; provide information on individual performance and/or team performance, e.g., for health care professionals' competency, High-stakes assessments for certification and/or recertification; provide data to design individualized training interventions for surgical and/or medical teams based on demonstrated performance deficits; identify critical safety deficiencies in human performance and/or safety processes, For example, to create personalized solutions designed to reduce risk and/or enhance patient safety; and/or to assess critical safety gaps in medical technology, and/or to provide feedback on design and/or performance improvements, analysis and monitoring Efficiency and safety processes in clinical settings.

In one aspect, embodiments described herein relate to a system for collecting and processing medical or surgical data. The system may have multiple hardware units for collecting real-time medical or surgical data streams with control interfaces coupled to cameras, sensors, audio devices and patient monitoring hardware via a network, real-time medical or surgical data streams In relation to real-time medical procedures within the surgical field or clinical field. A hardware unit may acquire or collect one or more independent data streams from different devices, and each data stream provided by a hardware unit may be independent of other data streams provided by other hardware units. Thus, the system can implement synchronization techniques for data streams as described herein. The system may have device middleware and hardware for translating, concatenating and formatting real-time medical or surgical data streams received independently from hardware units (which in turn may independently receive data feeds from different devices).

The system may have an encoder with a web server for synchronizing and recording real-time medical or surgical data streams to a common clock or timeline to generate session container files. As noted, this synchronization can aggregate independent data feeds in a consistent manner to generate a composite data feed generated from data from multiple independent devices.

The system may have a network infrastructure connecting encoders, device middleware and hardware and hardware units, and virtual private network switching or gateway hardware to transfer session container files.

In some example embodiments, the device middleware and hardware use the network infrastructure to establish secure and reliable connections for communicating with the encoders and hardware units.

In some example embodiments, device middleware and hardware use a clock-synchronized network protocol between hardware units to achieve data consistency and precise synchronization of real-time medical or surgical data streams to assist encoders in generating session container files.

In some example embodiments, the encoder, and device middleware and hardware are operable to interface with third party devices to receive additional data feeds as part of the real-time medical or surgical data stream.

In some example embodiments, the system has a central control station accessible using a control interface configured to respond to commands including play/pause, stop session, record session, move to session frame, split display, record status Input controls, including indicators and log files, to control the processing of data streams.

In some example embodiments, the network infrastructure provides increased failover and redundancy for real-time medical or surgical data streams from hardware units.

In some example embodiments, the system has a storage area network for storing data container files of real-time medical or surgical data streams until scheduled transmission.

In some example embodiments, the encoder implements identity anonymization and encryption of medical or surgical data.

In some example embodiments, encoders process real-time medical or surgical data streams to generate measurement metrics related to medical procedures.

In some example embodiments, the real-time medical or surgical data stream is associated with a timeline, wherein the encoder detects an event within the real-time medical or surgical data stream corresponding to a time on the timeline, and a tag of a session container file with the event and a timestamp, which corresponds to the time on the timeline.

In some example embodiments, the system has a smart dashboard interface for annotating and tagging synchronized medical or surgical data streams, where the smart dashboard can implement a viewer with playback viewing for reviewing content and Interface controls for marking up content.

In some example embodiments, the smart dashboard is multidimensional in that the union of all dimensional variables for a medical procedure as represented by a real-time medical or surgical data stream may indicate one or more applicable annotation dictionaries or codes A specific collection of templates.

In some example embodiments, example variables that may be used to determine annotation and markup dictionaries may be: the type of medical procedure being performed, the aspect of the procedure being analyzed, the geographic area/region where the procedure is being performed.

In another aspect, a multi-channel encoder is provided for collecting, integrating, synchronizing and recording streams of medical or surgical data onto a single interface with a common timeline or clock, receiving the data from multiple hardware units. The medical or surgical stream is processed as an independent real-time or live stream by an encoder with a web server that schedules the transfer of the session file container for recording, which processes the medical or surgical stream to generate a Measurement metrics related to medical procedures. This encoder aggregates multiple independent data streams or feeds received from different hardware units or smart devices.

In some example embodiments, the encoder uses a lossless compression operation to generate a single session transfer file as output.

In some example embodiments, the encoder detects the completion of the recording of the data stream and securely encrypts a single transport file.

In some example embodiments, the encoder anonymizes the identity of the medical or surgical data.

In some example embodiments, data streams include audio, video, text, metadata, quantitative, semi-quantitative, and data feeds.

In another aspect, a method for collecting and processing medical or surgical data is provided. The method includes: receiving, at a multi-channel encoder, a plurality of live or real-time independent input feeds from one or more data capture devices located in an operating room or other patient intervention area, the input feeds being related to a live or real-time medical procedure;

The method may include: synchronizing, by an encoder, multiple live independent input feeds onto a single interface with a common timeline or clock; and using a web server to record the synchronized input feeds. The method may include generating, by the encoder, the output session file using the synchronized input feed; and transmitting the output session file using the web server.

In some example embodiments, the method further includes: processing the data stream for identity anonymization.

In some example embodiments, the method further includes routing the data stream to the encoder using the switch router.

In yet another aspect, a cloud-based system for collecting and processing medical or surgical data is provided. The system may have an encoder with a control interface for triggering the collection of real-time medical or surgical data streams through smart devices including cameras, sensors, audio equipment, and patient monitoring hardware in response to receipt of control commands, The medical or surgical data is related to the real-time medical process in the surgical site or clinical site, the encoder is used to authenticate the smart device, and the smart device synchronizes the real-time medical or surgical data by embedding a timestamp marker within the real-time medical or surgical data stream Streams are time-stamped by each smart device via the device clock. The system also has a media management hub server with middleware and hardware for converting, connecting, formatting, and recording real-time medical or surgical data streams to generate session container files on network-accessible storage devices; and a wireless network Infrastructure to provide a secure network connection between encoders, smart devices, and media management hub servers for communicating real-time medical or surgical data streams. The system also has a central content server for storing and distributing session container files and providing a bi-directional communication interface to the media management hub for file delivery handshaking of session container files. The system also has switching or gateway hardware for a virtual private network tunnel for transferring session container files from the media management hub to the central content server. Cloud-based systems can enable anonymous independent smart devices to time-stamp collected data and implement synchronization techniques to aggregate independent data streams and feeds to generate a comprehensive real-time data representation of a medical or surgical procedure or unit.

In some example embodiments, the media management hub server broadcasts clock data to smart devices for use in synchronizing device clocks.

In some example embodiments, the encoder provides a user interface to receive control commands and display a real-time visual representation of the medical or surgical data.

In some example embodiments, the media management hub server aggregates, packs, compresses and encrypts the real-time data streams to generate session container files.

In some example embodiments, the media management hub server manages smart devices based on location, schedule, region, and requirements.

In some example embodiments, the media management hub server receives operational status data from the smart device to generate a management interface with a visual representation of the smart device's operational status data, including online, offline, run capture, and onboard storage .

In some example embodiments, the media management hub server processes the operational status data to detect smart devices operating outside of normal conditions, and in response generates alert notifications for detected smart devices operating outside of normal conditions.

In some example embodiments, the media management hub server implements a device communication interface for smart devices to enable device data transfer handshaking for real-time medical or surgical data streams.

In some example embodiments, the media management hub server authenticates the smart device.

In some example embodiments, the system has a computational intelligence platform for receiving a session container file to build an analytical model to identify clinical factors within the session container file for prognostic, cost, and safety concerns, the analytical model providing for extracting features, correlations, and event behavior from session container files involving multivariate datasets with time-varying parameters.

In some example embodiments, the system has a training or education server for receiving session container files, processing the session container files to identify root causes of adverse patient outcomes, and generating a training interface to use the identified root causes and session containers files to communicate training or performance feedback data.

In some exemplary embodiments, the smart device includes a motion tracking device for markerless motion tracking of objects within the surgical or clinical field, and the system further includes a processor configured to receive all data from the motion tracking device Captured motion data is transformed into data structures that identify artifacts, workflow design, and chains of events.

A platform can have different aspects, including hardware, software, front-end components, middleware components, back-end components, rich content analysis software, and analysis software (eg, databases).

Figure 1 illustrates an architectural platform according to some embodiments. The platform 10 includes various hardware components such as a network communication server 12 (also referred to as a "web server") and a network control interface 14 (including a monitor, keyboard, touch interface, tablet, processor and storage devices, web browsing device) for on-site private network management.

Multiple processors can be configured with operating system and client software (eg, Linux, Unix, Windows Server or equivalent), scheduling software, backup software. Data storage devices can be connected on a storage area network.

FIG. 1 shows a surgical or medical data encoder 22 . The encoder may be referred to herein as a data logger, a "black box" recorder, a "black box" encoder, and the like. Additional details are described herein. Platform 10 may also have physical and logical security to prevent accidental or unauthorized access. A network and signal router 16 connects the components.

The platform 10 includes a hardware unit 20 comprising a collection or set of data capture devices for capturing and generating a medical or surgical data feed for providing to an encoder 22 . The hardware unit 20 may include a camera 30 (e.g., a wide-angle, high-definition, pan and zoom camera, such as a Sony EVI-HD1 or other example camera) installed within a surgical unit, ICU, emergency unit, or clinical intervention unit to A video representation of the captured OR is provided to encoder 22 as a video feed. The video feed may be referred to as medical or surgical data. An example camera 30 is a laparoscopic or procedural view camera (AIDA, Karl Storz or equivalent) resident in a surgical unit, ICU, emergency unit, or clinical intervention unit. Example video hardware includes distributed amplifiers for signal separation of laparoscopic cameras. The hardware unit 20 has an audio device 32 (e.g., a capacitive gooseneck microphone such as the ES935ML6, Audio Technica, or other examples) installed in a surgical unit, ICU, emergency unit, or clinical intervention unit to provide as a medical or surgical procedure An audio feed of another example of data. Example sensors 34 installed or used in a surgical unit, ICU, emergency unit, or clinical intervention unit include, but are not limited to: environmental sensors (e.g., temperature sensors, humidity sensors, humidity sensors, etc.), acoustic sensors (e.g., ambient noise sensors, decibel sensors), electrical sensors (e.g. Hall sensors, magnetic sensors, current sensors, MEMS sensors, capacitive sensors, resistive sensors), flow sensors (e.g. air sensors, fluid sensors, gas sensors), angle/ Position/displacement sensors (eg, gyro sensors, attitude indicator sensors, piezoelectric sensors, photoelectric sensors), and other sensors (eg, strain sensors, liquid level sensors, load cell sensors, motion sensors, pressure sensors). Sensors 34 provide sensor data as another example of medical or surgical data. The hardware unit 20 also includes patient monitoring equipment 36 and a lot of equipment (LOT) 18 .

A customizable control interface 14 and GUI (which may include tablet devices, PDAs, hybrid devices, convertibles, etc.) are available for configuration of the hardware components of the control unit 20 . Platform 10 has middleware and hardware for device-to-device translation and connectivity and synchronization over private VLANs or other networks. Computing devices may be configured with anonymization software, data encryption software, lossless video and data compression software, speech distortion software, transcription software. Network hardware may include cables, such as Ethernet, RJ45, fiber optic, SDI, HDMI, coaxial, DVI, component audio, component video, etc., to support wired connectivity between components. The network hardware may also have wireless base stations to support wireless connectivity between components.

Description and Definitions of Illustrative Embodiments

Illustrative definitions of various components are provided as examples of various embodiments.

Private VLAN may refer to a networking technology that provides network isolation and secure hosting of the network (existing backbone) on the client side via restricted "private ports".

A VPN can extend a private network across a public network such as the Internet. It enables a computer or network-enabled device to send and receive data across a shared or public network as if it were directly connected to a private network, while benefiting from the functionality, security, and management policies of a private network. FIG. 1 shows an example VPN 24 (Virtual Private Network) connected to switch and gateway hardware and, and encoder 22 .

Anonymization software is used to anonymize and protect the identity of all medical professionals, patients, distinguishing objects or characteristics in a medical, clinical or emergency unit. The software implements methods and techniques for detecting faces, distinguishing objects or features in medical, clinical or emergency units, and distorting/blurring images of distinguishing elements. The degree of deformation/blurring is localized frame by frame until the identity is protected without limiting the quality of the analysis.

Voice or vocabulary alteration software is used to anonymize and protect the identity of all medical professionals, patients, distinguishing objects or characteristics in medical, clinical or emergency settings. The software implements methods and techniques, running on hardware in a medical, clinical or emergency setting, to alter speech, dialogue and/or remove phrases of everyday language to preserve the speaker's identity while maintaining the integrity of the input stream to prevent adversely affect the quality of analysis.

Data encryption software can be executed to encrypt computer data in a manner that cannot be recovered without access to the key. Content may be encrypted at the source as individual data streams, or as a composite container file for storage on electronic media (ie, computer, storage system, electronic device) and/or transmission over the Internet 26 . Encryption/decryption keys can be embedded in the container file and accessed via the master key, or delivered separately.

Lossless video and data compression software performs using a class of data compression techniques that allow perfect or near-perfect reconstruction of original data from compressed data.

Device middleware and hardware may be provided for converting, connecting, formatting and synchronizing individual digital data streams from source devices. Platform 10 may include hardware, software, algorithms and methods for communicating directly or indirectly (via routers, wireless base stations) with OR encoder 22 and third party equipment for use in surgical units, ICUs, emergency or other clinical intervention units (open or proprietary) to establish secure and reliable connections and communications.

Hardware and middleware ensure data consistency, formatting and accurate synchronization. Synchronization can be achieved using a networking protocol for clock synchronization between computer systems and electronic devices over a packet-switched network such as NTP or the like.

The hardware unit may include third-party devices (open or proprietary), non-limiting examples of which are _O2Sat monitors, anesthesia monitors, patient monitors, energy devices, smart surgical devices (i.e., smart staplers, smart abdominal mirror instruments), autonomous surgical robots, etc.; hospital patient management systems (i.e., electronic patient records); smart implants; sensors, including but not limited to environmental sensors, i.e., temperature sensors, moisture sensors, humidity sensors, etc.; Sensors, that is, environmental noise sensors, decibel sensors, etc.; electrical sensors, that is, Hall sensors, magnetic sensors, current sensors, MEMS sensors, capacitance sensors, resistance sensors, etc.; flow sensors, that is, air sensors, fluid sensors, Gas sensors, etc.; angle/position/displacement sensors, i.e., gyro sensors, attitude indicator sensors, piezoelectric sensors, photoelectric sensors, etc.; other sensors: strain sensors, liquid level sensors, load cell sensors, motion sensors, pressure sensors, etc. .

Transcription software can utilize technologies such as natural language speech recognition to assist in the conversion of human speech into text transcription.

OR or surgical encoder: An OR or surgical encoder (e.g., encoder 22) may be a multi-channel encoding device that records, integrates, incorporates, and/or synchronizes separate streams of audio, video, and digital data (quantitative, semi- quantitative and qualitative data feeds) into a single digital container. Digital data can be incorporated into the encoder as a metadata stream and originate from an array of potential sensor types and third-party devices (open or proprietary) used in surgical, ICU, emergency or other clinical intervention units. These sensors and devices can be connected through middleware and/or hardware devices that can be used to convert, format and/or synchronize real-time data streams from respected sources.

Customizable control interface and GUI. The control interface (e.g., 14) may include a central control station (non-limiting examples being one or more computers, tablets, PDAs, hybrids and/or convertibles, etc.), which may be located in the clinical unit or at another designated location. customer location. Customizable Control Interface and GUI A customizable Graphical User Interface (GUI) may be included that provides simple user-friendly and functional control of the system.

Example features of the customizable control interface and GUI may include, but are not limited to: a play/pause button, which may enable certain segments of the process not to be recorded. To omit these segments from the recording, the user interface can pause the recording, and restart it if necessary. Pause and play timestamps are recorded in the log file, indicating the exact time of the extraction process; the Stop Session button, which, when selected, closes the file and automatically passes it to the storage area network (SAN); split-screen quadrant display of the video feed , which may provide a visual display of the video in real-time during recording; the visual indicator of recording may be a colored flashing dot that appears on the screen to provide a visual indication to the team where the video and audio feed is being recorded; log files from time to time, which may generate log files (which indicate key points in time), including recording session start and end times, pauses, and replays; password protection, which may refer to use to ensure patient confidentiality and privacy is maintained One or more layers of password protection to protect the interface.

System-level applications may refer to the platform 10, which is designed as a scalable platform ranging from a small single clinical intervention unit to (multiple) large-scale clinical intervention units. When necessary, switch routers can be used in larger scale applications to maximize efficiency and/or deliver increased failover and redundancy capabilities.

sample application

In one aspect, the described embodiments may provide illustrative small-scale applications. As a small single encoder platform, the audio, video and data feeds are connected to the encoder 22 either directly via cables or indirectly via a connected wireless base station.

Using a customizable control interface and GUI, activation of the system can begin recording, collecting and streaming to encoder 22 all available audio, video, sensor and data feeds (which may be referred to as medical and surgical data feeds). It will use all available cameras including installed and laparoscopic cameras, all audio microphones and all available and implemented sensors and third-party devices used in surgical units, ICU, emergency or other clinical intervention units ( open or proprietary). A pause or stop or play command will send a corresponding command to the encoder 22 . The digital data will be formatted, transformed and synchronized through middleware hardware and software and using networking protocols for network synchronization across the network. The digital data will be included in encoder 22 as metadata.

Encoder 22 may be responsible for synchronizing all feeds, encoding them into signaling files using lossless audio/video/data compression software.

After the recording is complete, the container file will be securely encrypted. Encryption/decryption keys can be embedded in the container file and accessed via the master key, or be delivered separately.

The encrypted file may be stored on encoder 22, or on a storage area network, until scheduled for transmission.

A communication server on a dedicated VLAN is responsible for schedule management and automatic file and key transfers. This can be done via a private VLAN on the client environment and carried over a public data line leading back to the background via a virtual private network (VPN) 24 .

As part of the configuration, the communication server may be responsible for backing up data including audio, video, data, encrypted files, etc. using backup software.

The communication server may be responsible for hosting and directing all traffic between the private VLAN and the back office.

In another aspect, embodiments described herein may relate to an encoder configured to host and operate anonymization and voice or vocabulary alteration software for securing identities of medical professionals, patients, distinguishing objects or characteristics. This can be done before compressing, bundling and/or encrypting the collective data or after receiving the transmission and decrypting in the background.

In one aspect, the described embodiments may provide illustrative large-scale applications.

Larger application environments may be required. To maximize efficiency and deliver increased failover and redundancy capabilities, switch routers (eg, router 16 of FIG. 1 ) may be used. In this example, larger application audio, video and data feeds may be connected to switch router 16 by cable or via a connected wireless base station. The purpose of the router is to route audio, video and data feeds to one of the many encoders 22 available on the network. This can provide platform 10 with a more cost-effective implementation, greater spatial coverage, and increased redundancy and failover.

Using the customizable control interface 14 and GUI, the activation signal can trigger or start recording, collection and streaming of all available audio, video and data feeds (from components of the hardware unit 20) via the switching router 16 to multiple available encoders One of the 22 available encoders. For example, data streams or feeds may come from all available cameras including mounted cameras and laparoscopic cameras, all audio microphones, and all available and implemented sensors used in the hardware unit 20 as well as third party devices (open or proprietary), which may involve surgical units, ICUs, emergency or other clinical intervention units. Control commands received at the control interface 14 , such as pause/stop/play commands, may send corresponding control commands to the encoder 22 . Digital data can be formatted, transformed, and synchronized through middleware hardware and software, and using networking protocols that synchronize clocks across the network. The digital data stream may be incorporated into encoder 22 as metadata. Encoder 22 may be responsible for synchronizing and encoding all feeds into signaling files using lossless audio/video/data compression software.

After the recording is complete, the container file can be safely encrypted. Encryption/decryption keys can be embedded in the master file and accessible through the master key, or have separate keys. The encrypted file will be stored on the encoder 22 or on the storage area network until scheduled for transmission.

A communication server on a dedicated VLAN 24 can take care of schedule management and automatic file and key transfers. This can be done through a private VLAN on the client environment and communicated via the VPN 24 on a public data line leading back to the back office or other systems.

As part of the configuration, a communication server (eg, web server 12) may be responsible for backing up data, including audio, video, data, encrypted files, etc., using backup software. For example, a communications server could be responsible for hosting and directing all traffic between private VLANs and back-office systems.

In some examples, Encoder 22 may also be responsible for hosting and operating anonymization and speech/lexical distortion software(s) used to protect all medical, clinical or emergency settings captured in the data stream of hardware unit 20 The identity of a professional, patient, distinguishing object or characteristic. This can be done before compression, packaging and/or encryption, or after decryption in the background system.

In one aspect, embodiments described herein may provide an apparatus, system, method, platform, and/or computer-readable medium that is housed in a clinical setting and that allows individual, team, and/or Collect comprehensive information on every aspect of technical performance and their interaction. The data capture devices may be grouped into one or more hardware units 20 as shown in FIG. 1 .

According to some embodiments, this information may include: video from the process domain; video from the clinical environment; audio; physiological data from the patient; environmental factors from position/displacement sensors and other potential sensors); software data from medical devices used during the intervention; and/or individual data from healthcare providers (e.g., heart rate, blood pressure, skin conductance, motion, and eye tracking, etc. ).

According to some embodiments, this information can then be synchronized (e.g., via encoder 22) and/or used to assess: technical performance of healthcare providers; non-technical performance of clinical team members; patient safety (via registration errors and/or or number of adverse events); occupational safety; workflow; visual and/or noise disturbances; and/or interactions between medical/surgical equipment and/or healthcare professionals, etc.

According to some embodiments, this may be achieved through the use of objective structured assessment tools and questionnaires and/or through input from sensors 34, audio equipment 32, anesthesia equipment, medical/surgical equipment, implants, hospital patient management systems (electronic medical records) ) or other data capture devices of the hardware unit 20 etc. to retrieve one or more continuous data streams.

According to some embodiments, notable "events" may be detected, marked, time-stamped, and/or recorded as points in time on a timeline representing the entire duration of a procedure and/or clinical encounter. A timeline can overlay captured and processed data to mark the data with points in time.

After data processing and analysis is complete, one or more such events (and potentially all events) can be viewed on a single timeline represented in the GUI, for example, to allow evaluators to: (i) identify clusters of events;( ii) analyze correlations between two or more registration parameters (and potentially between all registration parameters); (iii) identify potential factors and/or patterns of events leading to adverse outcomes; A predictive model of one or more key steps of the intervention (which may be referred to herein as a "risk zone") statistically related to errors/adverse events/adverse outcomes; (v) identification of performance outcomes and clinical costs The relationship between. These are non-limiting examples of timelines presented by a GUI representing recorded events that may be used by an assessor.

Analysis of these underlying factors according to some embodiments may allow one or more of: (i) active monitoring of clinical manifestations; and/or (ii) monitoring of performance of healthcare technology/devices; (iii) establishment of educational interventions ( For example, personalized structured feedback (or coaching)), simulation-based crisis scenarios, virtual reality training programs, courses for certification/recertification of healthcare practitioners and institutions; and/or identification of safety/performance of medical equipment/surgical equipment deficiencies, and develop improvements and/or design recommendations for "smart" devices and implants to curb the proportion of risk factors in future procedures and/or ultimately improve patient safety outcomes and clinical costs.

Devices, systems, methods and computer readable media according to some embodiments can combine capture and synchronization, and ensure delivery of video/audio/metadata through rigorous data analysis to achieve/prove certain values. Devices, systems, methods, and computer-readable media according to some embodiments can combine multiple inputs, thereby enabling a complete picture of what occurs in a clinical area to be reproduced in a synchronized manner, enabling analysis and/or correlation of these factors (between factors) and external outcome parameters (clinical and economic). The system can bring together analytical tools and/or processes and use the approach for one or more purposes, examples of which are provided herein.

In addition to developing the data platform 10, some embodiments may also include comprehensive data collection and/or analysis techniques to evaluate multiple aspects of any process. One or more aspects of an embodiment may include recording and analyzing video, audio, and metadata feeds in a synchronized manner. The data platform 10 can be a modular system and is not limited in terms of data feeds - any measurable parameter in the OR/patient intervention area (e.g., via a combination of various ambient acoustic sensors, electrical sensors, flow sensors, angle/position / data captured by displacement sensors and other sensors, wearable technology video/data streams, etc.) can be added to the data platform 10 . One or more aspects of an embodiment may include analyzing data using a validated rating tool that looks at different aspects of a clinical intervention. These areas may include: technical performance, non-technical "team" performance, human factors, patient safety, occupational safety, workflow, audio/visual distractions, etc. Video, audio, and synchronized metadata can be analyzed using manual and/or automated data analysis techniques, which can detect predetermined "events" that can be tagged and/or time-stamped. All marked events can be recorded on a master timeline identifying the duration of the entire process. Statistical models can be used to identify and/or analyze patterns in marked events. Various embodiments may encompass a variety of such statistical models, current and future.

According to some embodiments, all video and audio feeds may be recorded and synchronized for the entire medical procedure. Rating tools designed to measure technical skills and/or non-technical skills during medical procedures may fail to capture institutionally useful data leading to adverse events/outcomes and establish Correlation between presentation and clinical outcome.

According to some embodiments, the measures taken (eg error rate, number of adverse events, individual/team/technical performance parameters) may be collected in a cohesive manner. According to some embodiments, data analysis may establish correlations between all enrollment parameters, as appropriate. Through these correlations, danger areas can be pointed out, high-risk assessment procedures can be developed and/or educational interventions can be designed.

In one aspect, embodiments described herein may provide an apparatus, system, method and/or computer readable medium for recording data comprising multiple Audio/video/metadata feeds (e.g., room cameras, microphones, process videos, patient physiological data, software data from devices used for patient care, via environmental sensors/acoustic sensors/electrical sensors/flow sensors/angle/position/ Metadata captured by displacement sensors and other parameters outlined in this paper). The captured data feeds may be simultaneously processed, synchronized, and recorded using an encoder (eg, encoder 22 of FIG. 1 ). These synchronized video, audio and time-series data can provide a complete overview of the clinical process/patient interaction. At the end of the process, the data can be synchronized, compressed, encrypted, and can be anonymized before being transmitted to a data analysis computing system/centre for evaluation and/or statistical analysis.

The data can be analyzed using an encoder 22 (which can include analysis software and a database), which preserves time synchronization of data captured using multiple evaluation tools/data parameters, and allows for export of the analyzed data into different statistical software . The exported data may be a session container file.

Devices, systems, methods and/or computer readable media according to some embodiments may record video, audio and digital data feeds from clinical areas in a synchronized manner. The platform can be a modular system and is not limited to the example data feeds described. Other data feeds related to medical procedures may also be collected and processed by the platform 10 . For example, any measurable parameter in the OR (e.g., data captured by various ambient acoustic sensors, electrical sensors, flow sensors, angle/position/displacement sensors, and other sensors, wearable technology video/data streams, etc.) added to a data logger (eg, encoder 22 of FIG. 1).

Devices, systems, methods and/or computer readable media according to some embodiments analyze comprehensive simultaneous data using validated rating tools that consider different aspects or measurements of surgical/clinical interventions. These aspects or measures may include: technical surgical performance, non-technical "team" performance, human factors, patient safety, occupational safety, workflow, audio/visual distractions, etc. Video, audio, and/or metadata can be analyzed using manual and/or automated data analysis techniques that can detect specific "events" that can be tagged and time-stamped in the session container file or processed data stream.

A device, system, method and/or computer readable medium according to some embodiments records all marker events on a master timeline representing the entire duration of a procedure/clinical interaction. Statistical models can be used to identify and analyze patterns in labeled events. A master timeline can be associated with processed medical data and session files.

Devices, systems, methods and/or computer-readable media according to some embodiments generate structured performance reports based on captured and processed medical data for identifying and determining individual/ Team/technical performance measurement and organizational deficits.

Devices, systems, methods, and/or computer-readable media according to some embodiments provide a basis for designing targeted educational interventions that address specific safety hazards. These may include personalized training sessions, simulation-based training scenarios, virtual reality simulated tasks and metrics, and educational software.

Devices, systems, methods and/or computer-readable media according to some embodiments may provide high-stakes assessment procedures for performance assessment, certification, and recertification.

Embodiments described herein can integrate multiple clinically relevant feeds (audio/video/metadata) for a medical process and allow comprehensive analysis of the medical process, the human and technical Efficiency and results are linked as events to develop solutions aimed at improving safety and efficiency and reducing costs.

Embodiments described herein may enable the successful identification, collection, and synchronization of multiple video, audio, and metadata feeds related to a medical procedure (e.g., to assess different metrics of a medical procedure), with all video and audio available in Sufficient processing power for form rendering.

Embodiments described herein may employ measurement tools and enable and include objective assessment of various aspects of human and technical performance and environmental factors in order to understand the causes of medical procedures and medical other A chain of events with adverse consequences in an aspect.

Possible applications of some embodiments include one or more of: (i) documentation of various aspects of patient care in clinical areas at high risk of adverse outcomes. Comprehensive data collection by an encoder according to some embodiments may enable and/or provide a detailed reconstruction of any clinical encounter. (ii) Analyze the chain of events leading to adverse outcomes. Data collection and processing according to some embodiments provides an opportunity to retrospectively evaluate one or more mechanisms and/or root causes leading to adverse medical and surgical outcomes. (iii) Analysis according to some embodiments may generate knowledge of the incidence and context of human errors, and may enable the development of strategies to mitigate the consequences of these errors. (iv) Designing training interventions for surgical teams. According to some embodiments, all identified hazard scenarios may be stored in a database and associated with simulated interventions aimed at preparing the clinical team for common clinical challenges and mitigating the impact of errors on clinical outcomes. (v) Evaluate/improve/develop existing/new healthcare technologies and new treatments. According to some embodiments, a comprehensive data set may be used to evaluate safety concerns associated with implementing new healthcare technologies. Furthermore, it can enable the evaluation of the impact of healthcare technologies on efficiency. (vi) For certification and accreditation purposes. According to some embodiments, the data may be used to assess human performance and develop a pass/fail score using a standard setting methodology.

Embodiments described herein may be used in association with an OR arrangement. However, embodiments are not limited thereto. Embodiments may also find application in medical settings, and more generally, surgical settings, intensive care units ("ICUs"), trauma units, interventional suites, endoscopy suites, obstetrical suites, and emergency room settings. Embodiments may be used in outpatient treatment facilities, dental centers, and emergency medical services vehicles. Embodiments may be used in simulation/training centers for healthcare professional education.

The example applications are presented for purposes of illustration, and are not intended to be exhaustive or to limit the embodiments to the precise forms disclosed. Other advantages, features, and/or characteristics of some embodiments, and methods of operation and/or functions of related elements of an apparatus, system, method, platform, and/or computer-readable medium, and/or steps, components, and/or manufacturing economies The combination of can become clearer when considering the attached drawings. According to some embodiments, certain features of systems, methods, devices and/or computer readable media as to their organization, use and/or method of operation and further objects thereof can be understood from the drawings in this exemplary embodiment and/or benefits. The drawings are for purposes of illustration and/or description only and are not intended as a definition of the limits of the invention.

Naturally, alternative designs and/or embodiments may be possible (e.g., alternative configurations of components, units, objects, features, steps, algorithms, etc., in place of one or more other components, units, objects, features , steps, algorithms, etc.).

Although some of the components, units, objects, features, steps, algorithms, relationships and/or configurations according to some embodiments may not be specifically related to each other references, however they may be used and/or are adapted to be used in connection therewith. Mentioned, depicted and/or various components, units, objects, structures, configurations, features, steps, algorithms, relationships, utilities, etc. herein may but not necessarily be incorporated into and/or realized by some embodiments. Any one or more of the components, units, objects, structures, configurations, features, steps, algorithms, relationships, utilities, etc. mentioned herein may be used in some embodiments or through certain embodiments on its own and/or Or realize in various permutations and combinations without referring to, considering or implementing any other components, units, objects, structures, configurations, features, steps, algorithms, relationships, utilities, etc. mentioned herein.

Other modifications and variations may be used in the design, manufacture and/or implementation of other embodiments according to the invention without departing from the spirit and scope of the invention.

multi-channel recorder or encoder

Figure 2 illustrates a schematic diagram of a multi-channel recording device 40, which may be referred to herein as an encoder. The multi-channel data recording device 40 of FIG. 2 may be the encoder 22 of FIG. 1 in some embodiments, or the encoder 1610 according to other embodiments.

The multi-channel recording device 40 may receive input feeds 42 from various data sources including, for example, feeds from cameras in the OR, feeds from wearable devices, feeds related to patient physiology from data storage, monitoring devices, and sensors, Feeds of environmental factors from various sensors (temperature sensors, decibel level sensors, room flow sensors), feeds of device performance parameters, etc. Multi-channel recording device 40 may synchronize and record feeds to generate output data 44 (eg, exported as a session file). Output data may include, for example, measures to assess individual and team performance, identify errors and adverse events and link to results, evaluate technology performance and safety, and evaluate efficiency.

There may be little research on causative factors and potential error mechanisms in surgery. The complex, dynamic, and/or data-intensive environment of OR can make it difficult to study the root causes of errors and/or patterns of events that could lead to adverse outcomes. A synchronous multi-channel recording device 40 according to some embodiments provides a comprehensive overview or data representation of the OR. This multi-channel recording device 40 or "black box encoder" can be modeled after the aviation black box and can register multiple aspects of the intraoperative OR environment, including room and/or process video, audio, sensors, anesthesia equipment, medical Equipment/surgical equipment, implants and hospital patient management systems (electronic medical records). The black box recording device 40 can be installed in real life OR/patient intervention areas such as hospitals, outpatient clinical facilities, emergency medical services vehicles, simulation/training centers, etc.

The black box recorder 40 can be used in anesthesiology, general minimally invasive surgery (MIS) surgery, interventional radiology, neurosurgery, and clinical practice. The black box recorder 40 can implement synchronization, audio, video, data capture, data storage, data privacy and analysis protocols, etc.

According to some embodiments, there is provided a multi-channel data recording device 40 for use in a clinical setting that simultaneously records multiple synchronized data feeds including process views, room cameras, audio, environmental factors through multiple sensors, anesthesia equipment, medical/surgical equipment, implants and hospital patient management systems (electronic medical records). A multi-angle view of the operating room may allow simultaneous analysis of technical and non-technical manifestations and identification of critical events leading to adverse outcomes. Implementation of a black box platform according to embodiments in a real life OR may reveal valuable insights into the interactions occurring within the OR/patient intervention area as a tool to identify, analyze and/or prevent errors in the intraoperative environment.

A multi-channel "black box" encoder 40 integrates and synchronizes audiovisual/digital data feeds and/or other quantitative, semi-quantitative and qualitative data feeds from the live OR or other patient intervention area onto a single interface.

hardware unit

The encoders are connected to one or more data capture devices, which may be grouped into hardware units 20 (FIG. 1), to monitor activity within the OR or other patient intervention area (and capture data representative of the monitored activity)

Hardware unit 20 may be located in an OR or other patient intervention area. For example, several recording devices could be installed in the OR/patient intervention area, e.g. as follows: wall-mounted wide-angle lens room camera to allow visualization of the entire room; several cardioid microphones to capture all conversations at a quality that allows analysis/ Details of noise/alarms; process video capture devices (endoscopic cameras, x-ray, MRI, etc.); and vital sign monitoring devices and sensors (environmental sensors, acoustic sensors, electrical sensors, flow sensors, angle/position/displacement sensors, and other sensors); medical/surgical equipment; and implants. Hardware units (eg, groups of data acquisition devices) interface with middleware hardware devices and encoders to connect and synchronize device feeds. Integration of platform 10 may be non-intrusive in the OR with minimal equipment setup. For example, anesthesia and laparoscopic feeds can be streamed in the OR, and without changing the room's infrastructure, microphones and room cameras can be installed.

room camera

According to some embodiments, the hardware unit 20 may have a camera 30 ( FIG. 1 ). FIG. 3 shows a schematic diagram of an example wide-angle video camera 50 in accordance with some embodiments. For example, two wide-angle cameras 50 (EVI-HD1 from Sony Tokyo, Japan) may be installed to capture data representing an entire view (eg, 180 degrees or more) of a room. As an illustrative example, room camera 50 may be mounted above the nurse's station and focused on the operating table for the purpose of capturing the surgical team in view. Two entrances into the room can be in view, which allows foot traffic to be measured by recording the opening and closing of doors and the number of individuals present in the room.

microphone

According to some embodiments, the hardware unit 20 may have an audio capture device 34 (FIG. 1). FIG. 4 shows an example audio capture device (e.g., Audio Technica of Tokyo, Japan) as three-way microphones 52, 54, 56. Schematic of a condenser gooseneck microphone, ES935ML6). The microphones 52, 54, 56 may be mounted to capture audio communications within or near the OR with the microphones 52, 54, 56 within range. Live surgical procedures can be observed in ORs or other patient intervention areas prior to installation to identify areas, locations or regions of high frequency communications and to assess major sources of environmental noise such as alarms from medical equipment, periodicity of anesthesia machines tones and/or noise from the intercom. This observation can be used to determine the location or placement of the microphones 52 , 54 , 56 . Different microphone settings can be tested by simulating the noise of a surgical procedure in an empty OR or other patient intervention area, and a setting for audio quality can be selected. According to some embodiments, the microphones 52, 54, 56 may be placed in two or more locations within the OR: (1) on an infield monitor (e.g., microphones 52, 54) pointing towards the surgical field; And (2) point over the scrub nurse and the nurse's station (eg, microphone 56) of the equipment cart. Each audio source can be recorded to a separate independent feed, with the option to mix audio feeds for post-recording. For example, they could be directional microphones mounted on the infield laparoscope monitor and above the nurses' station.

Process Camera View

According to some embodiments, the hardware unit 20 may have a camera 30 ( FIG. 1 ) providing a process camera view. On a separate stand-alone machine (AIDA, Karl Storz, Tuttlingen, Germany), laparoscopic video camera views can be recorded in the OR as part of diagnostic care. To incorporate this video feed into a black box recording device or encoder, a Distributed Amplifier (DA) can be used to split the video signal - allowing one signal to be displayed on an infield monitor while the other is streamed during operation. format to a black box recording device or encoder. In some example embodiments, the DA may also ensure that the aspect ratio of the black box laparoscopic recording corresponds to the 16:9 aspect ratio of the infield monitor. Video feeds can be recorded in high definition. Figure 5 shows a schematic diagram of example video hardware 60, which includes a DA for segmenting the video signal from the camera 30 used for diagnostic care, and a converter for converting the video signal into an appropriate video format for the encoder .

anesthesia equipment

According to some embodiments, the hardware unit 20 may have a patient monitoring device 36 (FIG. 1). For example, the patient monitoring device 35 may include an anesthesia machine monitor, which may be used to observe the patient's physiological data in real time and detect abnormal changes in the patient's vital signs. According to some embodiments, the vital signs display may be extracted from the anesthesia machine using a video card that generates a secondary feed of the VGA output. The vital signs video feed can be converted from VGA to HD-SDI format using a converter unit (VidBlox 3G-SL from PESA, Huntsville, Alabama, USA) prior to integration and synchronization with other video feeds.

In some embodiments, raw digital data may be extracted directly from the anesthesia device to be provided to encoder 22 for inclusion as metadata.

additional sensor

According to some embodiments, the hardware unit 20 may have a sensor 30 ( FIG. 1 ) installed or utilized in a surgical unit, ICU, emergency unit or clinical intervention unit. Example sensors include, but are not limited to: environmental sensors, i.e., temperature sensors, moisture sensors, humidity sensors, etc.; acoustic sensors, i.e., ambient noise sensors, decibel sensors, etc.; electrical sensors, i.e., Hall sensors, magnetic sensors, current sensors , MEMS sensors, capacitance sensors, resistance sensors, etc.; flow sensors, i.e., air sensors, fluid sensors, gas sensors, etc.; angle/position/displacement sensors, i.e., gyroscope sensors, attitude indicator sensors, piezoelectric sensors, Photoelectric sensors, etc.; other sensors, ie, strain sensors, level sensors, load sensors, motion sensors, pressure sensors, etc.

Hardware unit integrated into the operating room

According to some embodiments, hardware unit 20 may have a signal processor coupled to a data capture device. FIG. 6 illustrates a schematic diagram of a digital signal processor 62 according to some embodiments. According to some embodiments, video and audio data signals may be fed into a signal processor 62, which may be remotely located in a rack within the sterile core of the OR. The signal processor 62 is capable of supporting the incorporation of multiple video/audio signals and digital data as metadata. Signal processor 62 may be responsible for collecting audio and video signals from multiple independent data feeds or streams and encoding them into a compressed format.

FIG. 10 illustrates a simplified architecture of encoder 22 coupled to hardware unit 20 via network infrastructure 38 . This can be a direct or indirect network connection.

For larger application environments and to maximize efficiency and deliver increased failover and redundancy capabilities, a switch router (eg, router 16 of FIG. 1 ) may be used. Audio, video and data feeds may be connected to switch router 16 (FIG. 1) through network infrastructure such as cables or via connected wireless base stations. An example purpose of a router may be to route audio, video and data feeds to one of a number of encoders 22 available on the network. Using multiple encoders coupled to router 16 may provide a more cost-effective implementation, greater spatial coverage, and increased redundancy and failover for the system. Thus, the network infrastructure shown in Figure 10 may include one or more switches or routers. Further, although only one encoder 22 is shown for simplicity, there may be multiple encoders connected to one or more hardware units 20 via network infrastructure 38 . Although only one hardware unit 20 is shown for simplicity, there may be multiple hardware units 20 connected to one or more encoders 20 via a network infrastructure 38 .

FIG. 11 illustrates a schematic diagram of encoder 22 according to some embodiments.

For simplicity, only one encoder 22 is shown, but the system may include many more encoders 22 to collect feeds and exchange data from local or remote data capture devices (of hardware unit 20 ). Encoder 22 may be the same or a different type of computing hardware device. Encoder 22 has at least one processor, a data storage device (including volatile or non-volatile memory or other data storage elements or combinations thereof), and at least one communication interface. The components of encoder 22 may be connected in various ways, including directly coupled, indirectly coupled via a network, distributed over a wide geographic area, and connected via a network (which may be referred to as "cloud computing").

For example and without limitation, encoder 22 may be a server, network appliance, embedded device, computer expansion unit, personal computer, laptop computer, mobile device, tablet computer, desktop computer, or can be configured to perform the method of any other computing device.

As depicted, encoder 22 includes at least one processor 90 , memory 92 , at least one communication interface 94 and at least one web server 12 .

Each processor 90 may be, for example, any type of general purpose microprocessor or microcontroller, digital signal processing (DSP) processor, integrated circuit, field programmable gate array (FPGA), reconfigurable processor, programmable read memory (PROM), or any combination thereof. As described herein, processor 90 may be configured to synchronize collected data charges to generate a container session file. Processor 90 may also implement anonymization and encryption operations as described herein.

Memory 92 may comprise any suitable combination of any type of computer memory, internal or external, such as, for example, random access memory (RAM), read only memory (ROM), compact disc read only memory (CDROM), electro-optic memory, magneto-optical memory , Erasable Programmable Read-Only Memory (EPROM) and Electrically Erasable Programmable Read-Only Memory (EEPROM), Ferroelectric RAM (FRAM), etc.

Communication interface 94 may include I/O interface components to enable encoder 22 to interface with one or more input devices, such as a keyboard, mouse, camera, touch screen, and microphone, or with one or more input devices, such as a display screen and speakers. output devices are interconnected. Communication interface 94 may include a network interface component to enable encoder 22 to communicate with other components, to exchange data with other components, to access and connect to network resources, to service applications, and by connecting to a network capable of carrying data (or Networks) including the Internet, Ethernet, Plain Old Telephone Service (POTS) lines, Public Switched Telephone Network (PSTN), Integrated Services Digital Network (ISDN), Digital Subscriber Line (DSL), Coaxial cable, fiber optic, satellite, mobile, wireless (e.g., Wi-Fi, WiMAX), SS7 signaling networks, fixed line, private networks (including VPN 24), local area networks, wide area networks, etc., and combinations including these other networks. These are examples of network infrastructure (e.g., network infrastructure 38 of FIG. 10)

Figure 12 illustrates a flowchart of a method for collecting medical and surgical data, according to some embodiments.

At 102, using the customizable control interface 14 and GUI, a control command to activate the system may begin recording, collecting and streaming all available audio from the data capture device to one of the plurality of available encoders 22 , video and data feeds. Data capture equipment may include some or all available cameras, including mounted and laparoscopic cameras; all audio microphones and all available and implemented sensors used in surgical units, ICUs, emergency or other clinical intervention units, and Third-party devices (open or proprietary). Pause/stop/play are additional control commands received at the control interface 14 which may trigger the transmission of corresponding commands to the encoder 22 to control recording.

At 104 , in response to the control commands, the data capture device of the hardware unit 20 captures data representing various aspects of the OR or other medical unit and generates a feed or data stream for provision to the encoder 22 . Various example data capture devices are described herein.

At 106, the digital data can be formatted, converted, and synchronized by middleware hardware and software and using a networking protocol that synchronizes clocks across the network. The digital data will be included in encoder 22 as metadata.

At 108, encoder 22 may be responsible for synchronizing all feeds to generate a session record, as described herein.

At 110, encoder 22 may encode the synchronized feed into a signaling file using lossless audio/video/data compression software. According to some embodiments, Encoder 22 may also be responsible for hosting (or storing) and operating anonymization and speech/lexical distortion software(s) in order to protect all medical professionals, patients, distinguishing objects in a medical, clinical or emergency setting or characteristic identity. This can be done by the encoder 22 before compression, packing and encryption or after decryption in the background system.

Upon completion of recording, the container file may be securely encrypted by encoder 22 at 110 . Encryption/decryption keys can be embedded in the main session container file and accessible through the master key, or have separate keys.

The encrypted file may be stored on encoder 22 (eg, web server 16 of FIG. 1 ), or on a storage area network until scheduled for transmission. A communication or web server 16 on a dedicated VLAN may be responsible for schedule management and automated file and key transfers. This can be done through a private VLAN on the client environment and carried over a public data line to the backend via a virtual private network (VPN) (eg, VPN 24 of FIG. 1 ). As part of the configuration, the communication server 16 may be responsible for backing up data including audio, video, data, encrypted files, etc. using backup software. The communication server 16 may be responsible for hosting and directing all traffic between the private VLAN and the back office.

According to some embodiments, the synchronized compressed encoded signal may be fed into a touch screen monitor located inside the OR, which may be responsible for the real-time visual display feed and record directly to an external hard drive.

control interface

According to an embodiment, the user interface may be provided on a PC based touch screen monitor. The user interface may be referred to herein as control interface 14 ( FIG. 1 ), and may serve as a “central control” station that records video and audio feeds in real time, and communicates control commands to encoder 22 . A Graphical User Interface (GUI) and its parameters can incorporate principles of UI design to provide a simple user-friendly and functional interface.

According to an embodiment, the feature of the control interface 14 providing a central control station (eg, computer, tablet, PDA, hybrid, convertible) may be located in the clinical unit or another customer specified location. It contains a customizable Graphical User Interface (GUI) that provides simple, user-friendly and functional control of the system.

According to an embodiment, the control interface 14 may have a play/pause button. Some subsections of the process may not need to be documented. In order to skip these segments from the recording, the user interface can pause and restart the recording as needed through control commands generated in response to activation of the play/pause button. Pause and play timestamps can be recorded in the log file indicating the exact time of the extracted process.

According to an embodiment, the control interface 14 may have a stop session button. When the "Stop Session" button is selected, the file may be closed and automatically passed to the storage area network (SAN), encoder 22, or the like.

According to an embodiment, the control interface 14 may have a split screen quadrant display of the video feed. A visual display of the video can be provided in real-time during recording.

According to an embodiment, the control interface 14 may have a recorded visual indicator. For example, red flashing dots can appear on the screen to provide the team with a visual indication that video and audio information is being recorded.

According to an embodiment, the control interface 14 may have a log file. At the end of the recording, a log file can be generated indicating key points in time, including the beginning and end, pause and replay of the recording session.

According to an embodiment, the control interface 14 may have password protection. The interface can be secured using several layers of password protection to ensure patient confidentiality and privacy is maintained.

Figure 7 illustrates an example schematic diagram of a control interface according to some embodiments. The control interface 14 may provide password protection for the control screen 64 with a touch screen monitor (of the tablet device). Control interface 14 may provide display screen 66 with multiple views of the OR from multiple feeds from data capture devices located within the OR.

Figure 8 illustrates an example schematic diagram of an OR integrated with a hardware unit of a data capture device to capture data representing different views of the OR. The data capture equipment used for this example illustration includes a room camera 70, a microphone 72 (located at the infield monitor and above the nurse's station), distributed amplifiers and video converters 74 for processing the laparoscopic video signal, and Commands to control the recorded touchscreen monitor 76.

Rich content analysis unit (ie, video analysis software)

The rich content analysis unit facilitates the ability to simultaneously process, manage, review, analyze and tag rich content (eg, video, audio, real-time patient metadata such as heart rate, etc.) in multiple formats.

The rich content analysis unit may provide a user (ie, medical professional, surgical specialist, or medical researcher) with an intelligent dashboard that allows annotation and tagging of rich content streams. That is, smart dashboards may interview playback views to review content and interface controls to flag content. Smart dashboards can be multidimensional in that the union of all dimension variables (ie, case variables) can indicate a specific set of one or more applicable annotation dictionaries (ie, coding templates). Some examples of variables that may be used to determine the annotation and tagging dictionaries may be: the type of medical procedure being performed (e.g., laparoscopic bypass), aspects of the procedure being analyzed (e.g., technical skills, non-technical skills, etc.) , the geographic region/region where the process is performed (this may specify a region-specific annotation dictionary that maps to a generalized globally accepted dictionary), etc. These are example variables.

The rich content analysis unit may enable data modeling and cross-referencing between annotated dictionaries (ie, coding templates) across various medical procedures, country interpretations, and the like. Each annotation dictionary may allow simultaneous tagging of an entire rich content stream (ie, allow creation of descriptive content). For example, content streams can be tagged with well-formed descriptors suitable for different analysis goals. For example, an annotation dictionary may allow flagging technical skills (an example analysis target), such as stitching errors or bookbinding errors (ie, tags), and flagging every instance in a rich content stream where these types of errors may have occurred.

Rich content refers to multiple streams of content in various formats (audio, video, numerical data, etc.). The union of all case variables may require custom or multiple annotated dictionaries based on previously validated rating tools to assess different aspects of process and recoding, including but not limited to technical performance, non-technical performance, non-process errors and events, and human factors. Each annotation dictionary can be a well-formed relational dataset.

Another feature of the rich content analysis unit is that the final aggregate of the entire rich content stream and the entire descriptive content (eg, technical skill notes/markups, non-technical skill notes/markups, etc.) can be reviewed in a post-sync aggregate.

The rich content analysis unit can be distributed using web technologies to ensure that the content is centrally hosted in a secure healthcare facility approved environment. For each aspect of the procedure being analyzed, the rich content analysis unit can ensure that only applicable rich content streams are played simultaneously on a single user interface (e.g. an audio feed from an operating room when rating a surgeon's purely technical skill) Not applicable). The rich content analysis unit can provide a variety of customizations, which again are provided only depending on the aspect of the process being analyzed. These customizations include, but are not limited to: the ability to increase the granularity of any content stream (e.g., enlarge or reduce the size of a video stream), control the playback speed of any content stream (e.g., increase or decrease the speed at which a video is played back), improve the content stream quality (for example, to apply filtering to increase the clarity of the audio stream).

Black-box encoder analysis unit (ie, black-box database)

The black box encoder analysis unit may provide the second part in the two-part handshake between the rich content analysis unit. The Black Box Coder Analysis Unit may incorporate quantitative and qualitative analysis processes to facilitate reporting capabilities including, but not limited to, comparative analysis, benchmarking, negative trends, data mining, statistical reporting, failure analysis, and key performance indicators. The black box encoder analysis unit may also facilitate aspect-based integration to statistical software research tools such as Matlab.

An example feature of a black-box coder analysis unit may be its relational database, which captures and cross-references the entire dataset synthesis, which includes, but is not limited to: complete results annotated by rich content analysis software identified using structured metadata and generated by rich content analysis software. Tag content streams generated by content analysis software, such as technical procedure rating systems for laparoscopic bypass, etc.; facility variables, such as department, operating room, etc.; procedure case variables, such as case urgency, number of medical staff present, and designation, etc.; process case notes (in a structured well-formed relational data model), such as what type of stapler was used, what hemostatic agent was used, etc.; patient-centric data, such as blood work and OSATS score.

In addition to the listed example reporting capabilities, the Black Box Encoder Analysis Unit can provide visual comparative analysis. The data set may be displayed in its entirety or a subset thereof on a visual timeline distributed by relevant metadata such as components of annotated dictionaries (eg, technical errors) or case variables.

Visual comparative analysis can provide example benefits including, but not limited to: the ability to review errors and events and determine forward and backward actions and observations; the ability to define, execute and transform visual observations into programming algorithms that can be performed on large amounts of annotated content. For example, programmatically identify where technical error clusters lead to more severe technical incidents; benchmark, benchmark and improve inter-rater (i.e., content flow analysis) by comparing different observers' timelines The ability of the medical team to assess the cause of major adverse events (eg, human error, medical equipment failure, etc.) in a given situation.

Another exemplary feature of the Black Box Coder Analysis Unit is its dual purpose capability to improve patient outcomes through continual improvement using healthcare intelligence analysis defined in the Black Box Analysis software. For example, identifying small unnoticed possible small actions that could lead to serious consequences; and supporting continuous improvement through additional research initiatives by integrating with research-related software tools such as Matlab, etc. and providing research-driven comparative analysis, For example, use a "Year 1" vs. "Year 2" study model to compare specific outcomes.

Illustrative example application

An illustrative example embodiment of a black box recording device may include: two wall-mounted high-definition wide-angle cameras; two omnidirectional microphones; a laparoscopic camera view; and a vital signs display. These are example data capture devices for hardware units. This example application may use an Internet Protocol ("IP") network where each data signal may be fed to an Ethernet Switch ("ES"). The purpose of the ES may be to create a local area network (LAN) that establishes a central connection point for all sources. Each data feed may be assigned its own Internet Protocol (IP) address prior to connection to the ES. The video cameras and corresponding microphones can be IP-based built-in encoders, while the laparoscope and anesthesia source feeds can first pass through an additional encoder device that converts the analog or digital video signal into a Real Time Streaming Protocol (RTSP) video stream. Data signals can be bundled at the ES and directed to a touchscreen user interface on a PC-based platform (Patient Viewing System, "POS"). The POS can be responsible for decoding the data into a readable signal, and synchronizing the data feed.

In some IP networks, video feeds and/or audio feeds may be streamed separately through the network from endpoint to endpoint, which may create opportunities for network delays along the streaming path. Over time, the delay between the video feed and the audio feed can add up, and/or each feed can experience different network delays. Delay may be unknown and/or constantly changing over time, and/or may be difficult to quantify and/or account for delay and/or lead to an effect known as "drift". Another example embodiment of a black box platform may be provided without the same IP networking functionality of the examples described above. Another example embodiment may use a microcoder-synchronized self-clocking signal processor. According to an example embodiment, a self-synchronizing signal processor may ensure that the audio and video streams are "locked" from drifting, and thus allow the feeds to be shifted after recording to achieve synchronization.

Another example embodiment of a black box system may use omnidirectional microphones placed above the operating table and at the equipment boom in an attempt to capture audio surrounding the surgical field. However, omnidirectional microphones may have equal output/input at all angles, and/or may detect sound from all directions. These microphones can result in sub-optimal and/or inferior audio quality with excessive background noise and poor team communication detection.

In another example embodiment of a black box system, a directional cardioid microphone that is sensitive in the front and isolated from ambient sounds may be used. These microphones can be placed on the infield monitors, pointing towards the surgical field where communication exchanges among the surgical team may occur. This setup results in excellent audio quality with clear voice and sound detection.

FIG. 9 illustrates an example schematic 82 of a polar pattern for an omnidirectional microphone and an example schematic 80 of a polar pattern for a cardioid microphone. As shown in Figure 82, an omnidirectional microphone may have equal sensitivity at all angles. As shown in FIG. 80, a cardioid microphone may be directional with greater sensitivity in the front and less sensitivity in the rear.

According to embodiments described herein, a synchronized multi-channel video/audio/metadata recording platform can be used in an intraoperative environment. The development and installation of a black box platform may be an iterative process that may include minor and major changes to the system.

While other industries such as television broadcasting may have equipment to capture video and/or audio according to some embodiments, a "black box" platform for medical use can be cost effective, ensuring privacy for patients and healthcare professionals , compact storage in the OR, suitable for non-invasive installation with existing equipment in the OR, designed to meet hospital infection control standards, and more. Furthermore, the platform can integrate multiple feeds from multiple sources in multiple formats onto a single system and can ensure that records are encoded into a common format compatible with subsequent data analysis.

Black box recording equipment may include one or more of the following: audio capture and synchronization, and digital data capture. The integration of all these data streams can provide a complete reconstruction of the clinical encounter. Communications can be an integral part of performance analysis for non-technical and human factors. For example, communication failure may be a contributing factor to adverse events in the OR. Furthermore, team interactions in OR may rely on verbal communication, which may not be properly evaluated without adequate audio quality. For example, for stand-alone video files, in the absence of audio components, components of non-technical performance (including teamwork, leadership, and decision-making) may not have been evaluated. Audio can be difficult to capture in the OR due to multiple noise sources in the room. Major sources of noise in the OR may include the following: preparing for surgery (before incision), moving carts and equipment, doors opening and slamming, moving and setting down metal tools, suction, anesthesia monitors, noise from anesthesia and surgical equipment Alarms, and/or conversations between staff and/or on intercoms. Microphone systems can be designed to capture all audio in the OR, for example: omnidirectional microphones to capture ambient sound, hypercardioid microphones to capture the anesthesiologist's surroundings, cardioid microphones to pick up clinician conversations in the surrounding area, and A wireless microphone worn by anesthesiologists to capture their voice. While such a microphone setup is capable of capturing multiple noise sources, its intrusive nature in the OR may introduce the Hawthorne effect. Furthermore, mixing multiple audio feeds can result in poor audio quality, and analyzing each feed separately can be time-consuming.

According to some example embodiments, the platform may include an audio system with the fewest number of microphones, which produces the best audio quality. To analyze the performance of non-technical skills and human factors, team communications can be an audio source of interest. Since the communication can take place at the surgical field around the operating table, two cardioid microphones can be mounted on the infield monitor and pointed towards the surgical team. Additional microphones can be placed at the nurse's station and pointed towards the scrub nurse and equipment cart. A testing and validation phase may help with microphone setup. Testing can reproduce the noise of a surgical procedure in real life OR in order to identify settings likely to result in desired and/or optimal audio quality.

According to some example embodiments, the black box recording device may also provide audio-visual and multi-feed synchronization for proper data analysis. The audio and video feeds can be synchronized because even a delay of eg one-thirtieth of a second between the two signals can result in detectable echoes. Latency lag can increase exponentially over time. An example embodiment of a black box recording device may have a latency of less than 1/30 second, resulting in synchronization for proper data analysis. Multi-feed synchronization can be provided for multi-view analysis in surgical cases. The black-box device may enable analysis of events in the OR from multiple perspectives, such as, for example, room views, process camera views, vital signs, and digital data from various sensors. Latency between video/audio/data feeds can reduce the value of multi-channel video recording. In an example embodiment of a black box recording device, the digital data may be formatted, converted and synchronized by middleware hardware and software and using networking protocols for clock synchronization across a network. Numerical data can be included in encoders as metadata. The encoder can take care of synchronizing all the feeds, encoding them into signaling files using lossless audio/video/data compression software.

Regarding the design of the recording device, the recording device may have a user-friendly interface that satisfies privacy concerns. The recording system interface may have a visual display of the recorded feed, etc., providing participants with awareness of what was recorded, and when the recording took place. Additionally, in some example embodiments, recording devices may be designed to maximize confidentiality and privacy for both patient and staff participants. Room cameras can be positioned to keep the patient's identity out of view. Microphones may be placed to only capture communications around the surgical field, rather than off-the-record casual communications in the periphery. Some embodiments of the system may have a pause feature that allows recording to be easily and seamlessly paused during portions of the procedure that were not intended to be recorded (eg, the intubation phase or the extubation phase). Multiple layers of password protection ensure that the system of record can only be accessed by authorized personnel of the research team.

The black box can be built on a modular design, ie the recording system can be modified, feeds (and associated data capture devices) can be removed or added without changing the main/overall functionality of the system. This design approach may allow black box recording devices or encoders to be incorporated into other data feeds and/or adapted to different clinical settings (e.g., ER departments, ICU, endoscopy suites, maternity suites, trauma rooms, surgical wards/ medical ward, etc.). The system can be modular and can be expanded to accommodate modifications and larger applications. Depending on the nature of the medical procedure and the data capture equipment available, the system can, in other examples, incorporate additional video, audio, and/or time-series data feeds (eg, heart rate monitors, force-torque sensors).

"Black box" data logging equipment in the operating room

OR is a high-risk work environment where complications can occur. Root cause analysis may reveal that most complications result from multiple events rather than a single cause. However, previous efforts to identify these root causes may have been limited to retrospective analysis and/or self-reporting. Example embodiments of the platform may implement a multi-channel data recording system for analyzing audio-visual data and patient-related data in a real-life OR.

According to one or more embodiments, a "black box" data recording device or encoder can capture multiple simultaneous feeds in the OR/patient intervention area, e.g., room and procedure views, audio, patient physiological data from anesthesia equipment, and digital data from various sensors or other data capture devices. These feeds can be displayed on a single interface (eg, control interface 14), providing a comprehensive overview of operation. Technical skills, error/event rates and non-technical skills that can analyze data. According to some embodiments, postoperative human factors questionnaires may be completed by the surgical team.

13-15 illustrate schematic diagrams of various example views according to some embodiments. For example, FIG. 13 illustrates a schematic interface with a graphical indicator 150 showing a data feed and an OR layout with example positioning of various data capture devices.

Figure 14 illustrates a schematic diagram of data flow 160 between different system components. Differential data capture equipment is shown, including cameras 162, 166, 170, patient monitor 164, microphones 168, 172, and the like. The data capture device may provide output data feeds to encoders 174 , 176 , other data capture devices, or patient viewing system 178 . Medical or surgical data may be provided to the display device 180 for display or to receive interactive commands via a touch screen interface to control one or more components of the system (eg, view changes on the camera, start or stop recording). This is an example configuration, and other flows and connections may be used by different embodiments.

15 illustrates an example OR view 190 with different data capture devices such as patient monitor 192, microphone 194, laparoscopic camera 196, room mounted camera 198, and touch screen display device 199 to provide real-time medical data collected A visual representation of the feed is fed as output data, and control commands to start or stop the capture process are received, eg, as input data.

Black box recording devices or coders can provide technical and non-technical individual and team performance, errors, event patterns, risks and performance for analysis of medical/surgical equipment in the OR/patient intervention area. Black box recording devices or encoders can open up opportunities for further research to identify root causes of adverse outcomes and develop specific training sessions to improve clinical organizational processes, as well as surgical/device performance, efficiency, and safety.

cloud platform

Embodiments of the black box recording device may address technical considerations of improved synchronization, thereby reducing latency exposure, providing extended and multi-region modalities, and reducing cross-platform costs. A cloud platform may include developing smart devices and generating time stamps for collected data so that devices and data are synchronized.

FIG. 16 shows an example schematic diagram of a black box recording device 1600 that may provide a cloud-based platform, according to some embodiments. Example platform components that provide this capability include autonomous and semi-autonomous smart-enabled devices and adapters, such as medical devices 1602, cameras 1604, microphones 1606, sensors 1608, and the like. In some embodiments, the black box recording device 1600 may be provided by an encoder 1610 connected via a wireless station 1616 to a media management hub (MMH) 1612 storing client media management software instruction code (CMMS) 1620 . This is connected to a central content server and management software (CCS) 1614 via a client network infrastructure 1618 configured to employ and utilize high performance wireless communication standards.

Smart-enabled devices and adapters may be autonomous or semi-autonomous smart devices including, but not limited to, smart cameras 1604 , microphones 1606 , data and media converters 1612 , encoders 1610 , adapters and sensors 1608 . In this illustrative embodiment, a smart-enabled device or adapter may incorporate and utilize a SOC device (System on Chip) or FPGA device (Field Programmable Gate Array) in conjunction with onboard storage, power management, and radio(s). It manages device requirements, device-to-device authentication, storage, communication, content processing, clock synchronization, and time stamping. Depending on factors, the technology can be integrated directly into the device or as an adapter attached. In some example embodiments, smart-enabled devices and adapters may connect directly to the CCS 1614 to provide data from the operational site via the secure client network infrastructure 1618, and may receive data, commands, and configuration from the CCS 1624 either directly or via the MMH 1612 controls.

The black box encoder 1610 can consist of one or more computing devices, tablets and/or laptops, which can run a secure user interface for surgical personnel to operate the black box platform. It can reside on the client network connected via Ethernet or wirelessly (eg, via station 1616), and can comply with network security and IT policies. In some example embodiments, the black box encoder 1610 may connect directly to the CCS 1614 to provide data from the operational site via the secure client network infrastructure 1618 and may receive data, commands, and configuration controls from the CCS 1624 either directly or via the MMH 1612 .

Media management hub (MMH) 1612 may be a computer or server responsible for running client media management software and its associated services. As an illustrative example, it can run on Unix, Linux, or Windows Server. The Media Management Hub may reside on the client network and must comply with client network security and IT policies in addition to the necessary compute, IO and storage requirements.

Client Media Management Software (CMMS) 1620 may be an application running on Media Management Hub 1612 that acts as an intermediate conduit between the backend central server and the smart-enabled capture devices and adapters. It can be responsible for the management and control of a black box platform residing on the client network. The CMMS 1620 can aggregate, package, compress, and encrypt captured audio, video, medical device data, sensor data, logs, and the like. The CMMS 1620 can use standardized file naming conventions, keywords, folders, etc. to organize output files and categorize by event. The CMMS 1620 can provide device management, including passing commands from the console, device authentication, security, file transfer handshaking, and the like. The CMMS 1620 has a device status dashboard with log file management and error reporting. The CMMS 1620 provides workflow automation, document management and delivery between client sites and the central server. CMMS 1620 provides additional computing solutions that comply with client network security and policies. CMMS 1620 provides processing and data transformation of clock broadcasts for device synchronization.

A central content server and management software (CCS) server 1614 may be located at the main site and serve as a bi-directional interface to communicate with satellite or client site hubs. The CCS server 1614 supports remote management, automation, and file delivery handshaking for delivery of packaged, compressed, and encrypted content from client sites. As described herein, the CCS server 1614 serves as a conduit for black box analysis software and databases.

High performance wireless communications (HPWC) may be provided by one or more wireless stations 1616 . For example, HPWC may be implemented using multi-gigabit speed wireless communication technologies utilizing 802.11adWiGig, HD Wireless, or current standards that support high-bandwidth digital content transmission.

The workflow is provided as a functional illustrative example. Upon receiving commands from the platform console located in the operating or surgical suite, the smart-enabled device will begin capturing appropriate content (audio, video, digital data) to provide The numeric representation of the object. The smart device or smart adapter will process (eg, record, store, generate, manipulate, transform, convert, and render) the captured media and data, and embed timestamp marks in output files at precise timeline intervals.

Output files are transmitted from the smart-enabled device(s) to the MMH 1612 via Ethernet or high-performance wireless communication routers and/or devices, as indicated by wireless station 1616. A wireless router may be a multi-band wireless station using 802.11ad or the popular multi-gigabit speed standard.

The CMMS 1620 can aggregate all media and data (audio, video, device data, sensor data, logs, etc.) and package, compress, and encrypt to generate output files. Organize output files on network-accessible storage devices using standardized file naming conventions, keywords, folders, and more.

At predetermined intervals, files may be passed from the client site to the processing facility or back office through a VPN tunnel (eg, secure network infrastructure shown as client network 1618). The CCS 1614 at the receiving facility will manage file delivery and distribute content files, media and data to the black box analysis database.

System 1600 implements synchronization techniques. For example, hardware-based encoding and synchronization can be implemented in part using software methods. Data synchronization on smart-enabled devices by embedding timestamps from the device clock. Device clocks are synchronized across the network from the MMH 1612 via broadcast over a high-speed wireless network (shown as client network 1618, wireless station 1616, etc.). Since synchronization is performed by software at the source, media and data can have near-zero latency levels and the highest levels of accuracy.

System 1600 implements device management techniques. Devices and coverage areas can be managed under administrative authority on a central console or remotely via the CCS 1614 . Controls can be put in place to prevent device scheduling conflicts. User-selectable capture configurations may be presented based on location, regional requirements, or process type.

System 1600 implements area management techniques. Since current hardware-based encoding and synchronization solutions are limited by the number of IO ports available on the encoding device. Software synchronization and smart-enabled devices may allow for larger and easier deployments. Extended area and multi-area captures can be achieved, allowing richer content and longer visibility into the chain of events supporting data analysis.

System 1600 implements device state techniques. For example, the smart-enabled device or adapter operating status will be broadcast back to the CMMS 1620 from the authenticated device. An administrator at a client site and/or remotely via the CCS 1614 can access a device dashboard interface that automatically generates a visual representation of data reporting key operational metrics and status on all certified smart-enabled devices (e.g. , online, offline, run capture, onboard storage, etc.). In the event that a smart-enabled device or adapter is operating outside of normal conditions (eg, storage full, offline), an alert (email, SMS) will be sent to the administrator and logged appropriately.

System 1600 implements document management techniques. When capture and processing is complete on the smart-enabled device or adapter, the processed file will be passed to the MMH 1612. The CMMS 1614 will communicate with the device and delivery will be confirmed by a handshake. Each device or adapter can have onboard storage which will serve as cross-platform short-term file redundancy and recovery.

System 1600 can provide reduced cost, lower latency, and greater flexibility. Multicore encoders and copper cabling in a confined workspace can translate into high cost and debug complexity. Cable routing must be pulled through the catheter in the sterile core. Cable length affects signal latency. Hard wired connections may limit equipment placement and affect capture quality. Example embodiments described herein may be based on software solutions over the air (at least partially configuring various hardware components), and the use of smart-enabled devices may reduce overall hardware costs, resulting in higher accuracy and capture quality, larger flexibility and easy debugging.

motion tracking

Embodiments described herein may use 3D cameras or IR devices to achieve motion tracking. For example, a black box platform can collect and incorporate human and object motion tracking data at a surgical site. To maintain complete freedom in a clinical setting, marker-free motion tracking may be required. Data can be collected from 3D cameras or time-of-flight cameras/sensors.

The platform can implement motion tracking techniques using various components and data transformations. For example, a platform may include one or more autonomous or semi-autonomous 3D depth cameras or time-of-flight (TOF) sensors using lasers and/or infrared (IR) devices. As another example, the platform may generate distance and/or location information from the output signal of the TOF sensor and convert it into a 3D depth map or point cloud. Embodiments described herein may include a computing device for processing output data from a 3D camera or TOF sensor. Embodiments described herein may provide customized data processing to distinguish motion resulting from changes in captured depth maps. Embodiments described herein may provide media management hardware and software to aggregate, package, compress, encrypt, and synchronize captured point clouds as motion data with other collected media. Embodiments described herein can provide a central console for devices, and capture management and processing software to convert motion data into analyzable information to be used for research artifacts, workflow design, and chain-of-event analysis.

A workflow is described to provide an illustrative example of the functionality provided by the platform. In some examples, a 3D depth camera or TOF sensor is fixedly mounted in an operating or surgical suite. Upon receiving a command from the platform, the camera captures and generates viewable distance and position information of the capture area. The output data will be passed to a computing device that runs a custom process that creates and establishes a baseline measurement (static field map) and provides by comparing and measuring changes in positional information between adjacent 3D depth maps and point clouds Aggregated exercise data. The aggregated baseline and frame measurement data can be passed to media management software (eg, software 1620 on MMH 1612 ), which can aggregate, package, compress, encrypt, and synchronize motion data with other collected media.

At predetermined intervals, the files will be passed through the VPN tunnel from the client site to a processing facility or back office where the motion data will be processed into analyzable information to be used for research on human factors, workflow design and analysis chain events.

Example processes may involve different operations including, for example, computational operations for receiving a 3D depth map or point cloud of point-to-point measurements formatted and structured to enable changes. Computational operations can then create and establish baseline measurements (static field maps), and analyze and record changes in adjacent depth maps or point clouds. Computational operations may map changes to a common timeline, and summarize change data on a time-continuous basis for comparison with a reference static field map.

Embodiments described herein can provide synchronization of devices and collected data. For example, the platform could enable the synchronization of various media streams to a common timeline as a factor in determining the quality of the analysis. The following are examples of requirements to maintain synchronization accuracy: direct connection of all source devices into a general-purpose computer; sufficient IO and computing power to compress, encrypt, encode, and organize multiple streams of audio, video, and data files; evaluate, determine, and understand all Latency for incoming feeds; utilities or algorithms for adjusting and calibrating data feeds to ensure synchronization (examples introduce offsets); and calibrating timestamps in file headers to a common playback standard.

Embodiments described herein can provide analysis tools. In future embodiments, the process operation may convert point cloud and/or depth map position, distance and change measurements into real-world distance measurements. These measurements may permit the creation of key performance indicators (KPIs) in a semi-autonomous manner. KPIs can be used for further analysis and/or to provide recommendations for workflow and human factors affecting timelines and chains of events. These may include: steps taken, distance traveled, path taken versus best route, effects of accidental collisions or buildups, effects of space design, effects of personnel, equipment, arrangement and orientation of equipment, etc.

Analysis applied to black-box datasets

Embodiments described herein can implement data-driven surgical error analysis tools to investigate error mechanisms and evaluate error and event patterns. Embodiments described herein can enable process operations of formative feedback, self-assessment, learning and quality control, and identification of patterns, correlations, dependencies and signatures from collected data.

Embodiments described herein may provide the application of data-driven modeling to identify and extract features, correlations, and signatures from data collected and analyzed from OR black-box encoders. Data-driven modeling provides a good point of view to describe and analyze all those systems for which closed-form analytical expressions may be difficult to determine. Using a dataset of input-output sample pairs relevant to the problem, the goal is to use computational intelligence (CI) to reconstruct a mathematical model that identifies key factors and predicts clinical outcomes, costs, and safety concerns. CI tools can include neural networks, support vector machines, fuzzy inference systems, and several techniques from time series analysis and dynamic complex systems. Using a CI-based approach, offline and online solutions can be constructed for the analysis of errors, adverse events, and adverse outcomes in surgical procedures. The term offline refers to solutions that can be used to automatically reason about knowledge (e.g., causality rules, correlations) from examples describing past events recorded in the OR. An online approach can provide real-time tools to assist surgeons and OR teams during surgery. Such an instrument may operate by monitoring current conditions in the OR, reporting events that may result in conditions of potential error (eg, noise level, temperature, number of individuals in the room, etc.).

An overview of the computational intelligence methodology applied in the OR black-box encoder solution is provided below. Computational intelligence methodologies can be used to design networks capable of extracting features, correlations, and behaviors of events including complex multivariate processes with time-varying parameters. For the present application, methods may include Artificial Neural Networks (ANN), both feed-forward and recursive feed, Radial Basis Function Networks (RBFN), Fuzzy Logic Systems (FLS) and Support Vector Machines (SVM). Applied to data generated by OR black boxes, these systems will be able to implement a variety of functionalities including, for example, finding complex non-linear and hidden relationships between data representing human performance, patient physiology, sensors, clinical outcomes, and clinical costs , and predict outcomes and behaviors. Other example functionality includes functional generalization, so that it is acceptable to respond to situations where the OR black box encoder solution has not been exposed before, and to provide alternatives when the system cannot be expressed in equations or when the mathematical model does not exist or is ambiguous solution.

Example strengths of FLS are the ability to express non-linear input/output relationships through qualitative sets of if-then rules and to handle numerical data and linguistic knowledge, especially the latter, which can be difficult to quantify through conventional mathematics. On the other hand, the main advantage of ANN, RBFN and SVM is the inherent learning ability, which enables the network to improve its performance adaptively. This solution can apply CI methodologies, including ANN, RBFN, and SVM, to develop robust networks and models that will extract features, detect correlations, and identify event patterns from OR black-box datasets.

As noted, embodiments described herein may implement data analysis techniques using artificial neural networks. For example, time series modeling can include the application of time-delayed ANNs and feed-forward multilayer perceptron networks to model nonlinear dynamical systems. As another example, hybrid stochastic and feed-forward neural networks can be used to predict nonlinear and non-stationary time series by incorporating prior knowledge from stochastic modeling into neural network-based predictors. As another example, a two-layer neural network consists of a series of nonlinear forecasting units and Bayesian-based decision units for time series classification. As another example, ANNs are used for time series forecasting and affect the use of heuristics to select the optimal size of the sampling window. Other neural network topologies can be used, such as recurrent architectures, whereby temporal relationships can be built into the network via feedback connections. Periodic and chaotic time series forecasting with recurrent neural networks has been extensively studied. Some additional examples include the application of robust learning operations based on recurrent neural networks for filtering outliers from the input/output space suitable for time series forecasting; various methods for optimal parameter tuning in pipelined recurrent neural networks for forecasting nonlinear signals; a choice of methodologies; complex-valued pipelined recurrent neural networks for modeling/forecasting of nonlinear and non-stationary signals; Self-Organizing Maps and Recurrent Neural Networks for Nonlinear and Noisy Time Series.

Some exemplary embodiments may use radial basis function networks, where feed-forward and recursive RBFN's for time-series modeling of black-box datasets may be examined.

Some example embodiments may use neuro-fuzzy networks. Different adaptive neuro-fuzzy inference systems (ANFIS), alternative neuro-fuzzy architectures (ANFA), dynamic evolutionary neuro-fuzzy inference systems (DENFIS) to chaotic time series forecasting can be utilized. Examples of such applications include: (1) real-time neurofuzzy-based predictors for dynamical system forecasting; and (2) use of non-orthogonal wavelet-based, recursively compensated neuro-fuzzy systems and Systematic Weighted Recurrent Neuro-Fuzzy Networks Hybrid Recurrent Neuro-Fuzzy Networks.

Other example embodiments may use support vector machines. SVMs can be used for time-series prediction of clinically relevant performance outcomes, adverse events, complications, and cost/return on investment.

Some example embodiments may use nonlinear black-box data modeling techniques. In the absence of a priori information, embodiments described herein can use a model that describes the dynamic behavior (features/signatures) of the system based on a finite set of measured input-output pairs. Various nonlinear black-box modeling problems can be implemented as problems of using input-output data to select the best mapping mechanism, and then attempting to minimize the error between the model's output and the measured output.

Educational strategies generated using black-box data

Embodiments described herein can enable educational interventions based on OR black-box performance analysis. For example, an embodiment may provide a training solution or provide an output data file that may be used to generate a training solution.

Data obtained from systems analysis of surgical procedures can provide insights into complex processes within healthcare systems, allow assessment of individual and team-level performance, and evaluate human interactions with modern technologies. In addition, this data can be used to identify specific individual and team performance deficits, hidden areas in the process, and characterize the cascade of events leading to "near misses" or adverse patient outcomes. This information can deliver critical knowledge content needed to tailor effective educational interventions based on real-life observations rather than the hypothetical scenarios used in current training. This concept based on experiential learning theory can be used to create generalizable educational strategies that can be packaged and delivered to sites that do not have access to their own real-life data.

All training interventions can be tested using a rigorous research methodology to generate a collection of effective training solutions rooted in real-world observations.

Educational interventions can employ a variety of instructional strategies, such as team debriefing, individual and team coaching, error awareness and mitigation training, behavioral modeling, and warm-up simulation training.

Embodiments described herein can provide identification of root causes of adverse outcomes and design of training scenarios. By way of example, the causes of adverse patient outcomes may remain elusive because they are often multifactorial and based on retrospective analyses. Embodiments described herein with black box generated data may allow analysis of predictively documented adverse consequences. Patterns of recursive problems can be identified, characterized, and used to generate sets of scenarios based on actual experience. This knowledge may be relevant to all OR teams involved in patient care in similar clinical settings. Educational content can be compiled and delivered to information sheets, textbooks, e-learning software, virtual reality simulation tools and software, and integrated into SOPs at the institutional level.

In addition to summarizing common or significant root causes of adverse outcomes, these scenarios can be used to generate software packages for full-scale simulations in virtual ORs. These variables can be programmed into simulation software that can be packaged, commercialized and exported to educational institutions around the world.

Embodiments described herein may provide technical analysis to determine error frequency, distribution, and areas of concern. For example, end users of this data may be practicing physicians/surgeons and students. Mapping process complexity and identifying areas of potential hazard can be used to create instructional strategies that directly address these steps. Teaching strategies such as deliberate practice can then be used to train surgeons to prepare for these steps, minimizing the risk of adverse events. Informing the surgeon of complex or hidden steps also enables the design of SOPs (such as, for example, in aviation with a "sterile" cockpit concept during takeoff and landing) to limit distraction (no irrelevant conversations) during these sensitive steps , minimize room traffic, reduce overall noise).

Embodiments described herein can provide identification of beneficial and detrimental team interactions, as well as design and validation of simulated team training scenarios.

The functioning of a team can be affected by non-technical skills such as communication. Nontechnical skills were also associated with patient outcomes. Therefore, identifying specific patterns of behavior within the team that are beneficial or detrimental to patient outcomes is a step that may require subsequent development of specific team training interventions and debriefing sessions. Therefore, the core will use the data generated through OR black box observations to identify specific patterns of the team's non-technical performance. This information can be used as a basis for designing specific team interventions using OR simulations, role-plays, and debriefing sessions. Recurring topics identified as affecting team performance at the organizational level can be addressed through policy recommendations and the design of SOPs.

End users of this data can be all inter-professional OR teams. Instructional interventions derived from black-box data will be designed as instructional packages for interdisciplinary team training. Behavioral patterns identified as causing disruption to organizational processes will be addressed through policy changes at the local and regional levels.

Embodiments described herein may help to improve current and/or previous designs. For example, embodiments described herein can provide scalability. Additional equipment can be added to the configuration without undue costly hardware and cabling. As another example, embodiments described herein may provide optimization. They are likely to improve the ability to resolve variable physical spaces and add additional capture areas for a wider range of event chains. As yet another example, embodiments described herein may provide increased content with a greater ability to add additional data types for richer content. As additional examples, embodiments described herein may provide improved device synchronization to reduce reliance on expensive hardware encoders, increase accuracy, and reduce exposure to latency. Embodiments described herein can provide greater leverage of general-purpose computing devices and reduce overall platform costs.

Embodiments of the devices, systems and methods described herein may be implemented in a combination of hardware and software. The embodiments can be implemented on programmable computers, each computer including at least one processor, a data storage system (including volatile memory or nonvolatile memory or other data storage elements or combinations thereof), and at least one communication interface .

Program code is applied to input data to perform the functions described herein and generate output information. Output information is applied to one or more output devices. In some embodiments, the communication interface may be a network communication interface. In embodiments where elements may be combined, the communication interface may be a software communication interface, such as an interface for inter-process communication. In other embodiments, there may be a combination of communication interfaces implemented as hardware, software, and combinations thereof.

Throughout the foregoing discussion, numerous references will be made to servers, routers, portals, platforms, or other systems formed of computing device hardware. A computing device may have at least one processor configured to execute software instructions stored on a computer-readable tangible, non-transitory medium. For example, a server may include one or more computers operating as a web server, database server, or other type of computer server in a manner to fulfill described roles, responsibilities, or functions.

This description provides a number of example embodiments. While each embodiment represents a single combination of elements of the invention, other examples can include all possible combinations of the disclosed elements. Thus, if one embodiment includes elements A, B, and C, and a second embodiment includes elements B and D, the remaining combinations of A, B, C, or D may also be used.

The term "connected" or "coupled to" may include direct coupling (where two elements that are coupled to each other contact each other) and indirect coupling (where at least one additional element is located between the two elements).

The technical solutions of the embodiments may be in the form of software products. The software product may be stored on a non-volatile or non-transitory storage medium, which may be a compact disc read-only memory (CD-ROM), a USB flash drive or a removable hard drive. The software product comprises several instructions enabling a computer device (individual computer, server or network device) to carry out the method provided by the embodiments.

Although the embodiments have been described in detail, it should be understood that various changes, substitutions and alterations can be made in different embodiments.

Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the process, machine, manufacture, composition of matter, means, methods and steps described in the specification. Those of ordinary skill in the art will readily appreciate from this disclosure that currently existing or future developed processes that perform substantially the same function or achieve substantially the same results as the corresponding embodiments described herein can be utilized , machine, manufacture, composition of matter, means, method or step.

As can be appreciated, the examples described and illustrated above are intended to be exemplary only.

Claims (33)

1. A system for collecting and processing medical or surgical data comprising: A plurality of hardware units for collecting real-time medical or surgical data streams, the hardware units having control interfaces coupled to cameras, sensors, audio equipment, and patient monitoring hardware via a network, the real-time medical or surgical Data streams related to real-time medical procedures within the surgical field or clinical field; device middleware and hardware for converting, connecting and formatting said stream of real-time medical or surgical data received independently from said hardware unit; an encoder having a web server for synchronizing and recording the real-time medical or surgical data stream to a common clock or timeline to generate a session container file; a network infrastructure connecting said encoders, said device middleware and hardware, and said hardware units; and The switching or gateway hardware of the virtual private network is used to transfer the session container file. 2. The system of claim 1, wherein the device middleware and hardware establish a secure and reliable connection using the network infrastructure for communicating with the encoder and the hardware unit. 3. The system according to claim 1, wherein said device middleware and hardware use a clock-synchronized network protocol between said hardware units to achieve data consistency and accuracy of said real-time medical or surgical data stream synchronization to assist the encoder in generating the session container file. 4. The system of claim 1 , wherein the encoder, and the device middleware and hardware are operable to interface with third-party devices to receive additional data feeds as the real-time medical or surgical data stream a part of. 5. The system of claim 1 , further comprising a central control station accessible using the control interface, the control station configured to respond to commands including play/pause, stop session, record session, move to session frame, Input controls including segmentation displays, logging status indicators and log files to control the processing of the data stream. 6. The system of claim 1, wherein the network infrastructure provides increased failover and redundancy for the real-time medical or surgical data streams from the hardware units. 7. The system of claim 1, further comprising a storage area network for storing a data container file of the real-time medical or surgical data stream until scheduled for transmission. 8. The system of claim 1, wherein the encoder implements identity anonymization and encryption of the medical or surgical data. 9. The system of claim 1, wherein the encoder processes the real-time medical or surgical data stream to generate measurement metrics related to the medical procedure. 10. The system of claim 1, wherein the real-time medical or surgical data stream is associated with a timeline, wherein the encoder detects the real-time medical or surgical procedure at a corresponding time on the timeline An event within a data stream, and a tag and a timestamp of said session container file having said event, said timestamp corresponding to a time on said timeline. 11. The system of claim 1, further comprising a smart dashboard interface for annotating and tagging the synchronized stream of medical or surgical data, wherein the smart dashboard enables a viewer with playback viewing , to review the content and interface controls for marking it up. 12. The system of claim 11, wherein the smart dashboard is multidimensional in that the union of all dimensional variables for a medical procedure can indicate a specific set of one or more applicable annotation dictionaries or coding templates . 13. The system of claim 12, wherein example variables that can be used to determine the annotation and markup dictionary can be: the type of medical procedure being performed, the aspect of the procedure being analyzed, where the The geographic area/area of the process. 14. A multi-channel encoder for collecting, integrating, synchronizing and recording streams of medical or surgical data received from multiple hardware units onto a single interface with a common timeline or clock stream as an independent real-time or live data stream, the encoder has a web server for scheduling the transmission of the recorded session file container, the encoder processes the medical or surgical data stream to Generate measurement metrics related to real-time medical procedures. 15. The encoder of claim 14, wherein the encoder uses a lossless compression operation to generate a single session delivery file as output. 16. The encoder of claim 15, wherein the encoder detects completion of recording of the data stream and securely encrypts the single transport file. 17. The encoder of claim 14, wherein the encoder implements identity anonymization of the medical or surgical data. 18. The encoder of claim 14, said data streams comprising audio, video, text, metadata, quantitative, semi-quantitative, and data feeds. 19. A method for collecting and processing medical or surgical data comprising: receiving a plurality of live or real-time independent input feeds from one or more data capture devices located in an operating room or other patient intervention area at a multi-channel encoder, the input feeds being related to a live or real-time medical procedure; synchronizing said plurality of live independent input feeds via said encoder onto a single interface with a common timeline or clock; use a web server to record the synchronized input feed; generating an output session file by the encoder using the synchronized input feed; and The output session file is transmitted using the web server. 20. The method of claim 19, further comprising processing the data stream for identity anonymization. 21. The method of claim 19, further comprising routing the data stream to the encoder using a switch router. 22. A cloud-based system for collecting and processing medical or surgical data comprising: An encoder with a control interface for triggering the collection of real-time streams of medical or surgical data through smart devices including cameras, sensors, audio equipment, and patient monitoring hardware in response to receipt of control commands Surgical data pertains to real-time medical procedures within a surgical or clinical site, the encoder is used to authenticate the smart device, and the smart device synchronizes all The real-time medical or surgical data stream, the time stamp mark is generated by each smart device through the device clock; a media management hub server having middleware and hardware for converting, connecting, formatting and recording said stream of real-time medical or surgical data to generate a session container file on a network accessible storage device; a wireless network infrastructure for providing a secure network connection between said encoder, said smart device and said media management hub server for communicating said real-time medical or surgical data stream; a central content server for storing and distributing said session container files and providing a bi-directional communication interface to said media management hub to effectuate a file delivery handshake for said session container files; and Switching or gateway hardware of a virtual private network tunnel for transferring said session container file from said media management hub to said central content server. 23. The cloud-based system of claim 22, wherein the media management hub server broadcasts clock data to the smart devices for synchronizing the device clocks. 24. The cloud-based system of claim 22, wherein the encoder provides a user interface to receive the control commands and display a real-time visual representation of the medical or surgical data. 25. The cloud-based system of claim 22, wherein the media management hub server aggregates, packages, compresses and encrypts the real-time data streams to generate the session container file. 26. The cloud-based system of claim 22, wherein the media management hub server manages the smart devices based on location, schedule, region, and requirements. 27. The cloud-based system of claim 22, wherein the media management hub server receives operational status data from the smart device to generate a management interface having a visual representation of the operational status data of the smart device , the operational status data includes online, offline, run capture and onboard storage. 28. The cloud-based system of claim 27 , wherein the media management hub server processes the operating state data to detect smart devices operating outside of normal conditions, and in response generates all smart devices operating outside of normal conditions. Alert notifications for detected smart devices. 29. The cloud-based system of claim 22, wherein the media management hub server implements a device communication interface for the smart device to implement a device data transfer handshake for the real-time medical or surgical data stream. 30. The cloud-based system of claim 22, wherein the media management hub server authenticates the smart device. 31. The cloud-based system of claim 22, further comprising a Computational Intelligence Platform for receiving the session container file to build an analytical model to identify predictive, cost, and security concerns within the session container file The analysis model provides a network for extracting features, correlations, and event behaviors from the session container file involving a multivariate data set with time-varying parameters. 32. The cloud-based system of claim 22, further comprising a training or education server to receive the session container file, process the session container file to identify root causes of adverse patient outcomes, and generate a training interface to Training data is communicated using the identified root cause and the session container file. 33. The cloud-based system of claim 22, wherein the smart device comprises a motion tracking device for markerless motion tracking of objects within the surgical field or clinical field, the system further comprising a processor , the processor configured to convert the captured motion data from the motion tracking device into data structures identifying artifacts, workflow designs, and chains of events.