1.1 Problem Statement
Public safety field operations have challenges with accessing information and controlling equipment in many situations because their hands are not available to control a mobile data terminal, smartphone, or a Land Mobile Radio (LMR). It is paramount that advanced user-interfaces provide “Heads-Up Hands-Free” operation. A law enforcement officer needs to have two hands on their weapon when firing it. They can’t afford to have one hand occupied by a mobile device. A firefighter typically has gloves on at the scene, and will also need their hands to manipulate handheld tools and hoses. Again, they cannot afford to have their hands occupied by a mobile device. Finally, an EMS technician needs their hands to treat a patient and manipulate monitoring equipment. There are many instances across the spectrum of public safety where hands-free capabilities can improve the operational efficiency of emergency responders.
A voice-activated virtual assistant could enable capabilities to overcome these challenges. Consumer virtual assistants are becoming widely used for hands-free access to information and device control in very creative ways and this technology could be leveraged for the benefit of public safety. However, because consumer virtual assistants have limitations in the level of customization available to public safety users, a fully customized and optimized virtual assistant may be a very valuable advanced user-interface for public safety users.
The following are typical use cases where an emergency responder would benefit from having access to a virtual assistant:
- Suspicious Vehicle: An officer is patrolling an area on foot and needs to identify the owner and history of a suspicious vehicle. The virtual assistant could be used to allow the officer to verbally request the information using the license plate and/or VIN number while maintaining situational awareness (not having to look down at a screen) while maintaining free use of their hands. If they engage with the vehicle owner, they could use a virtual assistant to update dispatch that they have a traffic engagement in progress and to request driver license information. They could also use voice commands to request backup support or the location of other patrol officers in the area.
- Patient Information: An EMT or firefighter needs to obtain valuable patient data to provide better care. The virtual assistant would allow the EMT to request information about the patient and it would verbally convey the results. This allows the EMT to maintain use of their hands to administer support to the patient as needed.
- Commercial Structure Fire: A fire service team is dispatched to a multiple alarm fire at a commercial building. While in route, the team lead on the fire apparatus uses voice commands to request pre-plan information and previous dispatch history for the address of the fire. A voice command is also used by the team lead to request dispatch details which is read back to the full apparatus team including information on other responding units and their current location. The apparatus engineer uses a voice command to request the route to the dispatch location and a map is displayed within the vehicle showing the route. Additional voice commands can be used to automatically update dispatch or incident status such as leaving station, arrived on site, fire attack active, to rehab, dispatch complete.
Each participant will be expected to deliver a customized voice-controlled virtual assistant enabled application that can support law enforcement, fire services, and EMS emergency responders. Specifically, the expected solutions could include, certainly not limited to, the following:
- Support of a hierarchical contact list to support communication, location, and status reporting capability to individuals or groups within a public safety authority
- Integration with other virtual assistants to take advantage of their ecosystems
- Support an end-to-end solution including integration with back-end applications and data sources
- Optimized UX for efficient processing of emergency responder command vocabulary along with streamlined dialog
- Ability to function within challenging operational environments
- Provide a user-centered design based on public safety feedback
- Embrace cybersecurity requirements and best-practices throughput the development process (CJIS, HIPAA, etc.)
In this section, we will provide resources that we believe will be useful to all participants. We have provided resources that give context and structure to the virtual assistant product space, as well as example commercial solutions. In addition, we have provided references to data sources that should be leveraged for this challenge. Finally, we provide a collection of references to SDKs and APIs that we believe would be the foundation for the development of a solution.
Figure 1. A canonical model for how to organize and manage a system based on voice interaction (https://medium.muz.li/voice-user-interfaces-vui-the-ultimate-designers-guide-8756cb2578a1)
The model shown in Figure 1 represents an approach for thinking about the voice processing structure of a virtual assistant. We provide this as a reference for terminology used in this field and as context for your design.
Commercial Virtual Assistants
These virtual assistants are available to commercial users. We do not view these as “public safety grade” applications, but rather we provide these links to serve as reference locations to foster informed ideation.
- Mastermind: http://www.mastermindassistant.com
- Dragon Mobile: https://www.nuance.com/mobile/mobile-applications/dragon-mobile-assistant.html#standardpage-mainpar_multicolumn
- Hound: https://www.soundhound.com/hound
- Lyra: https://www.heylyra.com/
The following are data resources that should be used by participants to demonstrate an end-to-end integrated solution:
- License Plate Reader Database
- Driver License Database
- Patient Information Database
- Fire Pre-plan and Building Information Database
- Dispatch History
Cross-Platform cloud-based speech recognition for iOS, Android, and Web applications.
- iOS/Android (companion): https://developer.amazon.com/docs/alexa-voice-service/authorize-companion-app.html
- Web (companion): https://developer.amazon.com/docs/alexa-voice-service/authorize-companion-site.html
- iOS/Android (on product — no smart speaker): https://developer.amazon.com/docs/alexa-voice-service/authorize-on-product.html
Apple Siri Kit
Provides cloud-based speech recognition services for iOS applications.
- Main site: https://developer.apple.com/sirikit/
- Developers Documentation: https://developer.apple.com/documentation/sirikit
Provides offline cross-platform speech recognition that can be embedded into iOS and Android applications.
- Main site: https://www.politepix.com/openears/
Google Voice Actions
Provides cloud-based speech recognition services for Android applications.
- Main site: https://developers.google.com/voice-actions/
Developers Documentation: https://developers.google.com/voice-actions/system/
2. Evaluation criteria
Participants must adhere to the basic application requirements listed below. Failure to do so may result in non-grading of the application.
2. EVALUATION CRITERIA
In this section, information is provided about the expected development phases and the evaluation criteria to be used during the duration of the challenge.
2.1 Development Phases
The evaluation criteria being applied to the following phases of development throughout the challenge:
- Phase 1: Baseline assessment and implementation of core functionality using commercial and/or custom voice assistant and associated artificial intelligence solutions
- Phase 2: Optimization for emergency responder vocabulary and efficient access and dialog
- Phase 3: End-to-end integration with apps and data sources
- Phase 4: Engagement with public safety for assessment and recommendations for optimization of the solution and the preferred support of devices types and associated accessories.
2.2. Evaluation Criteria
The following section provides details on the evaluation criteria and judging areas which will be utilized as part of the challenge evaluation process.
Criteria #0: Core Functionality Checklist (Basic Requirements)
- The solution must provide both a manual and voice prompt (wake word) triggering mechanism to initiate the voice interaction sequence.
- The solution must provide a hierarchical contact list and group management functionality for control of information exchange.
- The solution must provide basic text messaging and email communication capabilities through voice control.
- The solution must provide the ability to exchange location information through voice control.
Criteria #1: Breadth and Depth of Public Safety Skill Development
- The breadth of skill support across each category of public safety users will be assessed against the recommended vocabulary support. These public safety disciplines include:
- General (common skills used across all disciplines)
- Law Enforcement
- Incident Command
- The depth of skill support within a single discipline will be assessed in accordance with the list of recommended vocabulary items.
- Some solutions may provide a wide range of support and others may focus on one discipline for a deep level of support in this area, and the scoring process will accommodate both options.
Criteria #2: Level of Public Safety Customization and Effectiveness in Operational Environments
- Level of customization to provided efficient use within emergency responder operations
- Streamline the virtual assistant vocabulary and dialog
- Flexibility to handle a range of utterances for the same intent to minimize amount of user training required
- Optimized triggering (manual, voice prompt, gesture) and dialog options for seamless and efficient use
- Capability to interrupt or modify the sequence of a dialog which is in progress
- Voice command accuracy when operating in challenging noise environments encountered by public safety. The accuracy levels for a range of voice types will be evaluated for the following conditions:
- Voice command accuracy in nominal environment
- Voice command accuracy in noisy law enforcement environments (Siren, vehicle noise, large crowd)
- Voice command accuracy in noisy fire service and EMS environments (Siren/Horn, Chainsaw, Fire Ground, Traffic Accident/Jaws of Life)
Criteria #3: Innovation and Creativity
- Development of innovative and complex skills beyond the items listed in the recommended skill set
- Integration of peripherals providing advanced audio processing capabilities or far field microphone technologies to address challenging operational environments
- Interworking with peripherals within the voice assistant ecosystem including smart speakers, displays, headsets, and other device accessories
- Audio-to-text conversion and language translation support
- Complementary use of other technologies — IoT for equipment control, gesture or other advanced UI capabilities
Criteria #4: Application and Data Integration
- Level of integration with back-end applications and data sources. These may include the following areas of application and data access support:
- Computer Aided Dispatch (CAD) including dispatch history information
- Department of Motor Vehicles (DMV) license plate information
- Driver License Information Database
- Local, regional, and national criminal history
- EMS medication data
- Patient treatment protocols
- Situational Awareness platforms (location, group status)
- Support unique security aspects to allow voice control access to sensitive databases and applications. The security concerns fall into the following areas:
- CJIS for law enforcement criminal database access
- HIPAA for medical record access
- Local agency enterprise security requirements
- Development of sample APIs to support scalable access to applications and databases through voice control
Criteria #5: Emergency responder Usability and Optimization
- Usability of solution to be assessed by public safety personnel in realistic operational and training scenarios
- Optimization of the end-to-end voice interaction process is considered to support:
- Highly efficient dialog to minimize time to complete voice command actions
- Streamlined process for data access
- Flexibility in the range of utterances for the same voice command intent
- Usability improvements and optimizations incorporated to accommodate recommendations from user trials
- The ability of the solution to be demonstrated in a real operational environment with live applications and databases.
3. EXPECTED DELIVERABLES FROM PARTICIPANTS
Review the How to Participate instructions in section 3 of this document to ensure that you are prepared for the in-person Regional Codeathon or Online Contest. The following deliverables will need to be included with the submission:
- A completed submission form through techtoprotectchallenge.org
- A 3-minute narrated PowerPoint file or 3-minute narrated video with the following sections:
- A summary of the submission.
- Basic information about the participant or participant’s team.
- Specific requirements of the contest that are being addressed by the participants.
- Two to four screenshots of the solutions or prototype.
- Overview of key functions included in the submission.
- Any available files or links to the developed solution or prototype.
- Any additional files based on the contest description.