Table of Contents
Jetstream is a Jibo Platform Service that mediates a skill's interaction with the Hub (it replaces the ASRservice in Jibo V1). In this role it must handle requests from skills to recognize speech, it must engage onboard recognizers for HJ spotting, name learning and speaker ID, and it must pass along proactive trigger information from the skills to the Hub and back. The genesis of Jetstream's basic structure is visible in the Jetstream Flow Diagram at the end of this document. Jetstream presents a URL-based API to skills via HTTP requests and Websocket messages. For the HTTP requests, the parameters are supplied as a JSON string in the body of a POST request to the given URL. Likewise the HTTP responses and Websocket messages supply their information to the skill as a JSON payload in the body of the response or message. In this document, red colored text with a ⚠ symbol indicates a feature that should not be counted on due to being either not-yet-implemented or likely to change soon. Herewith is a thematically arranged list of all the HTTP-based commands Jetstream offers. A local turn is the means by which a skill requests speech recognition services. The word "local" here refers to a turn started by the skill, in contrast to turns started when a user says "HJ + phrase" that have a (global) reach beyond the skill. Since turns are long operations that involve waiting for user speech, they run asynchronously after they're requested. The skill will post a turn request to Jetstream, which is acknowledged without delay and which causes Jetstream to begin the process of connecting and sending audio to the Hub to perform speech recognition. As the turn progresses, Jetstream posts messages about its status or conclusion to Start a local turn, it remains active until it reports a status of SUCCEEDED, FAILED, INTERRUPTED, REJECTED, CANCELED, or TIMEDOUT via a TURN_RESULT message on the name type Description nluRules string[] Names of rules that should be activated for this turn. If this list is empty, the text output of the ASR step is not parsed for meaning and is simply returned to the skill. ignoreHJ boolean If true and HJ occurs during this turn, ignore it (rather than interrupt the turn). (default: false) name type Description string A unique ID for this turn, which will be included with any status information about it posted to Update (that is, force the completion of) an in-progress local turn with the given NLU or ASR text information. This function is provided to allow a menu button push to simulate NLU input to the skill. When invoked, the audio being streamed to the Hub is stopped and the Hub accepts the provided ASR or NLU data as input. There is a potential race condition where the given ASR/NLU data is sent to the Hub but the Hub (which is also waiting for results from the Cloud ASR and Cloud Parser) receives the Cloud ASR or NLU result first, which causes the update command to be ignored. name type Description requestID string The requestID of the in-progress turn to be updated. If the turn for the given requestID is no longer active, this call is ignored. The semantic ("Natural Language Understanding") result desired from this turn. This will be the same JSON schema that arrives as the data.result.nlu field in a successful TURN_RESULT message, e.g. Similar to Update Local Turn, the skill provides ASR text or NLU values with which to simulate a global turn (i.e. as if the user had said "HJ" followed by the given ASR text or an utterance that resulted in the given NLU values). This function is provided to allow cloud skills to be launched from the robot. name type Description The semantic result desired from this turn. This will be the same JSON schema that arrives as the data.result.nlu field in a successful TURN_RESULT message, e.g. name type Description string A unique ID for this turn, which will be included with any status information about it posted to Cancel Local Turn and Cancel Any Turn allow turns currently in progress to be cancelled. When cancelled, a status message will appear on Cancel the turn matching the given requestID. URL Parameters name type Description requestID string The requestID of the turn to cancel Response NONE (empty JSON) Cancel any turn that is in progress, including a global turn (one started by the user uttering "HJ"). URL Parameters NONE (empty JSON) Response NONE (empty JSON) Grammars for handling user speech of the form "HJ+phrase" are installed and removed through Subscribe and Unsubscribe requests. Normally the set of grammars used for any particular "HJ+phrase" is the union of the grammars of all the active subscriptions. However, the "exclusive" parameter, when set, will cause only those subscriptions so marked to be active (i.e. the subscriptions not marked exclusive will be hidden by the ones marked exclusive). Create a subscription that contributes the given grammars to the set of all active global grammars. URL Parameters name type Description nluRules string[] A list of grammars to be added to the set of grammars with which to recognize HJ+phrase requests Response name type Description string A unique ID for this subscription, which can be used to later unsubscribe it. Remove a particular subscription, given by its ID, from the set of active subscriptions. URL Parameters name type Description string A ID of the subscription to be removed. Response NONE (empty JSON) Removes all subscriptions. URL Parameters NONE (empty JSON) Response NONE (empty JSON) Speaker voice enrollment consists of creating models of each speaker's voice by asking them to say HJ some number of times. In order for the skill to control this process there are commands for removing existing models, collecting HJ samples (aka "utterances"), and creating models from the previously collected samples. The usual workflow for a skill using the enrollment API is to start an enrollment turn for a particular speaker that collects their utterances. The skill can request any number of utterances, but it would typically be six (or just a single one if recreating the Jibo V1 UX). Once the turn begins (indicated by a TURN_STARTED event) the speaker says each HJ separated by at least a second. After each HJ an event is emitted reporting the utterance quality and the utterance number (out of the total requested) that this represents. When the requested number of acceptable HJs have been collected, a TURN_RESULT event is emitted (status=SUCCEEDED) and the enrollment turn is finished. Even if an enrollment turn is cancelled or interrupted, the acceptable utterances collected so far for each speaker remain as "pending" utterances in the system. The Finally, the skill requests that a new model be created for a given speaker from all the utterances that have been collected for them and (optionally) the utterances used to create the existing model. This returns a status of either SUCCEEDED or FAILED. This is the speech sample collection step for voice enrollment. It is designed so that the user can say several HJs in succession. Every time an HJ is said, a message is posted to In the case of a catastrophic failure of the enrollment recognizer the TURN_RESULT message will have a status of FAILED and the data.message field will contain the reason for the failure. URL Parameters name type Description numberOfUtterances number The number of (good) HJs the user must say before the turn reports SUCCEEDED speakerID string An arbitrary string that the skill provides to denote a particular speaker. This string will be used to identify the utterances collected here when the speaker model is created. Response name type Description string A unique ID for this turn, which will be included with any status information about it posted to This creates a speaker mode from a) the pending samples collected during the enrollment turn(s) for a particular speaker and –if append is true– b) the model samples used in the creation of the previous model. The SpeakerID provided can refer to any speaker from whom a sufficient number of samples is available (i.e. the sum of those pending and –if append is true– those already in the model). Currently at least 4 samples a required from any particular speaker. After the model is created, the pending samples used to create the model are moved into the so-called model category to join (append=true) or replace (append=false) the other model utterances there, if any. In this way the model utterances always represent the set of utterances used to build the most recently created model. By setting append to true in subsequent model creation operations, the samples collected then will be used together with (i.e. appended to) all the ones used to make the current model. URL Parameters name type Description speakerID string The ID of the speaker whose model is being created (the same value used for the speaker's enrollment turns). Response/Code name type Description string Error message (e.g. "Insufficient Utterances") This removes the speaker model and all the model utterances for a given speaker. The speaker is effectively removed from the Speaker ID system. The pending utterances for a given speaker are not affected (they can be removed with /enroll/remove_pending_samples) URL Parameters name type Description speakerID string The speaker ID for the model to be removed. Response/Code name type Description string Error message (e.g. "No Such Speaker") This removes the pending utterances collected for a given speaker (i.e those not yet made into a model). The speaker model and the model utterances are not affected (they can be removed with /enroll/remove_speaker_model) URL Parameters name type Description speakerID string The speaker ID for the pending samples to be removed. Response/Code name type Description string Error message (e.g. "No Such Speaker") This returns the number of good utterances collected so far for a given speaker (but not yet built into a model, the so-called pending_utterances) and the number of utterances used in the existing model for the speaker (the so-called model_utterances). URL Parameters name type Description speakerID string The ID of the speaker for whom the count of utterances is requested. Response name type Description number The number of utterances that have been collected so far for the given speaker but not yet built into a model. This returns the speaker IDs for all the speakers for which models have been created. URL Parameters NONE (empty JSON) Response name type Description Also called "name learning", Jetstream provides methods for finding the user's pronunciation of a name. The workflow for skills consists of just two steps: initializing the system with the name to be learned and then asking the system to listen to the user say the name and return an estimate of the way it was pronounced. This prepares the system for listening to the way a user pronounces a particular word. It builds data structures that will be used during the listening step. For normal length names this completes in under 50ms, but longer names can take several hundred ms. URL Parameters name type Description Response name type Description string A unique ID that is created every time This is the pronunciation estimation step where the user speaks the name to be learned. It first processes the incoming audio with a speech endpointer to determine when the user is speaking. Then, constrained by a phonetic graph built from the spelling of the word, it asks a recognizer to return the best path through that graph given what was said. The result is a (probabilistic) phonetic spelling of what it heard the user say. URL Parameters name type Description word_to_learn string This must be identical to the word provided in the init_pronunciation_learning step. If not, an error is returned. ignoreHJ boolean If true, any utterances of HJ will be ignored during this turn. If false, HJ will have its normal effect (usually interrupting the turn in progress and starting a global turn). (default: false) Response name type Description string A unique ID for this turn, which will be included with any status information about it posted to Normally when a user says HJ, Jetstream will interrupt whatever is in progress and begin a "Global Turn", sending the audio following the HJ to the Hub for speech-to-text and parsing, and then returning a possible skill switch directive from the Hub if what has been said matches one of the active global grammars. However, Jetstream can be set to handle HJ in other ways by using Set HJ Mode. When this is done the new setting stays in effect until the robot is rebooted or the setting is changed again. This sets the HJ handling mode. There are 3 ways of handling HJ in Jetstream: URL Parameters name type Description One of three possible values: None (empty json) This is used to discover Jetstream's current HJ handling mode. Note that because of the way Jetstream works it is possible to set the mode and then get the mode within a short enough interval (<10 milliseconds) that the change has not had time to take effect and in this case the old mode will be reported. URL Parameters None (empty json) Response name type Description Proactive Triggering is the way that Jibo "wakes up" the Hub to find out whether there is some interaction it can offer the user. Since the Hub cannot contact Jibo, it is up to Jibo to contact the Hub. Trigger does not take any JSON parameters itself, rather the entire body of the HTTP request is sent, verbatim, to the Hub. Internally Trigger does request the Jibo Context from Be and sends that to the Hub right after the information provided in the Trigger request. URL Parameters None – the contents of the HTTP request's body is passed directly to the Hub Response name type Description string A unique ID for this trigger, which will be included with Trigger response messages posted to To enable test and debug efforts, Jetstream offers a few commands for recording audio from Jibo's "ASRin" audio source or inserting audio from an external file so that Jetstream uses it in lieu of microphone input. This makes a recording from Jibo's "ASRin" audio device, which is actually the output of the Jibo AudioService (the Audio Service provides a single, noise-cancelled, virtual audio source derived from Jibo’s six individual microphones). The audio's sampling rate is 16kHz with 16 bit, little-endian samples. While recording, the audio is stored in a file. When the Retrieve Recording command is issued, the recording process is stopped and the recorded audio is returned. URL Parameters name type Description When true, recording will begin as soon as the command is issued, when false, recording will begin when HJ is recognized. This stops the recording (if in progress) and returns the audio and cue markers from the recording. URL Parameters None (empty json) A JSON object containing the base64 encoded audio of the recording and cue markers, as follows: Note that the Jetstream repo includes a "record-from-robot.py" utilitiy for making robot recordings from the Mac that places the recording's cue points into the wave file it creates. These cue points will be displayed as labeled markers in Mac audio editors like "TwistedWave": For testing purposes this command allows you to temporarily replace Jibo's normal microphone audio input with audio data you supply. The supplied data is fed to the robot at the same rate as normal audio input, so (in theory) this should be indistinguishable from normal audio input. Problems arise however when using TTS synthesized audio since the background noise from the microphone is quite different from the background noise of TTS (which is essentially zero noise) and the switching between the two noise levels when the external audio is introduced and removed causes recognition problems. In practice we have found that noise must be mixed with TTS audio for the best recognition results. Our noise source was simply a recording made from Jibo in a quiet room. Use of this command while audio from a previous command is still being used simply results in the new audio being queued for use after the existing audio finishes. In addition to the audio, it is possible to specify pre- and post- insertion delay times that will cause a delay before the next audio in the queue is used (microphone input is used during the delay period). This can be useful for experimenting with the size of the gap in e.g. "HJ <gap> <phrase>" or other speech strings, without having to make new recordings. URL Parameters name type Description the base64 encoded audio data at 16kHz and 16 bits per sample. This command will stop any currently active mimicked input. It returns the sample number of the last sample given to the robot's audio system. This allows you to find the spot where playing left off in the file that was presented Note that any queued "mimicked input" is not affected and after this command runs, the next item in the queue (if any) will be feed to the audio system. URL Parameters None (empty json) Jetstream reports the status and conclusion of most of its commands via Websocket messages through the URL Jetstream does not accept any input on its websocket ports and will ignore what is sent to them. All Jetstream WS messages have these fields in common. Name Parameters name type Description the name of the message e.g. "TURN_RESULT" During the processing of a turn, these messages can appear. This message is sent as a status update for all turns (local, global, enrollment, and pronunciation learning). For local and global turns this indicates that a connection to the Hub has been successfully established and that audio is being sent. For enrollment and pronunciation learning turns, this indicates that the underlying recognizer has been engaged and is processing user audio. Name Parameters NONE Upon receiving a Hub notification that its speech recognizer has reported Start of Speech, this message is sent. Name Parameters NONE Upon receiving a Hub notification that its speech recognizer has reported End of Speech, this message is sent. In response to that Hub notification, Jetstream stops sending audio to it. Name Parameters NONE This message is sent at the conclusion of a turn and the Name Parameters name type Description The reason the turn concluded, one of: Hub Error messages cover a lot of ground: Jetstream itself can produce them in response to Proactivity Trigger requests if it fails to open a Hub connection or if an already open Hub connection goes down. Or the Hub can ask them to be produced as a response to a previous request (e.g. LISTEN or TRIGGER). In the case of Jetstream-sourced messages only the "data.message" parameter is present, which gives the reason for the failure. In the case of a Hub-promulgated message, the data object is a verbatim copy of the JSON field the Hub supplied. Name Parameters name type Description There are a class of messages the Hub produces that start with the string "SKILL_". These are directly transcribed into Jetstream messages containing a verbatim copy of the JSON from the original Hub message. Name Parameters name type Description These messages correspond to events that occur in the onboard recognizers. HJ Heard is emitted as soon as the onboard recognizer is confident it has heard someone say HJ, this may actually happen before the "b" in Jibo. Name HJ Only is emitted when the Hub reports that no speech after the HJ was detected or recognized. The delay between the time the HJ was said and this event is emitted is determined by the HubAsr.global_sosTimeout_sec setting in the config file. Name Speaker ID is emitted only if there are enrolled speakers. Name An array of SpeakerRecogResult objects (sorted in descending score order) with one entry for each enrolled speaker that states how well that speaker was recognized. A Speaker Enrollment message is emitted after every HJ recognition during an enrollment turn. Name Jetstream has a built-in voice activity detector that emits websocket messages at 20hz to the endpoint The system time corresponding to the beginning of the results An array of VADevent elements. Each corresponds to a 20ms span of audio. Since the VAD uses a 20ms frame internally and Jetstream uses a 50ms frame (that is, always giving the VAD 50ms of audio at a time), the length of the VADevents array will alternate between being two and three elements long. This is the probability that there is voice in each 20ms of audio. These probabilities are float values from approx. -1.0 to approx. +1.0, where 0.0 can be seen as the recommended threshold for the no-voice (<0) or voice (>0) decision. To lower sensitivity, increase the threshold (and vice versa). Sensible ranges for a threshold are between -0.3 and + 0.4. The closer the values are to -1.0 and +1.0, the more certain the decision as to whether the audio is voice or not-voice. As described in the prior section, Jetstream provides information to skills via the This port shows all the events being sent to the Listener Loop (Jetstream's central process). Looking on the Jetstream debug page while it is running, we see output like this in the listen_loop window: Each line is the JSON representation of a message that the Listener Loop has received. The state the Listener Loop was in when the message was received is placed in the "state" property (hjw: HJ Wait, phw: Phrase Wait, plw: Pronunciation Learning Wait, enw: Enrollment Wait). In the list above we can see how an utterance of "Hey Jibo, do you like penguins?" has become 7 events submitted to the Listener Loop: Monitoring the timestamps here can show how long these events are taking to arrive. These two ports log all the messages Jetstream is sending to the Hub and (respectively) that the Hub is sending to Jetstream. This is what they produced during the same interaction above: The LISTEN message indicates to the Hub that recognition is requested. It is followed by a CONTEXT message holding information about the state of the robot obtained from the Be "Context" service. In this case the service was not running, so the runtime field is "null". After the LISTEN and CONTEXT messages, the audio will begin being sent. The audio messages are of type binary (which is how the hub distinguishes them from non-audio messages) and are not sent to the hub_outgoing port. The timestamps in the messages appearing in hub_incoming are generated on the Hub, so a way to get a sense of the time skew between the two systems is to look at e.g. the difference between the timestamp on the SOS message from he Hub and the one on the (related) SOS message being posted to the Listener Loop. In this case they report being 20ms apart (1514742225709 (Jetstream message) - 1514742225689 (Hub msg) = 20). Jetstream was originally developed from the following diagram showing how Local and Global turns are handled. Further functions and turn types have since been added to Jetstream (i.e. Enrollment and Pronunciation Learning), but the basic structure still holds and serves as a useful sketch of its operation. Source: link And this concludes our tour of the Jetstream system. Comments and corrections are welcome.Service Description
The Jetstream HTTP API
Local Turns: Skill-driven ASR
/events.Start Local Turn
/events Websocket. URL /listen/start_local_turn
Parameters ignoreGlobalRules boolean If true, global rules are not active during the turn, if false, the global rules (other than the launch rule) are active in addition to the provided nluRules. (default: false) rejectIfBusy boolean If true and there is already a turn in progress (global, local, enrollment or pronunciation learning) then reject this turn, otherwise interrupt the turn in progress and run this one instead. (default:false) language string ISO language label, e.g. "en-US" (default: "") meta string Arbitrary JSON data to pass along to the hub. (default: "") earlyEOS string[] A list of words and phrases that will be used to detect a valid response from incremental ASR results (i.e. prior to EOS and grammar parsing). hintPhrases string[] A list of hint phrases to add to the ASR vocabulary clientNLU NLUResult The semantic result desired from this turn. If present, no audio will be sent to the hub. See this entry in update_local_turn for further details. clientASR string The ASR text result desired from this turn. If present, no audio will be sent to the hub. See this entry in update_local_turn for further details. sosTimeout float The period of time to wait (in seconds) for the Start Of Speech after hearing HJ. If HJ is not heard within this period, the turn completes with a status of TIMEDOUT and message of "sos". If this property is set to -1 (or not provided) the value used is the one found in the jibo-jetstream-service.json file with the property name "local_sosTimeout_sec", which is currently 10 seconds. (default: -1) maxSpeechTimeout float The maximum number of seconds that a user may speak after saying HJ. If the user speaks for more than this number of seconds, the turn completes with a status of TIMEDOUT and message of "maxSpeech". As with sosTimeout, a value of -1 causes the config file property "local_maxSpeechTimeout_sec" to be used, which is currently 60 seconds. (default: -1) Response requestID /events and which can later be used to cancel it.Update Local Turn
URL /listen/update_local_turn
Parameters meta string Arbitrary JSON data to pass along to the hub. clientNLU NLUResult "nlu" : {
"entities" : {},
"intent" : "generalWhatQuestions",
"rules" : [
"launch"
]
}NLUResult name type description rules string[] listen rules to activate intent string intent entities object a map of EntityNames (strings) to EntityValues (which are either string or string[]) clientASR string The ASR text to be used to produce the NLU result from this turn. This is a simple string of words, equivalent to what the ASR returns. Only one of the clientASR and clientNLU properties should be provided, if both are provided only the clientNLU property will be used. Response: NONE (empty JSON) Mimic Global Turn
URL /listen/mimic_global_turn
Parameters language string ISO language label, e.g. "en-US" suppressedEvents string[] A list of events to suppress during this turn, e.g. ["HJ_HEARD", "TURN_STARTED"]. All events that can be emitted during a global turn can be suppressed. And note that only those events associated with this global turn can be suppressed. If, for example, someone says HJ during a mimic_global_turn, that HJ_HEARD event will not be suppressed, even if HJ_HEARD is among the events to be suppressed during this turn. clientNLU NLUResult "nlu" : {
"entities" : {},
"intent" : "generalWhatQuestions",
"rules" : [
"launch"
]
}NLUResult name type description rules string[] listen rules to activate intent string intent entities object a map of EntityNames (strings) to EntityValues (which are either string or string[]) clientASR string The ASR text to be used to produce the NLU result from this turn. This is a simple string of words, equivalent to what the ASR returns. It should not include the "Hey Jibo" text. Only one of the clientASR and clientNLU properties should be provided, if both are provided only the clientNLU property will be used. Response requestID /events and which can later be used to cancel it.Cancelling Turns
/events indicating the turn was cancelled. Enrollment and Pronunciation Learning turns are also cancelled this way. If there was no turn in progress (or if it completed before the cancel was executed) then the cancel request is ignored and no status is given. Cancel Local Turn
/listen/cancel_local_turn
Cancel Any Turn
/listen/cancel_any_turn
Global Subscriptions: Handling "HJ + phrase"
Subscribe Global
/listen/subscribe_global
exclusive boolean When true, only the subscriptions marked exclusive will be active. language string an ISO language tag, e.g. "en-US" requestID Unsubscribe Global
/listen/unsubscribe_global
requestID Unsubscribe All Globals
/listen/unsubscribe_all_globals
Speaker Voice Enrollment
/enroll/get_utterance_count method may be used to find out how many utterances a speaker has, both those for use in creating a new model (pending utterances) and of those that were used to create the existing model (model utterances).Start Enrollment Turn
/events giving the status of the utterance (i.e. whether it was acceptable or not). After the requested number of (acceptable) HJs is given, the turn will complete with a result of SUCCEEDED. Acceptable utterances collected during an enrollment turn are collected into a set of pending utterances that exist until a model is created from them. /listen/start_enrollment_turn
rejectIfBusy boolean If true and there is already a turn in progress (global, local, enrollment or pronunciation learning) then reject this turn, otherwise interrupt the turn in progress and run this one instead. requestID /events and which can later be used to cancel it (via cancel_local_turn)Create Speaker Model
/enroll/create_speaker_model
append bool When true, append the pending utterances to the model utterances and use both sets to create the model. When false, only the pending utterances are used to create the model. Default: false HTTP 200 Empty JSON HTTP 400 message Remove Speaker Model
/enroll/remove_speaker_model
HTTP 200 Empty JSON HTTP 400 message Remove Pending Samples
/enroll/remove_pending_samples
HTTP 200 Empty JSON HTTP 400 message Get Utterance Count
/enroll/get_utterance_count
pending_utterances model_utterances number The number of utterances that were used to build the current model. Get Enrolled Speakers
/enroll/get_enrolled_speakers
speakers string[] A list of the speaker IDs of the enrolled speakers. Pronunciation Learning
Initialize Pronunciation Learning
/pronunciation/init_pronunciation_learning
word_to_learn string The word that the system will try to learn the pronunciation of. requestID init_pronunciation_learning is called that is included in messages and audio files logged during subsequent pronunciation learning turns. Its sole purpose is to enable post-hoc speech science experiments by allowing the initialization of the pronunciation learning system and the audio data logged in subsequent turns to be tied together.Start Pronunciation Learning Turn
/listen/start_pronunciation_learning_turn
rejectIfBusy boolean If true and there is already a turn in progress (global, local, enrollment or pronunciation learning) then reject this turn, otherwise interrupt the turn in progress and run this one instead. (default: false) requestID /events and which can later be used to cancel it (via cancel_local_turn)Altering how HJ is handled
Set HJ Mode
/listen/set_hj_mode
mode string Response Get HJ Mode
/listen/get_hj_mode
mode string The current setting of Jetstream's HJ handling mode. Proactive Triggering
Trigger
/proactive/trigger
requestID /events. Test and Debug Commands
Start Recording
/debug/start_recording
start_immediately boolean max_recording_duration_ms number Sets the maximum length of time (in ms) that the recording will proceed. Default is 2000 ms. Retrieve Recording
/debug/retrieve_recording
Response name type description recording RetrievedRecording Retrieved Recording name type description base64_audio string binary audio data in base64 format. start_time number always 0 end_time number duration of the recording in ms cue_points CuePoint[] an array of CuePoints, described below. CuePoint name type description name string the name of the point, one of HJ_BEG, HJ_END, EP_BEG, or EP_END time number the time point (number of milliseconds into the recording) where the cue point occurs. 
Mimic Mic Input
/debug/mimic_mic_input
audio string start_delay_ms number the number of ms to wait before using the supplied audio finish_delay_ms number the number of ms to wait after using the supplied audio before using the next audio in the queue. Stop Mimicking Input
/debug/stop_mimicked_input
Response name type Description stopping_point number the sample number at which audio input to the robot stopped (the first sample in the file that was not sent to the robot – accurate to ~50ms) The Jetstream Websocket API
/events. The Jetstream Service Debug page (http://172.24.84.101:8090/index.html) displays the messages being sent to /events as well as messages from several other debug-related URLs.WSmsg: the base class
(base class)
type string requestID string if this is in response to a command, this will be the requestID Jetstream returned when the command was issued. transID string if this is related to a Hub transaction, this is the transaction ID Jetstream used with the Hub during that transaction. ts number the time in milliseconds since the Unix epoch data object Jetstream messages that need to provide information beyond the base class properties populate a "data" object with the information. The data object schema is message dependent. Turn-related Messages
Turn Started
TURN_STARTED
Start of Speech
SOS
End of Speech
EOS
Turn Result
status field contains the reason it concluded. In the case where the status is SUCCEEDED, the recognition result from the turn is also provided, in the case where the status is FAILED, the error message giving the failure reason is provided. In the case of TIMEDOUT the message field will be either "sos" or "maxSpeech". In all other status cases no further information is provided. All turns are guaranteed to conclude with a Turn Result message of some sort.TURN_RESULT
data.status string rejectIfBusy set to true and another turn is already in progress) data.message string (If present) In the case of status==FAILED or TIMEDOUT this contains the reason for the failure or timeout. data.message string (If present) In the case of status==SUCCEEDED and a Pronunciation Learning turn, this contains the estimated pronunciation of what the user said. data.result object (If present) In the case of status==SUCCEEDED (and not a Pronunciation Learning turn) this contains the "result" object from the Hub. Hub-related Messages
Hub Error
HUB_ERROR
data.message string (If present) The problem Jetstream encountered trying to communicate with the Hub. data object (If present) The Hub's error information it supplied in response to a LISTEN or TRIGGER request. SKILL_*
SKILL_*
data object The JSON the Hub supplied. Onboard Recognizer-related Messages
HJ Heard
HJ_HEARD
Parameters NONE HJ Only
HJ_ONLY
+ Parameters NONE Speaker ID
SPEAKER_ID
Parameters name type Description data.speakers SpeakerRecogResult[] SpeakerRecogResult name type description speaker string The ID string used to enroll this speaker. score number authentication score (not useful without knowing the threshold) accepted boolean If true, this speaker's score was high enough to be considered "recognized" high_confidence boolean If true, this speaker's score was either high enough that the system is confident in accepting it, or low enough that it is confident in rejecting it. data.snr number The estimated signal-to-noise ratio of the HJ input used in the speaker recognition operation. Speaker Enrollment
SPEAKER_ENROLLMENT
Parameters name type Description data.speaker string The ID of the speaker being enrolled. data.snr number The estimated SNR of the HJ utterance. data.accepted boolean True if the utterance quality was acceptable. If this is false, the utterance being reported on was rejected. data.problems string[] If data.accepted is false, this is a string vector containing the problem(s) with the utterance. The possible values are: CLIPPING, POOR_SNR, NOT_SPEECH_LIKE, NOT_CONSISTENT, BAD_DURATION, BAND_LIMITED, POOR_QUALITY. data.good_utterance_count number The number of utterances accepted so far during this turn. data.total_utterance_count number The total number of utterances seen so far during this turn. data.final boolean True if the number of acceptable utterances matches the numberOfUtterances parameter specified when the enrollment turn was started. After this the turn will complete and no further Speaker Enrollment messages will appear. Voice Activity Detector events
/vad. These messages indicate the degree to which the sound that Jibo's microphones are picking up resembles human speech. Name VAD_EVENT_DATAParameter Type Description audio_time uint64_t avg_frame_db float The average power of the 50ms frame of audio that was used to produce the VAD results. VADevents VADevent[] VADevent name type description time uint64_t The system time corresponding to the audio that produced this result. vad float Jetstream's Debugging and Testing Facilities
The
/debug/* Websocket ports/events websocket. To assist debugging and testing Jetstream also produces WS events on 3 other ports, The output from these ports is visible on the Jetstream debug page. We describe them below.The Listener Loop monitor:
/debug/listenloopThe Hub monitors:
/debug/hub_outgoing and /debug/hub_incominghub_outgoing (from Jetstream to the Hub)
hub_incoming (from the Hub to Jetstream)
Jetstream Flow Diagram
.png%3Fversion=1&modificationDate=1526940533000&api=v2)
Summary
-
+
+
- + + + + +
+ Add Comment +
+