Compare commits

...

7 Commits

Author SHA1 Message Date
Jacob Dubin
ab47ad7a2d jibo clock family skills and feature backlog updates 2026-04-20 22:25:08 -05:00
Jacob Dubin
2ea7afe2e7 added cloud versioning 2026-04-20 22:13:37 -05:00
Jacob Dubin
efdb5bcf01 jibo news skill by voice 2026-04-20 22:09:23 -05:00
Jacob Dubin
3a150faf4b backup yes/no path improvements 2026-04-20 22:03:17 -05:00
Jacob Dubin
d28d735d16 expanding backlog 2026-04-20 22:00:07 -05:00
Jacob Dubin
7b05452937 more Jibo patches 2026-04-20 21:45:55 -05:00
Jacob Dubin
32d63584d6 mapping radio 2026-04-20 20:55:49 -05:00
14 changed files with 1154 additions and 18 deletions

View File

@@ -8,6 +8,12 @@ It is intentionally broader than the current Node server. The Node server is a p
Day-to-day feature sequencing now lives in [feature-backlog.md](/C:/Projects/JiboExperiments/OpenJibo/docs/feature-backlog.md).
Cloud release hygiene:
- keep a visible OpenJibo Cloud version string
- expose it through diagnostics such as `/health` and the spoken `cloud version` command
- bump the shared version constant whenever we deploy a meaningful hosted-cloud change
## Current Scope
- stable .NET cloud scaffold
@@ -158,6 +164,26 @@ Latest stock-OS WOD findings:
- Spoken WOD guesses should preferentially snap to the closest offered hint when Whisper lands very close to one of the menu words, since near-misses like `haglet` for `aglet` are common in live testing.
- The stock robot still misroutes constrained local turns if the cloud echoes `globals/*` rules back on the reply. For spoken WOD guesses and settings/update `no`, we should only return the local rule (`word-of-the-day/puzzle`, `settings/download_now_later`, etc.) so Global Service does not relaunch Nimbus.
Latest radio discovery findings:
- `@be/radio` is a true local skill, not a cloud placeholder.
- Its `open(result, refresh, previousSkillName)` path treats `result.nlu.intent === "menu"` as a `play` launch.
- `result.nlu.entities.station` is the genre selector, and `Country` is a real supported station key from the robot's `genres.json`.
- The smallest stock-shaped cloud handoff for voice launch is therefore a local `SKILL_REDIRECT` to `@be/radio` with `nlu.intent = "menu"`, optional `entities.station`, and a silent completion to settle the hotphrase cloud response.
Latest news discovery findings:
- Nimbus explicitly treats `match.cloudSkill === "news"` like the GQA path and waits on `cloudSkillResponse`.
- The first OpenJibo news pass should therefore use a real cloud-skill shape, not a generic placeholder chat reply.
- For now, the content can stay synthetic while the protocol is grounded: `match.cloudSkill = "news"` plus a supported `SLIM` announcement response is enough to validate the robot path before provider-backed headlines arrive later.
Latest clock discovery findings:
- `@be/clock` is a real local skill with `clock`, `timer`, and `alarm` domains.
- Menu launches use `intent = "menu"` with `entities.domain` set to the target sub-area.
- Direct timer and alarm actions use `timerValue` and `alarmValue` utterances, not a generic chat path.
- A practical first OpenJibo slice is therefore: keep custom spoken time/date answers for now, but route `open clock`, `open timer`, `open alarm`, `set a timer ...`, and `set an alarm ...` through stock-shaped local `@be/clock` handoffs.
## Speech, Animation, And ESML
The current joke flow is only a small foothold into Jibo expressiveness.

View File

@@ -40,6 +40,7 @@ Parallel tags:
- Current evidence:
- [index.js](C:/Projects/JiboOs/V3.1/build/opt/jibo/Jibo/Skills/@be/be/node_modules/@be/radio/index.js) resumes from `lastStation`
- the same file treats `menu` as a `play` launch and reads `result.nlu.entities.station`
- the same file confirms `menu + no station` is the clean resume path and `menu + station=Country` becomes a direct genre launch
- Implementation notes:
- add phrase routing for radio open/resume and genre launch
- inspect radio genre and station metadata before locking the outbound entity values
@@ -88,6 +89,7 @@ Parallel tags:
- the attached `jibo test 13` session includes both examples in one bundle:
- a proactive or share-style prompt where spoken `yes` was treated as generic speech
- a later update prompt where spoken `no` was accepted correctly
- the share prompt uses `surprises-date/offer_date_fact` with `$YESNO`, and the failing reply leaked `globals/*` rules back into a Nimbus relaunch
- Implementation notes:
- compare the active listen rules, ASR hints, and local skill ownership for the share-style prompt versus OTA prompts
- make constrained yes-no detection cover this prompt family without regressing the already-working update `no` path
@@ -109,21 +111,29 @@ Parallel tags:
- Implementation notes:
- decide whether the first pass is a simple headline summary or a closer personal-report style payload
- confirm whether stock OS expects `news` as a dedicated cloud skill or under the broader personal-report family
- Latest progress:
- first pass should use Nimbus's supported cloud path by setting `match.cloudSkill = news` and returning a supported `SLIM` announcement
- provider-backed headlines can follow later under the `Lasso / Knowledge And Event Aggregation` track
- Exit criteria:
- `tell me the news` reaches a non-placeholder live path
- robot behavior feels Nimbus-native rather than generic chat playback
### 6. Clock Family Audit
- Status: `ready`
- Status: `in_progress`
- Tags: `protocol`
- Why now: clock, date, timer, and alarm menu hooks are already visible in captures and the robot repo has a real `@be/clock` skill.
- Current evidence:
- [protocol-inventory.md](C:/Projects/JiboExperiments/OpenJibo/docs/protocol-inventory.md) already tracks menu intents for `askForTime`, `askForDate`, `timerValue`, and `alarmValue`
- `@be/clock` exists in the robot skill inventory
- `JiboOs` shows `@be/clock` branches on `entities.domain = clock | timer | alarm`, uses `intent = menu` for menu launches, and accepts direct `timerValue` / `alarmValue` utterances with structured entities
- Implementation notes:
- compare our custom time/date path against actual menu payloads
- decide whether timer and alarm should stay robot-local with cloud acknowledgement, or whether cloud needs to shape the launch and follow-up turns
- Progress so far:
- voice `open clock`, `open timer`, and `open alarm` now synthesize stock-shaped local `@be/clock` launches
- voice `set a timer for five minutes` and `set an alarm for 7:30 am` now emit direct `timerValue` / `alarmValue` payloads with the domain and value entities the local skill expects
- time/date remain on the existing custom cloud reply path for now
- Exit criteria:
- time/date behavior stays correct
- timer and alarm launch or set correctly from both menu and voice where applicable
@@ -161,7 +171,25 @@ Parallel tags:
- what payload shape triggers the local animation / embodiment layer
- whether the first pass should be cloud speech only or forecast plus presentation metadata
### 9. Surprises Routing
### 9. Proactivity Selector And Surprise Offers
- Status: `discovery`
- Tags: `protocol`, `content`, `docs`
- Why later: the original architecture and recent proactive captures suggest proactivity is a first-class cloud subsystem, not just ordinary chat that starts itself.
- Current evidence:
- the attached original Jibo architecture diagram shows a cloud-side `Proactivity Selector`, `Proactivity Catalog`, and robot-side proactive trigger plumbing
- [jibo test 13.txt](C:/Projects/JiboExperiments/artifact-output/jibo-test-13/jibo%20test%2013.txt) and its websocket artifacts show a proactive-style `I have something to share with you` offer and later proactive `TRIGGER` traffic
- `@be/surprises`, `@be/surprises-date`, and `@be/surprises-ota` already exist as local robot-side building blocks
- Questions to answer:
- what minimum cloud-side selector we need for stock-OS-compatible surprise offers
- how proactive `TRIGGER` traffic should map into a hosted OpenJibo proactivity service
- whether `surprises-date/offer_date_fact` should be the first end-to-end proactive offer we intentionally support
- Implementation notes:
- model proactivity as its own orchestrator separate from ordinary conversational turn routing
- include offer, constrained yes/no, fulfillment, and dismissal behavior in the design
- preserve the artifact linkage to the original architecture diagram and `jibo-test-13`
### 10. Surprises Routing
- Status: `discovery`
- Tags: `protocol`, `content`
@@ -174,7 +202,40 @@ Parallel tags:
- which categories still depend on cloud services versus fully local logic
- whether stock OS `1.9` differs materially from the `3.1` source snapshot here
### 10. Personal Report, Calendar, And Commute
### 11. History / Memory Layer
- Status: `discovery`
- Tags: `content`, `docs`
- Why later: the original architecture explicitly calls out `History`, and that likely maps to the kind of durable personal memory we want for names, preferences, and remembered facts.
- Current evidence:
- the attached original Jibo architecture diagram includes a dedicated `History` component in cloud storage
- stock Jibo behavior historically included awareness of names, birthdays, holidays, and special dates
- Questions to answer:
- what data belongs in memory versus account/profile versus skill-specific storage
- how much of the original behavior was robot-local versus cloud-backed
- what the first safe OpenJibo memory slice should be
- Implementation notes:
- plan for person identity, preferred name, birthday, relationship facts, and notable dates
- keep the first design privacy-aware and easy to host
- treat this as shared infrastructure that other skills can consume rather than a standalone feature
### 12. Lasso / Knowledge And Event Aggregation
- Status: `discovery`
- Tags: `content`
- Why later: the original architecture diagram suggests `Lasso` sits between the hub and outside data sources, which likely explains how Jibo knew about news, calendar items, holidays, and other structured world events.
- Current evidence:
- the attached original Jibo architecture diagram shows `Lasso` connected to 3rd-party data such as AP News, Dark Sky, GCalendar, Wolfram, and other external sources
- stock Jibo behavior historically covered holidays, birthdays, special events, and topical knowledge
- Questions to answer:
- whether `Lasso` should be recreated as a single aggregation service or as several focused providers behind a shared interface
- which parts are needed for news, weather, calendar, commute, astrology/date facts, and holidays
- what subset is practical for a hosted OpenJibo v1
- Implementation notes:
- treat holidays and special dates as first-class backlog scope here
- use this item to drive future provider work for news, weather, calendar, commute, and event awareness
### 13. Personal Report, Calendar, And Commute
- Status: `discovery`
- Tags: `protocol`, `content`
@@ -185,10 +246,19 @@ Parallel tags:
- Questions to answer:
- should calendar and commute be independent feature paths or sections inside personal report
- what minimum provider data shape lets Jibo present these naturally
### 14. Stop Command
- Status: `ready`
- Tags: `protocol`
- Why later: Jibo can be interrupted by any command, but it would be nice to have a dedicated "stop" type of command.
- Current evidence:
- There is an Idle skill or subskill under @be so I think we can utilize it, but I am not sure if that was the default behavior.
- Questions to answer:
- Can we find in the original source evidence for this skill or stop word phrase?
## Support Tracks
### 11. Hosted Capture And Storage Plan
### 15. Hosted Capture And Storage Plan
- Status: `ready`
- Tags: `docs`
@@ -197,7 +267,7 @@ Parallel tags:
- define a clean boundary between local capture sinks and hosted archival/export
- document how group testers should submit sessions without touching repo paths directly
### 12. STT Upgrade And Noise Screening
### 16. STT Upgrade And Noise Screening
- Status: `ready`
- Tags: `stt`
@@ -219,6 +289,9 @@ Parallel tags:
6. Clock family
7. Photo family
8. Weather
9. Surprises
10. Personal report, calendar, and commute
11. Hosted capture/storage and STT improvements as parallel tracks
9. Proactivity selector and surprise offers
10. Surprises
11. History / memory layer
12. Lasso / knowledge and event aggregation
13. Personal report, calendar, and commute
14. Hosted capture/storage and STT improvements as parallel tracks

View File

@@ -6,6 +6,13 @@
This is the production-oriented path for restoring device connectivity and creating a foundation for future runtime, AI, and OTA work.
Current spoken cloud version: `Open Jibo Cloud version 1.0.10.`
Release hygiene reminder:
- bump [OpenJiboCloudBuildInfo.cs](/C:/Projects/JiboExperiments/OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Application/Services/OpenJiboCloudBuildInfo.cs) whenever we ship a meaningful hosted-cloud update
- keep the spoken version response and `/health` version field aligned from that single source of truth
## Architecture
The first implementation is a modular monolith:

View File

@@ -125,7 +125,12 @@ app.Use(async (context, next) =>
await telemetrySink.RecordConnectionClosedAsync(closeEnvelope, closeSession, $"socket-loop-ended{(isPrematureClose ? "-prematurely" : string.Empty)}", context.RequestAborted);
});
app.MapGet("/health", () => Results.Json(new { ok = true, service = "OpenJibo Cloud Api" }));
app.MapGet("/health", () => Results.Json(new
{
ok = true,
service = "OpenJibo Cloud Api",
version = OpenJiboCloudBuildInfo.Version
}));
app.MapMethods("/{**path}", ["GET", "POST", "PUT"], async (HttpContext context, JiboCloudProtocolService service, IProtocolTelemetrySink telemetrySink, CancellationToken cancellationToken) =>
{

View File

@@ -17,5 +17,6 @@ public sealed class JiboExperienceCatalog
public IReadOnlyList<string> CalendarReplies { get; init; } = [];
public IReadOnlyList<string> CommuteReplies { get; init; } = [];
public IReadOnlyList<string> NewsReplies { get; init; } = [];
public IReadOnlyList<string> NewsBriefings { get; init; } = [];
public IReadOnlyList<string> GenericFallbackReplies { get; init; } = [];
}

View File

@@ -72,6 +72,14 @@ public sealed class DemoConversationBroker(JiboInteractionService interactionSer
{
"word_of_the_day" => false,
"word_of_the_day_guess" => false,
"radio" => false,
"radio_genre" => false,
"clock_menu" => false,
"timer_menu" => false,
"alarm_menu" => false,
"timer_value" => false,
"alarm_value" => false,
"news" => false,
_ => true
};
}

View File

@@ -1,6 +1,7 @@
using Jibo.Cloud.Application.Abstractions;
using Jibo.Runtime.Abstractions;
using System.Text.Json;
using System.Text.RegularExpressions;
namespace Jibo.Cloud.Application.Services;
@@ -29,6 +30,15 @@ public sealed class JiboInteractionService(
"dance" => BuildDanceDecision(catalog),
"time" => new JiboInteractionDecision("time", $"It is {DateTime.Now:h:mm tt}."),
"date" => new JiboInteractionDecision("date", $"Today is {DateTime.Now:dddd, MMMM d}."),
"day" => new JiboInteractionDecision("day", $"Today is {DateTime.Now:dddd}."),
"cloud_version" => new JiboInteractionDecision("cloud_version", OpenJiboCloudBuildInfo.SpokenVersion),
"radio" => BuildRadioLaunchDecision(),
"radio_genre" => BuildRadioGenreLaunchDecision(lowered),
"clock_menu" => BuildClockLaunchDecision("clock", "Opening the clock."),
"timer_menu" => BuildClockLaunchDecision("timer", "Opening the timer."),
"alarm_menu" => BuildClockLaunchDecision("alarm", "Opening the alarm."),
"timer_value" => BuildTimerValueDecision(lowered),
"alarm_value" => BuildAlarmValueDecision(lowered),
"hello" => new JiboInteractionDecision("hello", randomizer.Choose(catalog.GreetingReplies)),
"how_are_you" => new JiboInteractionDecision("how_are_you", randomizer.Choose(catalog.HowAreYouReplies)),
"yes" => new JiboInteractionDecision("yes", "Yes."),
@@ -40,7 +50,7 @@ public sealed class JiboInteractionService(
"weather" => new JiboInteractionDecision("weather", randomizer.Choose(catalog.WeatherReplies)),
"calendar" => new JiboInteractionDecision("calendar", randomizer.Choose(catalog.CalendarReplies)),
"commute" => new JiboInteractionDecision("commute", randomizer.Choose(catalog.CommuteReplies)),
"news" => new JiboInteractionDecision("news", randomizer.Choose(catalog.NewsReplies)),
"news" => BuildNewsDecision(catalog),
_ => new JiboInteractionDecision("chat", BuildGenericReply(catalog, transcript, lowered))
};
}
@@ -73,6 +83,22 @@ public sealed class JiboInteractionService(
});
}
private JiboInteractionDecision BuildNewsDecision(JiboExperienceCatalog catalog)
{
var briefing = randomizer.Choose(catalog.NewsBriefings);
return new JiboInteractionDecision(
"news",
briefing,
"news",
new Dictionary<string, object?>(StringComparer.OrdinalIgnoreCase)
{
["skillId"] = "news",
["cloudSkill"] = "news",
["mim_id"] = "runtime-news",
["mim_type"] = "announcement"
});
}
private string BuildGenericReply(JiboExperienceCatalog catalog, string transcript, string lowered)
{
if (string.IsNullOrWhiteSpace(transcript))
@@ -130,6 +156,33 @@ public sealed class JiboInteractionService(
return "date";
}
if (string.Equals(clientIntent, "askForDay", StringComparison.OrdinalIgnoreCase))
{
return "day";
}
if (string.Equals(clientIntent, "timerValue", StringComparison.OrdinalIgnoreCase))
{
return "timer_value";
}
if (string.Equals(clientIntent, "alarmValue", StringComparison.OrdinalIgnoreCase))
{
return "alarm_value";
}
if (string.Equals(clientIntent, "menu", StringComparison.OrdinalIgnoreCase) &&
clientEntities.TryGetValue("domain", out var clockDomain))
{
return clockDomain.ToLowerInvariant() switch
{
"clock" => "clock_menu",
"timer" => "timer_menu",
"alarm" => "alarm_menu",
_ => "chat"
};
}
if (MatchesAny(
loweredTranscript,
"word of the day",
@@ -151,6 +204,53 @@ public sealed class JiboInteractionService(
return "joke";
}
if (MatchesAny(
loweredTranscript,
"cloud version",
"open jibo cloud version",
"openjibo cloud version",
"what version is the cloud",
"what s the cloud version",
"what's the cloud version"))
{
return "cloud_version";
}
if (TryResolveRadioGenre(loweredTranscript) is not null)
{
return "radio_genre";
}
if (TryParseAlarmValue(loweredTranscript) is not null)
{
return "alarm_value";
}
if (TryParseTimerValue(loweredTranscript) is not null)
{
return "timer_value";
}
if (MatchesAny(loweredTranscript, "open the clock", "open clock", "show the clock", "show clock"))
{
return "clock_menu";
}
if (MatchesAny(loweredTranscript, "open the timer", "open timer", "show the timer", "show timer"))
{
return "timer_menu";
}
if (MatchesAny(loweredTranscript, "open the alarm", "open alarm", "show the alarm", "show alarm"))
{
return "alarm_menu";
}
if (MatchesAny(loweredTranscript, "open the radio", "play the radio", "turn on the radio", "radio"))
{
return "radio";
}
if (MatchesAny(loweredTranscript, "dance", "boogie"))
{
return "dance";
@@ -212,6 +312,11 @@ public sealed class JiboInteractionService(
return "time";
}
if (MatchesAny(loweredTranscript, "what day is it", "what day is today"))
{
return "day";
}
if (MatchesAny(loweredTranscript, "what day is it", "what is the date", "today s date", "today's date") ||
loweredTranscript.Contains("date", StringComparison.Ordinal) ||
loweredTranscript.Contains("day", StringComparison.Ordinal))
@@ -235,6 +340,84 @@ public sealed class JiboInteractionService(
});
}
private static JiboInteractionDecision BuildRadioLaunchDecision()
{
return new JiboInteractionDecision(
"radio",
"Opening the radio.",
"@be/radio",
new Dictionary<string, object?>(StringComparer.OrdinalIgnoreCase)
{
["skillId"] = "@be/radio"
});
}
private static JiboInteractionDecision BuildClockLaunchDecision(string domain, string replyText)
{
return new JiboInteractionDecision(
$"{domain}_menu",
replyText,
"@be/clock",
new Dictionary<string, object?>(StringComparer.OrdinalIgnoreCase)
{
["skillId"] = "@be/clock",
["domain"] = domain,
["clockIntent"] = "menu"
});
}
private static JiboInteractionDecision BuildTimerValueDecision(string loweredTranscript)
{
var timer = TryParseTimerValue(loweredTranscript) ?? new ClockTimerValue("0", "1", "null");
return new JiboInteractionDecision(
"timer_value",
"Setting your timer.",
"@be/clock",
new Dictionary<string, object?>(StringComparer.OrdinalIgnoreCase)
{
["skillId"] = "@be/clock",
["domain"] = "timer",
["clockIntent"] = "timerValue",
["hours"] = timer.Hours,
["minutes"] = timer.Minutes,
["seconds"] = timer.Seconds
});
}
private static JiboInteractionDecision BuildAlarmValueDecision(string loweredTranscript)
{
var alarm = TryParseAlarmValue(loweredTranscript) ?? new ClockAlarmValue("7:00", "am");
return new JiboInteractionDecision(
"alarm_value",
"Setting your alarm.",
"@be/clock",
new Dictionary<string, object?>(StringComparer.OrdinalIgnoreCase)
{
["skillId"] = "@be/clock",
["domain"] = "alarm",
["clockIntent"] = "alarmValue",
["time"] = alarm.Time,
["ampm"] = alarm.AmPm
});
}
private static JiboInteractionDecision BuildRadioGenreLaunchDecision(string loweredTranscript)
{
var station = TryResolveRadioGenre(loweredTranscript) ?? "Country";
return new JiboInteractionDecision(
"radio_genre",
$"Playing {FormatRadioGenreForSpeech(station)} on the radio.",
"@be/radio",
new Dictionary<string, object?>(StringComparer.OrdinalIgnoreCase)
{
["skillId"] = "@be/radio",
["station"] = station
});
}
private static JiboInteractionDecision BuildWordOfTheDayGuessDecision(
IReadOnlyDictionary<string, string> clientEntities,
string transcript,
@@ -301,6 +484,7 @@ public sealed class JiboInteractionService(
string.Equals(rule, "$YESNO", StringComparison.OrdinalIgnoreCase) ||
string.Equals(rule, "create/is_it_a_keeper", StringComparison.OrdinalIgnoreCase) ||
string.Equals(rule, "settings/download_now_later", StringComparison.OrdinalIgnoreCase) ||
string.Equals(rule, "surprises-date/offer_date_fact", StringComparison.OrdinalIgnoreCase) ||
string.Equals(rule, "surprises-ota/want_to_download_now", StringComparison.OrdinalIgnoreCase));
}
@@ -417,6 +601,176 @@ public sealed class JiboInteractionService(
{
return candidates.Any(candidate => loweredTranscript.Contains(candidate, StringComparison.Ordinal));
}
private static string? TryResolveRadioGenre(string loweredTranscript)
{
foreach (var (phrase, station) in RadioGenreAliases)
{
if (loweredTranscript.Contains(phrase, StringComparison.Ordinal))
{
return station;
}
}
return null;
}
private static string FormatRadioGenreForSpeech(string station)
{
return station switch
{
"EightiesAndNinetiesHits" => "eighties and nineties hits",
"ChristianAndGospel" => "Christian and gospel",
"ClassicRock" => "classic rock",
"CollegeRadio" => "college radio",
"HipHop" => "hip hop",
"NewsAndTalk" => "news and talk",
"ReggaeAndIsland" => "reggae and island music",
"SoftRock" => "soft rock",
_ => station
};
}
private static ClockTimerValue? TryParseTimerValue(string loweredTranscript)
{
if (!loweredTranscript.Contains("timer", StringComparison.Ordinal))
{
return null;
}
var hours = ExtractDurationValue(loweredTranscript, "hour");
var minutes = ExtractDurationValue(loweredTranscript, "minute");
var seconds = ExtractDurationValue(loweredTranscript, "second");
if (hours is null && minutes is null && seconds is null)
{
return null;
}
return new ClockTimerValue(
(hours ?? 0).ToString(),
(minutes ?? 0).ToString(),
seconds is null ? "null" : seconds.Value.ToString());
}
private static ClockAlarmValue? TryParseAlarmValue(string loweredTranscript)
{
if (!loweredTranscript.Contains("alarm", StringComparison.Ordinal))
{
return null;
}
var match = AlarmPattern.Match(loweredTranscript);
if (!match.Success)
{
return null;
}
var hourToken = match.Groups["hour"].Value;
var minuteToken = match.Groups["minute"].Success ? match.Groups["minute"].Value : "00";
var hour = ParseNumberToken(hourToken);
if (hour is null || hour is < 1 or > 12)
{
return null;
}
if (!int.TryParse(minuteToken, out var minute) || minute is < 0 or > 59)
{
return null;
}
var ampm = match.Groups["ampm"].Value.StartsWith("p", StringComparison.Ordinal) ? "pm" : "am";
return new ClockAlarmValue($"{hour}:{minute:00}", ampm);
}
private static int? ExtractDurationValue(string loweredTranscript, string unitStem)
{
var pattern = new Regex($@"\b(?<value>\d+|[a-z\-]+)\s+{unitStem}s?\b", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant);
var match = pattern.Match(loweredTranscript);
if (!match.Success)
{
return null;
}
return ParseNumberToken(match.Groups["value"].Value);
}
private static int? ParseNumberToken(string token)
{
var normalized = token.Trim().ToLowerInvariant();
if (int.TryParse(normalized, out var numeric))
{
return numeric;
}
return normalized switch
{
"a" or "an" => 1,
"one" => 1,
"two" => 2,
"three" => 3,
"four" => 4,
"five" => 5,
"six" => 6,
"seven" => 7,
"eight" => 8,
"nine" => 9,
"ten" => 10,
"eleven" => 11,
"twelve" => 12,
"thirteen" => 13,
"fourteen" => 14,
"fifteen" => 15,
"sixteen" => 16,
"seventeen" => 17,
"eighteen" => 18,
"nineteen" => 19,
"twenty" => 20,
"thirty" => 30,
"forty" => 40,
"fifty" => 50,
_ => null
};
}
private sealed record ClockTimerValue(string Hours, string Minutes, string Seconds);
private sealed record ClockAlarmValue(string Time, string AmPm);
private static readonly Regex AlarmPattern = new(
@"\b(?<hour>\d{1,2}|one|two|three|four|five|six|seven|eight|nine|ten|eleven|twelve)(?:[:\s](?<minute>\d{2}))?\s*(?<ampm>a\.?m\.?|p\.?m\.?)\b",
RegexOptions.IgnoreCase | RegexOptions.CultureInvariant | RegexOptions.Compiled);
private static readonly (string Phrase, string Station)[] RadioGenreAliases =
[
("country music", "Country"),
("country radio", "Country"),
("country", "Country"),
("classic rock", "ClassicRock"),
("soft rock", "SoftRock"),
("hip hop", "HipHop"),
("hip-hop", "HipHop"),
("news and talk", "NewsAndTalk"),
("news talk", "NewsAndTalk"),
("news radio", "NewsAndTalk"),
("sports radio", "Sports"),
("christian music", "ChristianAndGospel"),
("gospel music", "ChristianAndGospel"),
("oldies", "Oldies"),
("pop music", "Pop"),
("jazz", "Jazz"),
("latin music", "Latin"),
("dance music", "Dance"),
("reggae", "ReggaeAndIsland"),
("island music", "ReggaeAndIsland"),
("alternative", "Alternative"),
("blues", "Blues"),
("classical music", "Classical"),
("classical", "Classical"),
("college radio", "CollegeRadio"),
("comedy radio", "Comedy"),
("npr", "NPR")
];
}
public sealed record JiboInteractionDecision(

View File

@@ -0,0 +1,8 @@
namespace Jibo.Cloud.Application.Services;
public static class OpenJiboCloudBuildInfo
{
public const string Version = "1.0.10";
public static string SpokenVersion => $"Open Jibo Cloud version {Version}.";
}

View File

@@ -23,10 +23,26 @@ public sealed class ResponsePlanToSocketMessagesMapper
string.Equals(plan.IntentName, "no", StringComparison.OrdinalIgnoreCase);
var isWordOfDayLaunch = string.Equals(plan.IntentName, "word_of_the_day", StringComparison.OrdinalIgnoreCase);
var isWordOfDayGuess = string.Equals(plan.IntentName, "word_of_the_day_guess", StringComparison.OrdinalIgnoreCase);
var isRadioLaunch = string.Equals(plan.IntentName, "radio", StringComparison.OrdinalIgnoreCase) ||
string.Equals(plan.IntentName, "radio_genre", StringComparison.OrdinalIgnoreCase);
var isClockSkillLaunch = string.Equals(skill?.SkillName, "@be/clock", StringComparison.OrdinalIgnoreCase);
var clockIntent = ReadSkillPayloadString(skill, "clockIntent");
var clockDomain = ReadSkillPayloadString(skill, "domain");
var timerHours = ReadSkillPayloadString(skill, "hours");
var timerMinutes = ReadSkillPayloadString(skill, "minutes");
var timerSeconds = ReadSkillPayloadString(skill, "seconds");
var alarmTime = ReadSkillPayloadString(skill, "time");
var alarmAmPm = ReadSkillPayloadString(skill, "ampm");
var radioStation = ReadSkillPayloadString(skill, "station");
var cloudSkill = ReadSkillPayloadString(skill, "cloudSkill");
var nluGuess = ReadClientEntity(turn, "guess");
var wordOfDayGuess = ResolveWordOfDayGuess(turn, transcript, nluGuess);
var outboundIntent = isWordOfDayLaunch
? "menu"
: isRadioLaunch
? "menu"
: isClockSkillLaunch && !string.IsNullOrWhiteSpace(clockIntent)
? clockIntent
: isWordOfDayGuess
? "guess"
: string.Equals(messageType, "CLIENT_NLU", StringComparison.OrdinalIgnoreCase) && !string.IsNullOrWhiteSpace(clientIntent)
@@ -36,6 +52,10 @@ public sealed class ResponsePlanToSocketMessagesMapper
? wordOfDayGuess
: isWordOfDayLaunch
? string.Empty
: isRadioLaunch
? transcript
: isClockSkillLaunch
? transcript
: string.Equals(clientIntent, "guess", StringComparison.OrdinalIgnoreCase) && !string.IsNullOrWhiteSpace(nluGuess)
? nluGuess
: isYesNoTurn && isYesNoIntent
@@ -45,10 +65,31 @@ public sealed class ResponsePlanToSocketMessagesMapper
: transcript;
var outboundRules = isWordOfDayLaunch
? ["word-of-the-day/menu"]
: isRadioLaunch
? Array.Empty<string>()
: isClockSkillLaunch
? string.Equals(messageType, "CLIENT_NLU", StringComparison.OrdinalIgnoreCase) ? rules : Array.Empty<string>()
: isWordOfDayGuess
? ["word-of-the-day/puzzle"]
: isYesNoTurn && isYesNoIntent ? [yesNoRule!] : rules;
var entities = ReadEntities(turn, messageType, isYesNoTurn && isYesNoIntent, isWordOfDayLaunch, isWordOfDayGuess, wordOfDayGuess);
var entities = ReadEntities(
turn,
messageType,
isYesNoTurn && isYesNoIntent,
ShouldIncludeCreateDomain(yesNoRule),
isWordOfDayLaunch,
isRadioLaunch,
isWordOfDayGuess,
wordOfDayGuess,
radioStation,
isClockSkillLaunch,
clockDomain,
clockIntent,
timerHours,
timerMinutes,
timerSeconds,
alarmTime,
alarmAmPm);
var listenMessage = new
{
type = "LISTEN",
@@ -61,12 +102,20 @@ public sealed class ResponsePlanToSocketMessagesMapper
final = true,
text = outboundAsrText
},
nlu = BuildNluPayload(outboundIntent, outboundRules, entities, isWordOfDayLaunch ? "@be/word-of-the-day" : null),
nlu = BuildNluPayload(
outboundIntent,
outboundRules,
entities,
isWordOfDayLaunch ? "@be/word-of-the-day" :
isRadioLaunch ? "@be/radio" :
isClockSkillLaunch ? "@be/clock" :
null),
match = new
{
intent = outboundIntent,
rule = outboundRules.FirstOrDefault() ?? string.Empty,
score = 0.95
score = 0.95,
cloudSkill
}
}
};
@@ -100,6 +149,39 @@ public sealed class ResponsePlanToSocketMessagesMapper
DelayMs: 125));
}
if (isRadioLaunch)
{
messages.Add(new SocketReplyPlan(
JsonSerializer.Serialize(BuildSkillRedirectPayload(
transId,
"@be/radio",
outboundIntent,
outboundAsrText,
outboundRules,
entities)),
DelayMs: 75));
messages.Add(new SocketReplyPlan(
JsonSerializer.Serialize(BuildCompletionOnlySkillPayload(transId, "@be/radio")),
DelayMs: 125));
}
if (isClockSkillLaunch &&
!string.Equals(messageType, "CLIENT_NLU", StringComparison.OrdinalIgnoreCase))
{
messages.Add(new SocketReplyPlan(
JsonSerializer.Serialize(BuildSkillRedirectPayload(
transId,
"@be/clock",
outboundIntent,
outboundAsrText,
outboundRules,
entities)),
DelayMs: 75));
messages.Add(new SocketReplyPlan(
JsonSerializer.Serialize(BuildCompletionOnlySkillPayload(transId, "@be/clock")),
DelayMs: 125));
}
if (emitSkillActions && speak is not null)
{
messages.Add(new SocketReplyPlan(
@@ -240,13 +322,29 @@ public sealed class ResponsePlanToSocketMessagesMapper
private static object ReadEntities(
TurnContext turn,
string? messageType,
bool yesNoCreateTurn,
bool yesNoTurn,
bool includeCreateDomain,
bool wordOfDayLaunch,
bool radioLaunch,
bool wordOfDayGuess,
string? guess)
string? guess,
string? radioStation,
bool clockSkillLaunch,
string? clockDomain,
string? clockIntent,
string? timerHours,
string? timerMinutes,
string? timerSeconds,
string? alarmTime,
string? alarmAmPm)
{
if (yesNoCreateTurn)
if (yesNoTurn)
{
if (!includeCreateDomain)
{
return new Dictionary<string, object?>();
}
return new Dictionary<string, object?>
{
["domain"] = "create"
@@ -261,6 +359,41 @@ public sealed class ResponsePlanToSocketMessagesMapper
};
}
if (radioLaunch)
{
var entities = new Dictionary<string, object?>();
if (!string.IsNullOrWhiteSpace(radioStation))
{
entities["station"] = radioStation;
}
return entities;
}
if (clockSkillLaunch)
{
var entities = new Dictionary<string, object?>(StringComparer.OrdinalIgnoreCase);
if (!string.IsNullOrWhiteSpace(clockDomain))
{
entities["domain"] = clockDomain;
}
if (string.Equals(clockIntent, "timerValue", StringComparison.OrdinalIgnoreCase))
{
entities["hours"] = timerHours ?? "0";
entities["minutes"] = timerMinutes ?? "0";
entities["seconds"] = timerSeconds ?? "null";
}
if (string.Equals(clockIntent, "alarmValue", StringComparison.OrdinalIgnoreCase))
{
entities["time"] = alarmTime ?? string.Empty;
entities["ampm"] = alarmAmPm ?? string.Empty;
}
return entities;
}
if (wordOfDayGuess)
{
return new Dictionary<string, object?>
@@ -293,9 +426,16 @@ public sealed class ResponsePlanToSocketMessagesMapper
.FirstOrDefault(static rule =>
string.Equals(rule, "create/is_it_a_keeper", StringComparison.OrdinalIgnoreCase) ||
string.Equals(rule, "settings/download_now_later", StringComparison.OrdinalIgnoreCase) ||
string.Equals(rule, "surprises-date/offer_date_fact", StringComparison.OrdinalIgnoreCase) ||
string.Equals(rule, "surprises-ota/want_to_download_now", StringComparison.OrdinalIgnoreCase));
}
private static bool ShouldIncludeCreateDomain(string? yesNoRule)
{
return string.Equals(yesNoRule, "create/is_it_a_keeper", StringComparison.OrdinalIgnoreCase) ||
string.Equals(yesNoRule, "surprises-ota/want_to_download_now", StringComparison.OrdinalIgnoreCase);
}
private static IEnumerable<string> ReadRuleValues(TurnContext turn)
{
return ReadRuleValues(turn, "listenRules").Concat(ReadRuleValues(turn, "clientRules"));
@@ -677,8 +817,7 @@ public sealed class ResponsePlanToSocketMessagesMapper
.Replace("&", "&amp;", StringComparison.Ordinal)
.Replace("<", "&lt;", StringComparison.Ordinal)
.Replace(">", "&gt;", StringComparison.Ordinal)
.Replace("\"", "&quot;", StringComparison.Ordinal)
.Replace("'", "&apos;", StringComparison.Ordinal);
.Replace("\"", "&quot;", StringComparison.Ordinal);
}
private static string? ReadPayloadString(IDictionary<string, object?>? payload, string key)

View File

@@ -493,6 +493,24 @@ public sealed class WebSocketTurnFinalizationService(
turnState.FinalizeAttemptCount += 1;
}
if (allowFallbackOnMissingTranscript &&
turnState.BufferedAudioBytes >= AutoFinalizeMinBufferedAudioBytes &&
IsYesNoTurn(finalizedTurn))
{
turnState.AwaitingTurnCompletion = false;
session.LastTranscript = string.Empty;
session.LastIntent = null;
session.LastListenType = "no-input";
var localRule = ReadPrimaryYesNoRule(finalizedTurn);
var noInputReplies = ResponsePlanToSocketMessagesMapper.MapNoInput(
turnState.TransId ?? session.LastTransId ?? string.Empty,
string.IsNullOrWhiteSpace(localRule) ? turnState.ListenRules : [localRule])
.Select(map => new WebSocketReply { Text = map.Text, DelayMs = map.DelayMs })
.ToArray();
ResetBufferedAudio(session);
return noInputReplies;
}
if (allowFallbackOnMissingTranscript &&
turnState.BufferedAudioBytes >= AutoFinalizeMinBufferedAudioBytes &&
string.IsNullOrWhiteSpace(turnState.LastSttError))
@@ -544,6 +562,13 @@ public sealed class WebSocketTurnFinalizationService(
: DateTimeOffset.UtcNow.Add(WebSocketTurnState.DefaultLateAudioIgnoreWindow);
var emitSkillActions = !string.Equals(plan.IntentName, "word_of_the_day", StringComparison.OrdinalIgnoreCase) &&
!string.Equals(plan.IntentName, "radio", StringComparison.OrdinalIgnoreCase) &&
!string.Equals(plan.IntentName, "radio_genre", StringComparison.OrdinalIgnoreCase) &&
!string.Equals(plan.IntentName, "clock_menu", StringComparison.OrdinalIgnoreCase) &&
!string.Equals(plan.IntentName, "timer_menu", StringComparison.OrdinalIgnoreCase) &&
!string.Equals(plan.IntentName, "alarm_menu", StringComparison.OrdinalIgnoreCase) &&
!string.Equals(plan.IntentName, "timer_value", StringComparison.OrdinalIgnoreCase) &&
!string.Equals(plan.IntentName, "alarm_value", StringComparison.OrdinalIgnoreCase) &&
(messageType != "CLIENT_NLU" ||
string.Equals(plan.IntentName, "word_of_the_day_guess", StringComparison.OrdinalIgnoreCase));
var replies = ResponsePlanToSocketMessagesMapper.Map(plan, finalizedTurn, session, emitSkillActions).Select(map => new WebSocketReply
@@ -708,6 +733,18 @@ public sealed class WebSocketTurnFinalizationService(
string.Equals(rule, "$YESNO", StringComparison.OrdinalIgnoreCase) ||
string.Equals(rule, "create/is_it_a_keeper", StringComparison.OrdinalIgnoreCase) ||
string.Equals(rule, "settings/download_now_later", StringComparison.OrdinalIgnoreCase) ||
string.Equals(rule, "surprises-date/offer_date_fact", StringComparison.OrdinalIgnoreCase) ||
string.Equals(rule, "surprises-ota/want_to_download_now", StringComparison.OrdinalIgnoreCase));
}
private static string? ReadPrimaryYesNoRule(TurnContext turn)
{
return ReadRules(turn, "listenRules")
.Concat(ReadRules(turn, "clientRules"))
.FirstOrDefault(static rule =>
string.Equals(rule, "create/is_it_a_keeper", StringComparison.OrdinalIgnoreCase) ||
string.Equals(rule, "settings/download_now_later", StringComparison.OrdinalIgnoreCase) ||
string.Equals(rule, "surprises-date/offer_date_fact", StringComparison.OrdinalIgnoreCase) ||
string.Equals(rule, "surprises-ota/want_to_download_now", StringComparison.OrdinalIgnoreCase));
}

View File

@@ -66,6 +66,11 @@ public sealed class InMemoryJiboExperienceContentRepository : IJiboExperienceCon
"I heard your news request. That path is still a future cloud integration.",
"News is recognized, but I do not have the full news service behind it yet."
],
NewsBriefings =
[
"Here are your headlines. Space missions are preparing for new launches, climate and weather systems are staying active across the country, and AI tools keep pushing into everyday products.",
"Here is a quick news brief. Technology companies are still racing on AI, global leaders are trading policy updates, and science teams are sharing new research findings."
],
GenericFallbackReplies =
[
"Okay. You said, {transcript}.",

View File

@@ -115,6 +115,26 @@ public sealed class JiboInteractionServiceTests
Assert.Equal("No.", decision.ReplyText);
}
[Fact]
public async Task BuildDecisionAsync_SurprisesDateOfferPrompt_MapsShortAffirmationToYesIntent()
{
var service = CreateService();
var decision = await service.BuildDecisionAsync(new TurnContext
{
RawTranscript = "Yes!",
NormalizedTranscript = "Yes!",
Attributes = new Dictionary<string, object?>
{
["listenRules"] = new[] { "surprises-date/offer_date_fact", "globals/global_commands_launch" },
["listenAsrHints"] = new[] { "$YESNO" }
}
});
Assert.Equal("yes", decision.IntentName);
Assert.Equal("Yes.", decision.ReplyText);
}
[Fact]
public async Task BuildDecisionAsync_SkillPhraseVariant_MapsToKnownIntent()
{
@@ -129,6 +149,128 @@ public sealed class JiboInteractionServiceTests
Assert.Equal("joke", decision.IntentName);
}
[Fact]
public async Task BuildDecisionAsync_OpenTheRadio_MapsToRadioLaunchIntent()
{
var service = CreateService();
var decision = await service.BuildDecisionAsync(new TurnContext
{
RawTranscript = "open the radio",
NormalizedTranscript = "open the radio"
});
Assert.Equal("radio", decision.IntentName);
Assert.Equal("@be/radio", decision.SkillName);
Assert.Equal("@be/radio", decision.SkillPayload!["skillId"]);
}
[Fact]
public async Task BuildDecisionAsync_PlayCountryMusic_MapsToRadioGenreLaunchIntent()
{
var service = CreateService();
var decision = await service.BuildDecisionAsync(new TurnContext
{
RawTranscript = "play country music",
NormalizedTranscript = "play country music"
});
Assert.Equal("radio_genre", decision.IntentName);
Assert.Equal("@be/radio", decision.SkillName);
Assert.Equal("Country", decision.SkillPayload!["station"]);
}
[Fact]
public async Task BuildDecisionAsync_OpenTimer_MapsToLocalClockTimerMenu()
{
var service = CreateService();
var decision = await service.BuildDecisionAsync(new TurnContext
{
RawTranscript = "open timer",
NormalizedTranscript = "open timer"
});
Assert.Equal("timer_menu", decision.IntentName);
Assert.Equal("@be/clock", decision.SkillName);
Assert.Equal("timer", decision.SkillPayload!["domain"]);
Assert.Equal("menu", decision.SkillPayload["clockIntent"]);
}
[Fact]
public async Task BuildDecisionAsync_SetTimerForFiveMinutes_MapsToTimerValue()
{
var service = CreateService();
var decision = await service.BuildDecisionAsync(new TurnContext
{
RawTranscript = "set a timer for five minutes",
NormalizedTranscript = "set a timer for five minutes"
});
Assert.Equal("timer_value", decision.IntentName);
Assert.Equal("@be/clock", decision.SkillName);
Assert.Equal("timer", decision.SkillPayload!["domain"]);
Assert.Equal("timerValue", decision.SkillPayload["clockIntent"]);
Assert.Equal("0", decision.SkillPayload["hours"]);
Assert.Equal("5", decision.SkillPayload["minutes"]);
Assert.Equal("null", decision.SkillPayload["seconds"]);
}
[Fact]
public async Task BuildDecisionAsync_SetAlarmForSevenThirtyAm_MapsToAlarmValue()
{
var service = CreateService();
var decision = await service.BuildDecisionAsync(new TurnContext
{
RawTranscript = "set an alarm for 7:30 am",
NormalizedTranscript = "set an alarm for 7:30 am"
});
Assert.Equal("alarm_value", decision.IntentName);
Assert.Equal("@be/clock", decision.SkillName);
Assert.Equal("alarm", decision.SkillPayload!["domain"]);
Assert.Equal("alarmValue", decision.SkillPayload["clockIntent"]);
Assert.Equal("7:30", decision.SkillPayload["time"]);
Assert.Equal("am", decision.SkillPayload["ampm"]);
}
[Fact]
public async Task BuildDecisionAsync_TellMeTheNews_UsesNimbusCloudSkillPath()
{
var service = CreateService();
var decision = await service.BuildDecisionAsync(new TurnContext
{
RawTranscript = "tell me the news",
NormalizedTranscript = "tell me the news"
});
Assert.Equal("news", decision.IntentName);
Assert.Equal("news", decision.SkillName);
Assert.Equal("news", decision.SkillPayload!["skillId"]);
Assert.Equal("news", decision.SkillPayload["cloudSkill"]);
Assert.Equal("runtime-news", decision.SkillPayload["mim_id"]);
Assert.DoesNotContain("future cloud integration", decision.ReplyText, StringComparison.OrdinalIgnoreCase);
}
[Fact]
public async Task BuildDecisionAsync_CloudVersion_UsesSharedBuildInfo()
{
var service = CreateService();
var decision = await service.BuildDecisionAsync(new TurnContext
{
RawTranscript = "what's the cloud version",
NormalizedTranscript = "what's the cloud version"
});
Assert.Equal("cloud_version", decision.IntentName);
Assert.Equal(OpenJiboCloudBuildInfo.SpokenVersion, decision.ReplyText);
}
[Fact]
public async Task BuildDecisionAsync_WordOfDayGuess_UsesStructuredClientNluGuess()
{

View File

@@ -4,6 +4,7 @@ using Jibo.Cloud.Domain.Models;
using Jibo.Cloud.Infrastructure.Content;
using Jibo.Cloud.Infrastructure.Persistence;
using Jibo.Cloud.Tests.Fixtures;
using Jibo.Runtime.Abstractions;
namespace Jibo.Cloud.Tests.WebSockets;
@@ -342,6 +343,77 @@ public sealed class JiboWebSocketServiceTests
Assert.Equal("clock/clock_menu", listenPayload.RootElement.GetProperty("data").GetProperty("match").GetProperty("rule").GetString());
}
[Fact]
public async Task ClientAsr_SetTimerForFiveMinutes_RedirectsIntoClockSkillWithTimerEntities()
{
await _service.HandleMessageAsync(new WebSocketMessageEnvelope
{
HostName = "neo-hub.jibo.com",
Path = "/listen",
Kind = "neo-hub-listen",
Token = "hub-clock-timer-token",
Text = """{"type":"LISTEN","transID":"trans-clock-timer","data":{"rules":["globals/global_commands_launch"]}}"""
});
var replies = await _service.HandleMessageAsync(new WebSocketMessageEnvelope
{
HostName = "neo-hub.jibo.com",
Path = "/listen",
Kind = "neo-hub-listen",
Token = "hub-clock-timer-token",
Text = """{"type":"CLIENT_ASR","transID":"trans-clock-timer","data":{"text":"set a timer for five minutes"}}"""
});
Assert.Equal(4, replies.Count);
Assert.Equal("LISTEN", ReadReplyType(replies[0]));
Assert.Equal("EOS", ReadReplyType(replies[1]));
Assert.Equal("SKILL_REDIRECT", ReadReplyType(replies[2]));
Assert.Equal("SKILL_ACTION", ReadReplyType(replies[3]));
using var listenPayload = JsonDocument.Parse(replies[0].Text!);
Assert.Equal("timerValue", listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("intent").GetString());
Assert.Equal("@be/clock", listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("skill").GetString());
Assert.Equal("timer", listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("entities").GetProperty("domain").GetString());
Assert.Equal("0", listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("entities").GetProperty("hours").GetString());
Assert.Equal("5", listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("entities").GetProperty("minutes").GetString());
Assert.Equal("null", listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("entities").GetProperty("seconds").GetString());
using var redirectPayload = JsonDocument.Parse(replies[2].Text!);
Assert.Equal("@be/clock", redirectPayload.RootElement.GetProperty("data").GetProperty("match").GetProperty("skillID").GetString());
Assert.Equal("timerValue", redirectPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("intent").GetString());
}
[Fact]
public async Task ClientAsr_SetAlarmForSevenThirtyAm_RedirectsIntoClockSkillWithAlarmEntities()
{
await _service.HandleMessageAsync(new WebSocketMessageEnvelope
{
HostName = "neo-hub.jibo.com",
Path = "/listen",
Kind = "neo-hub-listen",
Token = "hub-clock-alarm-token",
Text = """{"type":"LISTEN","transID":"trans-clock-alarm","data":{"rules":["globals/global_commands_launch"]}}"""
});
var replies = await _service.HandleMessageAsync(new WebSocketMessageEnvelope
{
HostName = "neo-hub.jibo.com",
Path = "/listen",
Kind = "neo-hub-listen",
Token = "hub-clock-alarm-token",
Text = """{"type":"CLIENT_ASR","transID":"trans-clock-alarm","data":{"text":"set an alarm for 7:30 am"}}"""
});
Assert.Equal(4, replies.Count);
using var listenPayload = JsonDocument.Parse(replies[0].Text!);
Assert.Equal("alarmValue", listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("intent").GetString());
Assert.Equal("@be/clock", listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("skill").GetString());
Assert.Equal("alarm", listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("entities").GetProperty("domain").GetString());
Assert.Equal("7:30", listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("entities").GetProperty("time").GetString());
Assert.Equal("am", listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("entities").GetProperty("ampm").GetString());
}
[Fact]
public async Task ClientAsr_YesNoCreateFlow_PreservesCreateRuleAndDomain()
{
@@ -403,6 +475,265 @@ public sealed class JiboWebSocketServiceTests
Assert.Equal("surprises-ota/want_to_download_now", listenPayload.RootElement.GetProperty("data").GetProperty("match").GetProperty("rule").GetString());
}
[Fact]
public async Task BufferedAudio_YesNoPromptWithSttFailure_AutoFinalizesAsLocalNoInput()
{
await _service.HandleMessageAsync(new WebSocketMessageEnvelope
{
HostName = "neo-hub.jibo.com",
Path = "/listen",
Kind = "neo-hub-listen",
Token = "hub-yesno-noinput-token",
Text = """{"type":"LISTEN","transID":"trans-yesno-noinput","data":{"rules":["surprises-ota/want_to_download_now","globals/gui_nav","globals/global_commands_launch"],"asr":{"hints":["$YESNO"]}}}"""
});
await _service.HandleMessageAsync(new WebSocketMessageEnvelope
{
HostName = "neo-hub.jibo.com",
Path = "/listen",
Kind = "neo-hub-listen",
Token = "hub-yesno-noinput-token",
Text = """{"type":"CONTEXT","transID":"trans-yesno-noinput","data":{"topic":"conversation"}}"""
});
for (var index = 0; index < 4; index += 1)
{
var interimReplies = await _service.HandleMessageAsync(new WebSocketMessageEnvelope
{
HostName = "neo-hub.jibo.com",
Path = "/listen",
Kind = "neo-hub-listen",
Token = "hub-yesno-noinput-token",
Binary = new byte[3000]
});
Assert.Single(interimReplies);
Assert.Equal("OPENJIBO_AUDIO_RECEIVED", ReadReplyType(interimReplies[0]));
}
var session = _store.FindSessionByToken("hub-yesno-noinput-token");
Assert.NotNull(session);
session.TurnState.FirstAudioReceivedUtc = DateTimeOffset.UtcNow - TimeSpan.FromSeconds(2);
session.TurnState.LastSttError = "whisper.cpp returned no transcript";
var replies = await _service.HandleMessageAsync(new WebSocketMessageEnvelope
{
HostName = "neo-hub.jibo.com",
Path = "/listen",
Kind = "neo-hub-listen",
Token = "hub-yesno-noinput-token",
Binary = new byte[3000]
});
Assert.Equal(2, replies.Count);
Assert.Equal("LISTEN", ReadReplyType(replies[0]));
Assert.Equal("EOS", ReadReplyType(replies[1]));
using var listenPayload = JsonDocument.Parse(replies[0].Text!);
Assert.Equal(string.Empty, listenPayload.RootElement.GetProperty("data").GetProperty("asr").GetProperty("text").GetString());
Assert.Equal(string.Empty, listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("intent").GetString());
var rules = listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("rules");
Assert.Single(rules.EnumerateArray());
Assert.Equal("surprises-ota/want_to_download_now", rules[0].GetString());
}
[Fact]
public async Task ClientAsr_SurprisesDateOfferPrompt_MapsYesWithoutGlobalRuleLeak()
{
await _service.HandleMessageAsync(new WebSocketMessageEnvelope
{
HostName = "neo-hub.jibo.com",
Path = "/listen",
Kind = "neo-hub-listen",
Token = "hub-share-yesno-token",
Text = """{"type":"LISTEN","transID":"trans-share-yes","data":{"rules":["surprises-date/offer_date_fact","globals/gui_nav","globals/mim_repeat","globals/global_commands_launch"],"asr":{"hints":["$YESNO"]}}}"""
});
var replies = await _service.HandleMessageAsync(new WebSocketMessageEnvelope
{
HostName = "neo-hub.jibo.com",
Path = "/listen",
Kind = "neo-hub-listen",
Token = "hub-share-yesno-token",
Text = """{"type":"CLIENT_ASR","transID":"trans-share-yes","data":{"text":"Yes!"}}"""
});
Assert.Equal(3, replies.Count);
using var listenPayload = JsonDocument.Parse(replies[0].Text!);
Assert.Equal("yes", listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("intent").GetString());
var rules = listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("rules");
Assert.Single(rules.EnumerateArray());
Assert.Equal("surprises-date/offer_date_fact", rules[0].GetString());
Assert.Equal("surprises-date/offer_date_fact", listenPayload.RootElement.GetProperty("data").GetProperty("match").GetProperty("rule").GetString());
Assert.Equal(0, listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("entities").EnumerateObject().Count());
}
[Fact]
public void ResponsePlanMapper_EscapesSpeechWithoutEncodingApostrophes()
{
var plan = new ResponsePlan
{
IntentName = "chat",
Actions =
{
new SpeakAction
{
Sequence = 0,
Text = "I'm glad you're here.",
Voice = "griffin"
},
new InvokeNativeSkillAction
{
Sequence = 1,
SkillName = "chitchat-skill",
Payload = new Dictionary<string, object?>()
}
}
};
var turn = new TurnContext
{
Attributes = new Dictionary<string, object?>
{
["transID"] = "trans-apostrophe"
}
};
var replies = ResponsePlanToSocketMessagesMapper.Map(plan, turn, new CloudSession(), emitSkillActions: true);
using var payload = JsonDocument.Parse(replies[2].Text);
var esml = payload.RootElement
.GetProperty("data")
.GetProperty("action")
.GetProperty("config")
.GetProperty("jcp")
.GetProperty("config")
.GetProperty("play")
.GetProperty("esml")
.GetString();
Assert.Contains("I'm glad you're here.", esml, StringComparison.Ordinal);
Assert.DoesNotContain("&apos;", esml, StringComparison.Ordinal);
}
[Fact]
public async Task ClientAsr_TellMeTheNews_EmitsNimbusCloudSkillMatchAndNewsSkillAction()
{
await _service.HandleMessageAsync(new WebSocketMessageEnvelope
{
HostName = "neo-hub.jibo.com",
Path = "/listen",
Kind = "neo-hub-listen",
Token = "hub-news-token",
Text = """{"type":"LISTEN","transID":"trans-news","data":{"hotphrase":true,"rules":["launch","globals/global_commands_launch"]}}"""
});
var replies = await _service.HandleMessageAsync(new WebSocketMessageEnvelope
{
HostName = "neo-hub.jibo.com",
Path = "/listen",
Kind = "neo-hub-listen",
Token = "hub-news-token",
Text = """{"type":"CLIENT_ASR","transID":"trans-news","data":{"text":"tell me the news"}}"""
});
Assert.Equal(3, replies.Count);
Assert.Equal("LISTEN", ReadReplyType(replies[0]));
Assert.Equal("EOS", ReadReplyType(replies[1]));
Assert.Equal("SKILL_ACTION", ReadReplyType(replies[2]));
using var listenPayload = JsonDocument.Parse(replies[0].Text!);
Assert.Equal("news", listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("intent").GetString());
Assert.Equal("news", listenPayload.RootElement.GetProperty("data").GetProperty("match").GetProperty("cloudSkill").GetString());
using var skillPayload = JsonDocument.Parse(replies[2].Text!);
Assert.Equal("news", skillPayload.RootElement.GetProperty("data").GetProperty("skill").GetProperty("id").GetString());
var meta = skillPayload.RootElement
.GetProperty("data")
.GetProperty("action")
.GetProperty("config")
.GetProperty("jcp")
.GetProperty("config")
.GetProperty("play")
.GetProperty("meta");
Assert.Equal("runtime-news", meta.GetProperty("mim_id").GetString());
Assert.Equal("announcement", meta.GetProperty("mim_type").GetString());
}
[Fact]
public async Task ClientAsr_OpenTheRadio_EmitsRadioRedirectAndSilentCompletion()
{
await _service.HandleMessageAsync(new WebSocketMessageEnvelope
{
HostName = "neo-hub.jibo.com",
Path = "/listen",
Kind = "neo-hub-listen",
Token = "hub-radio-open-token",
Text = """{"type":"LISTEN","transID":"trans-radio-open","data":{"hotphrase":true,"rules":["launch","globals/global_commands_launch"]}}"""
});
var replies = await _service.HandleMessageAsync(new WebSocketMessageEnvelope
{
HostName = "neo-hub.jibo.com",
Path = "/listen",
Kind = "neo-hub-listen",
Token = "hub-radio-open-token",
Text = """{"type":"CLIENT_ASR","transID":"trans-radio-open","data":{"text":"open the radio"}}"""
});
Assert.Equal(4, replies.Count);
Assert.Equal("LISTEN", ReadReplyType(replies[0]));
Assert.Equal("EOS", ReadReplyType(replies[1]));
Assert.Equal("SKILL_REDIRECT", ReadReplyType(replies[2]));
Assert.Equal("SKILL_ACTION", ReadReplyType(replies[3]));
using var listenPayload = JsonDocument.Parse(replies[0].Text!);
Assert.Equal("menu", listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("intent").GetString());
Assert.Equal("@be/radio", listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("skill").GetString());
Assert.Equal(0, listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("rules").GetArrayLength());
Assert.Equal(0, listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("entities").EnumerateObject().Count());
using var redirectPayload = JsonDocument.Parse(replies[2].Text!);
Assert.Equal("@be/radio", redirectPayload.RootElement.GetProperty("data").GetProperty("match").GetProperty("skillID").GetString());
Assert.True(redirectPayload.RootElement.GetProperty("data").GetProperty("match").GetProperty("launch").GetBoolean());
Assert.Equal("menu", redirectPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("intent").GetString());
using var completionPayload = JsonDocument.Parse(replies[3].Text!);
Assert.Equal("@be/radio", completionPayload.RootElement.GetProperty("data").GetProperty("skill").GetProperty("id").GetString());
}
[Fact]
public async Task ClientAsr_PlayCountryMusic_EmitsRadioRedirectWithCountryStation()
{
await _service.HandleMessageAsync(new WebSocketMessageEnvelope
{
HostName = "neo-hub.jibo.com",
Path = "/listen",
Kind = "neo-hub-listen",
Token = "hub-radio-country-token",
Text = """{"type":"LISTEN","transID":"trans-radio-country","data":{"hotphrase":true,"rules":["launch","globals/global_commands_launch"]}}"""
});
var replies = await _service.HandleMessageAsync(new WebSocketMessageEnvelope
{
HostName = "neo-hub.jibo.com",
Path = "/listen",
Kind = "neo-hub-listen",
Token = "hub-radio-country-token",
Text = """{"type":"CLIENT_ASR","transID":"trans-radio-country","data":{"text":"play country music"}}"""
});
Assert.Equal(4, replies.Count);
using var listenPayload = JsonDocument.Parse(replies[0].Text!);
Assert.Equal("menu", listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("intent").GetString());
Assert.Equal("Country", listenPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("entities").GetProperty("station").GetString());
using var redirectPayload = JsonDocument.Parse(replies[2].Text!);
Assert.Equal("Country", redirectPayload.RootElement.GetProperty("data").GetProperty("nlu").GetProperty("entities").GetProperty("station").GetString());
Assert.Equal("play country music", redirectPayload.RootElement.GetProperty("data").GetProperty("asr").GetProperty("text").GetString());
}
[Fact]
public async Task ClientNlu_WordOfDayGuess_UsesGuessEntityAsAsrTextAndCompletesTurn()
{

Binary file not shown.

After

Width:  |  Height:  |  Size: 176 KiB