diff --git a/OpenJibo/docs/feature-backlog.md b/OpenJibo/docs/feature-backlog.md index f6d5802..f422318 100644 --- a/OpenJibo/docs/feature-backlog.md +++ b/OpenJibo/docs/feature-backlog.md @@ -887,6 +887,8 @@ For `1.0.19`: 7. Personal report parity track (weather visuals, live news path, commute path, calendar parity matrix) - in progress (`2026-05-10` first live-news provider slice implemented) 8. Holidays and seasonal personality behavior built on the new memory/proactivity foundation 9. Durable memory persistence path (multi-tenant backing store) + - reference design captured in `docs/persistence-architecture.md` + - next implementation pass should tighten the store contracts around account/loop/device/person scoping and record versioning 10. Update, backup, and restore proof 11. STT upgrade and noise screening 12. Hosted capture/storage plan / indexing for group testing diff --git a/OpenJibo/docs/persistence-architecture.md b/OpenJibo/docs/persistence-architecture.md new file mode 100644 index 0000000..ec68c58 --- /dev/null +++ b/OpenJibo/docs/persistence-architecture.md @@ -0,0 +1,136 @@ +# Persistence Architecture + +## Goal + +Keep OpenJibo's stateful behavior portable now and Azure-ready later. + +The current in-memory stores are fine as the default implementation, but the app should depend on stable persistence contracts rather than directly on in-memory collections or file formats. + +## Design Principles + +- Application code talks to small, intent-specific interfaces. +- Persistence keys are always scoped by tenant and person where relevant. +- In-memory, local JSON, and hosted Azure stores are adapters, not behavior sources. +- Long-lived data should be versioned so we can add optimistic concurrency later. +- Ephemeral turn/session state should stay separate from durable user and device state. + +## Current Seams + +These are the contracts we should preserve: + +- `IPersonalMemoryStore` + - personal facts: names, birthdays, preferences, affinities, important dates, household lists + - scope: account + loop + device + optional person +- `ICloudStateStore` + - account, robot, loops, people, sessions, updates, media, backups, holidays, keys + - scope: system-level state with loop/device/person records inside it +- `IJiboExperienceContentRepository` + - catalog/content layer only + +## Recommended Storage Split + +### 1. Identity and topology store + +Responsible for: + +- account profile +- robot/device registration +- loop membership +- person records +- greeting/proactive presence metadata when it becomes durable + +This is the seam most likely to become Azure SQL or Cosmos later. + +### 2. Personal memory store + +Responsible for: + +- names +- birthdays +- preferences +- affinities +- important dates +- household lists + +This can remain in memory now and later move to a durable store keyed by account/loop/device/person. + +### 3. Session and short-lived orchestration state + +Responsible for: + +- websocket/session tokens +- temporary skill state +- active report/list/greeting interaction state + +This can stay in-process for now, but should be clearly separated from durable memory. + +### 4. Media and backup store + +Responsible for: + +- uploaded media metadata +- backup manifests +- binary references + +This is a good candidate for Azure Blob Storage plus a metadata table later. + +## Record Shape Guidance + +For durable records, prefer a small shared envelope: + +- `AccountId` +- `LoopId` +- `DeviceId` +- `PersonId` when relevant +- `RecordType` +- `RecordKey` +- `Value` +- `CreatedUtc` +- `UpdatedUtc` +- `Revision` or `ETag` + +That gives us: + +- easy partitioning later +- clear tenant boundaries +- room for concurrency checks +- a path to Azure Table, Cosmos, or SQL without changing behavior code + +## Adapter Plan + +### Phase 1 + +- keep `InMemoryPersonalMemoryStore` +- keep `InMemoryCloudStateStore` +- make sure all callers use the interfaces only +- add tests against behavior, not implementation details + +### Phase 2 + +- introduce durable adapters behind the same interfaces +- likely split: + - SQL or Cosmos for identity/topology + - Blob or table-backed store for media/backup metadata + - table/SQL-backed memory store for personal facts + +### Phase 3 + +- add replication/sync primitives if we need multi-server state convergence +- prefer explicit change records or versioned snapshots over hidden shared state + +## Non-Goals For Now + +- no Azure SDK types in application logic +- no event-sourcing rewrite +- no giant generic repository +- no distributed transaction work before single-node semantics are stable + +## Immediate Next Step + +Before building durable adapters, tighten the store contracts around: + +- tenant/person scoping +- record versioning +- explicit load/save operations for durable state + +That lets us swap the backing store later without changing the personality, report, greeting, or list behaviors already built on top. diff --git a/OpenJibo/docs/release-1.0.19-plan.md b/OpenJibo/docs/release-1.0.19-plan.md index 0d6cdaa..ca4551d 100644 --- a/OpenJibo/docs/release-1.0.19-plan.md +++ b/OpenJibo/docs/release-1.0.19-plan.md @@ -102,6 +102,10 @@ The goal is to port these in small batches, capture the source-backed phrasing w - prefer explicit change records or versioned state snapshots over implicit last-writer wins when we outgrow a single node - keep cross-server reconciliation out of the hot path until the single-server semantics are stable +Reference design: + +- [persistence-architecture.md](persistence-architecture.md) + ## First Implemented Slice In `1.0.19` The first delivered slice in this release is persona expansion: diff --git a/OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Application/Abstractions/IJiboExperienceContentRepository.cs b/OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Application/Abstractions/IJiboExperienceContentRepository.cs index 6cd0e99..1cc23fb 100644 --- a/OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Application/Abstractions/IJiboExperienceContentRepository.cs +++ b/OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Application/Abstractions/IJiboExperienceContentRepository.cs @@ -30,6 +30,14 @@ public sealed class JiboExperienceCatalog public IReadOnlyList WeatherTodayHighLowReplies { get; init; } = []; public IReadOnlyList WeatherTomorrowHighLowReplies { get; init; } = []; public IReadOnlyList WeatherServiceDownReplies { get; init; } = []; + public IReadOnlyList CalendarNothingTodayReplies { get; init; } = []; + public IReadOnlyList CalendarNothingReplies { get; init; } = []; + public IReadOnlyList CalendarOutroReplies { get; init; } = []; + public IReadOnlyList CommuteNowReplies { get; init; } = []; + public IReadOnlyList CommuteServiceDownReplies { get; init; } = []; + public IReadOnlyList NewsIntroReplies { get; init; } = []; + public IReadOnlyList NewsCategoryIntroReplies { get; init; } = []; + public IReadOnlyList NewsOutroReplies { get; init; } = []; public IReadOnlyList WeatherReplies { get; init; } = []; public IReadOnlyList CalendarReplies { get; init; } = []; public IReadOnlyList CommuteReplies { get; init; } = []; diff --git a/OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Application/Services/PersonalReportOrchestrator.cs b/OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Application/Services/PersonalReportOrchestrator.cs index e3a2760..e1175a0 100644 --- a/OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Application/Services/PersonalReportOrchestrator.cs +++ b/OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Application/Services/PersonalReportOrchestrator.cs @@ -292,17 +292,50 @@ internal static class PersonalReportOrchestrator if (toggles.CalendarEnabled) { - reportSections.Add(randomizer.Choose(catalog.CalendarReplies)); + reportSections.Add( + RenderReportSkillTemplate( + ChooseReportSkillTemplate( + catalog.CalendarNothingTodayReplies, + catalog.CalendarNothingReplies, + "Looking at your calendar, I don't see anything scheduled today."), + userName)); + reportSections.Add( + RenderReportSkillTemplate( + ChooseReportSkillTemplate( + catalog.CalendarOutroReplies, + [], + "And that's it."), + userName)); } if (toggles.CommuteEnabled) { - reportSections.Add(randomizer.Choose(catalog.CommuteReplies)); + reportSections.Add( + RenderReportSkillTemplate( + ChooseReportSkillTemplate( + catalog.CommuteServiceDownReplies, + catalog.CommuteNowReplies, + "Sorry, commute information isn't available right now."), + userName)); } if (toggles.NewsEnabled) { + reportSections.Add( + RenderReportSkillTemplate( + ChooseReportSkillTemplate( + catalog.NewsIntroReplies, + catalog.NewsCategoryIntroReplies, + "Here's today's news, from the associated press."), + userName)); reportSections.Add(randomizer.Choose(catalog.NewsBriefings)); + reportSections.Add( + RenderReportSkillTemplate( + ChooseReportSkillTemplate( + catalog.NewsOutroReplies, + [], + "And that's what's new in the news."), + userName)); } reportSections.Add( @@ -697,4 +730,33 @@ internal static class PersonalReportOrchestrator .Replace(" ", " ", StringComparison.Ordinal) .Trim(); } + + private static string ChooseReportSkillTemplate( + IReadOnlyList primaryTemplates, + IReadOnlyList secondaryTemplates, + string fallback) + { + var primary = primaryTemplates.FirstOrDefault(static template => !string.IsNullOrWhiteSpace(template)); + if (!string.IsNullOrWhiteSpace(primary)) + { + return primary!; + } + + var secondary = secondaryTemplates.FirstOrDefault(static template => !string.IsNullOrWhiteSpace(template)); + if (!string.IsNullOrWhiteSpace(secondary)) + { + return secondary!; + } + + return fallback; + } + + private static string RenderReportSkillTemplate(string template, string userName) + { + return template + .Replace("${speaker}", userName, StringComparison.OrdinalIgnoreCase) + .Replace("${speaker}'s", $"{userName}'s", StringComparison.OrdinalIgnoreCase) + .Replace(" ", " ", StringComparison.Ordinal) + .Trim(); + } } diff --git a/OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Infrastructure/Content/LegacyMimCatalogImporter.cs b/OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Infrastructure/Content/LegacyMimCatalogImporter.cs index f31f7b2..a7fada7 100644 --- a/OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Infrastructure/Content/LegacyMimCatalogImporter.cs +++ b/OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Infrastructure/Content/LegacyMimCatalogImporter.cs @@ -154,6 +154,46 @@ public static class LegacyMimCatalogImporter return LegacyMimBucket.WeatherServiceDown; } + if (fileName.StartsWith("CalendarNothingToday", StringComparison.OrdinalIgnoreCase)) + { + return LegacyMimBucket.CalendarNothingToday; + } + + if (fileName.StartsWith("CalendarNothing", StringComparison.OrdinalIgnoreCase)) + { + return LegacyMimBucket.CalendarNothing; + } + + if (fileName.StartsWith("CalendarOutro", StringComparison.OrdinalIgnoreCase)) + { + return LegacyMimBucket.CalendarOutro; + } + + if (fileName.StartsWith("CommuteNow", StringComparison.OrdinalIgnoreCase)) + { + return LegacyMimBucket.CommuteNow; + } + + if (fileName.StartsWith("CommuteServiceDown", StringComparison.OrdinalIgnoreCase)) + { + return LegacyMimBucket.CommuteServiceDown; + } + + if (fileName.StartsWith("NewsIntroCategory", StringComparison.OrdinalIgnoreCase)) + { + return LegacyMimBucket.NewsCategoryIntro; + } + + if (fileName.StartsWith("NewsIntro", StringComparison.OrdinalIgnoreCase)) + { + return LegacyMimBucket.NewsIntro; + } + + if (fileName.StartsWith("NewsOutro", StringComparison.OrdinalIgnoreCase)) + { + return LegacyMimBucket.NewsOutro; + } + if (fileName.StartsWith("Weather", StringComparison.OrdinalIgnoreCase) || string.Equals(fileName, "WetNowDryLater", StringComparison.OrdinalIgnoreCase)) { @@ -266,6 +306,14 @@ public static class LegacyMimCatalogImporter WeatherTodayHighLowReplies = Merge(baseCatalog.WeatherTodayHighLowReplies, importedCatalog.WeatherTodayHighLowReplies), WeatherTomorrowHighLowReplies = Merge(baseCatalog.WeatherTomorrowHighLowReplies, importedCatalog.WeatherTomorrowHighLowReplies), WeatherServiceDownReplies = Merge(baseCatalog.WeatherServiceDownReplies, importedCatalog.WeatherServiceDownReplies), + CalendarNothingTodayReplies = Merge(baseCatalog.CalendarNothingTodayReplies, importedCatalog.CalendarNothingTodayReplies), + CalendarNothingReplies = Merge(baseCatalog.CalendarNothingReplies, importedCatalog.CalendarNothingReplies), + CalendarOutroReplies = Merge(baseCatalog.CalendarOutroReplies, importedCatalog.CalendarOutroReplies), + CommuteNowReplies = Merge(baseCatalog.CommuteNowReplies, importedCatalog.CommuteNowReplies), + CommuteServiceDownReplies = Merge(baseCatalog.CommuteServiceDownReplies, importedCatalog.CommuteServiceDownReplies), + NewsIntroReplies = Merge(baseCatalog.NewsIntroReplies, importedCatalog.NewsIntroReplies), + NewsCategoryIntroReplies = Merge(baseCatalog.NewsCategoryIntroReplies, importedCatalog.NewsCategoryIntroReplies), + NewsOutroReplies = Merge(baseCatalog.NewsOutroReplies, importedCatalog.NewsOutroReplies), WeatherReplies = Merge(baseCatalog.WeatherReplies, importedCatalog.WeatherReplies), CalendarReplies = Merge(baseCatalog.CalendarReplies, importedCatalog.CalendarReplies), CommuteReplies = Merge(baseCatalog.CommuteReplies, importedCatalog.CommuteReplies), @@ -347,6 +395,14 @@ public static class LegacyMimCatalogImporter WeatherTodayHighLow, WeatherTomorrowHighLow, WeatherServiceDown, + CalendarNothingToday, + CalendarNothing, + CalendarOutro, + CommuteNow, + CommuteServiceDown, + NewsIntro, + NewsCategoryIntro, + NewsOutro, ReportSkillTemplate } @@ -365,6 +421,14 @@ public static class LegacyMimCatalogImporter private readonly List _weatherTodayHighLowReplies = []; private readonly List _weatherTomorrowHighLowReplies = []; private readonly List _weatherServiceDownReplies = []; + private readonly List _calendarNothingTodayReplies = []; + private readonly List _calendarNothingReplies = []; + private readonly List _calendarOutroReplies = []; + private readonly List _commuteNowReplies = []; + private readonly List _commuteServiceDownReplies = []; + private readonly List _newsIntroReplies = []; + private readonly List _newsCategoryIntroReplies = []; + private readonly List _newsOutroReplies = []; public void Add(LegacyMimBucket bucket, string? condition, string text) { @@ -438,6 +502,30 @@ public static class LegacyMimCatalogImporter case LegacyMimBucket.WeatherServiceDown: AddDistinct(_weatherServiceDownReplies, text); return; + case LegacyMimBucket.CalendarNothingToday: + AddDistinct(_calendarNothingTodayReplies, text); + return; + case LegacyMimBucket.CalendarNothing: + AddDistinct(_calendarNothingReplies, text); + return; + case LegacyMimBucket.CalendarOutro: + AddDistinct(_calendarOutroReplies, text); + return; + case LegacyMimBucket.CommuteNow: + AddDistinct(_commuteNowReplies, text); + return; + case LegacyMimBucket.CommuteServiceDown: + AddDistinct(_commuteServiceDownReplies, text); + return; + case LegacyMimBucket.NewsIntro: + AddDistinct(_newsIntroReplies, text); + return; + case LegacyMimBucket.NewsCategoryIntro: + AddDistinct(_newsCategoryIntroReplies, text); + return; + case LegacyMimBucket.NewsOutro: + AddDistinct(_newsOutroReplies, text); + return; case LegacyMimBucket.ReportSkillTemplate: AddDistinct(_reportSkillTemplates, text); return; @@ -463,6 +551,15 @@ public static class LegacyMimCatalogImporter WeatherTodayHighLowReplies = [.. _weatherTodayHighLowReplies], WeatherTomorrowHighLowReplies = [.. _weatherTomorrowHighLowReplies], WeatherServiceDownReplies = [.. _weatherServiceDownReplies] + , + CalendarNothingTodayReplies = [.. _calendarNothingTodayReplies], + CalendarNothingReplies = [.. _calendarNothingReplies], + CalendarOutroReplies = [.. _calendarOutroReplies], + CommuteNowReplies = [.. _commuteNowReplies], + CommuteServiceDownReplies = [.. _commuteServiceDownReplies], + NewsIntroReplies = [.. _newsIntroReplies], + NewsCategoryIntroReplies = [.. _newsCategoryIntroReplies], + NewsOutroReplies = [.. _newsOutroReplies] }; } diff --git a/OpenJibo/tests/Jibo.Cloud.Tests/Content/LegacyMimCatalogImporterTests.cs b/OpenJibo/tests/Jibo.Cloud.Tests/Content/LegacyMimCatalogImporterTests.cs index 8d9dbaf..86d86bb 100644 --- a/OpenJibo/tests/Jibo.Cloud.Tests/Content/LegacyMimCatalogImporterTests.cs +++ b/OpenJibo/tests/Jibo.Cloud.Tests/Content/LegacyMimCatalogImporterTests.cs @@ -213,8 +213,10 @@ public sealed class LegacyMimCatalogImporterTests Assert.Contains("Looks like our weather service is offline. Sorry.", catalog.WeatherServiceDownReplies); Assert.Contains("Sure ${speaker}. Here it is.", catalog.PersonalReportKickOffReplies); Assert.Contains("And that's your report for the day. I hope you had as much fun as I did.", catalog.PersonalReportOutroReplies); - Assert.Contains(catalog.ReportSkillTemplates, reply => - reply.Contains("Checking your calendar, I see ${skill.calendar.numEventsToday} items today.", StringComparison.OrdinalIgnoreCase)); + Assert.Contains("Looking at your calendar, I don't see anything scheduled today.", catalog.CalendarNothingTodayReplies); + Assert.Contains("Sorry, commute information isn't available right now.", catalog.CommuteServiceDownReplies); + Assert.Contains("Here's today's news, from the associated press.", catalog.NewsIntroReplies); + Assert.Contains("And that's what's new in the news.", catalog.NewsOutroReplies); } [Fact] @@ -260,6 +262,7 @@ public sealed class LegacyMimCatalogImporterTests Assert.Contains("Something's off with the connection to my sources. Maybe ask me again in a little while.", catalog.GenericFallbackReplies); Assert.Contains("For your weather.", catalog.WeatherIntroReplies); Assert.Contains("Today's high is {high}, and the low is {low}.", catalog.WeatherTodayHighLowReplies); + Assert.Contains("Looking at your calendar, I don't see anything scheduled today.", catalog.CalendarNothingTodayReplies); } private static string CreateSeedDirectory() diff --git a/OpenJibo/tests/Jibo.Cloud.Tests/WebSockets/JiboInteractionServiceTests.cs b/OpenJibo/tests/Jibo.Cloud.Tests/WebSockets/JiboInteractionServiceTests.cs index 74e0e42..41c3622 100644 --- a/OpenJibo/tests/Jibo.Cloud.Tests/WebSockets/JiboInteractionServiceTests.cs +++ b/OpenJibo/tests/Jibo.Cloud.Tests/WebSockets/JiboInteractionServiceTests.cs @@ -1727,6 +1727,10 @@ public sealed class JiboInteractionServiceTests Assert.Contains("Sure alex. Here it is.", decision.ReplyText, StringComparison.OrdinalIgnoreCase); Assert.Contains("First, your weather.", decision.ReplyText, StringComparison.OrdinalIgnoreCase); Assert.Contains("For your weather. In Boston, U.S., it's light rain and 61 degrees Fahrenheit. Today's high is 65, and the low is 54.", decision.ReplyText, StringComparison.OrdinalIgnoreCase); + Assert.Contains("Looking at your calendar, I don't see anything scheduled today.", decision.ReplyText, StringComparison.OrdinalIgnoreCase); + Assert.Contains("Sorry, commute information isn't available right now.", decision.ReplyText, StringComparison.OrdinalIgnoreCase); + Assert.Contains("Here's today's news, from the associated press.", decision.ReplyText, StringComparison.OrdinalIgnoreCase); + Assert.Contains("And that's what's new in the news.", decision.ReplyText, StringComparison.OrdinalIgnoreCase); Assert.Contains("alex that wraps up your report for the day. Hope you have a good one.", decision.ReplyText, StringComparison.OrdinalIgnoreCase); Assert.NotNull(decision.ContextUpdates); Assert.Equal("idle", decision.ContextUpdates![PersonalReportStateKey]);