Document persistence architecture and report-skill parity

OpenJibo/docs/feature-backlog.md

@@ -887,6 +887,8 @@ For `1.0.19`:
 7. Personal report parity track (weather visuals, live news path, commute path, calendar parity matrix) - in progress (`2026-05-10` first live-news provider slice implemented)
 8. Holidays and seasonal personality behavior built on the new memory/proactivity foundation
 9. Durable memory persistence path (multi-tenant backing store)
+  - reference design captured in `docs/persistence-architecture.md`
+  - next implementation pass should tighten the store contracts around account/loop/device/person scoping and record versioning
 10. Update, backup, and restore proof
 11. STT upgrade and noise screening
 12. Hosted capture/storage plan / indexing for group testing

OpenJibo/docs/persistence-architecture.md

@@ -0,0 +1,136 @@
+# Persistence Architecture
+
+## Goal
+
+Keep OpenJibo's stateful behavior portable now and Azure-ready later.
+
+The current in-memory stores are fine as the default implementation, but the app should depend on stable persistence contracts rather than directly on in-memory collections or file formats.
+
+## Design Principles
+
+- Application code talks to small, intent-specific interfaces.
+- Persistence keys are always scoped by tenant and person where relevant.
+- In-memory, local JSON, and hosted Azure stores are adapters, not behavior sources.
+- Long-lived data should be versioned so we can add optimistic concurrency later.
+- Ephemeral turn/session state should stay separate from durable user and device state.
+
+## Current Seams
+
+These are the contracts we should preserve:
+
+- `IPersonalMemoryStore`
+  - personal facts: names, birthdays, preferences, affinities, important dates, household lists
+  - scope: account + loop + device + optional person
+- `ICloudStateStore`
+  - account, robot, loops, people, sessions, updates, media, backups, holidays, keys
+  - scope: system-level state with loop/device/person records inside it
+- `IJiboExperienceContentRepository`
+  - catalog/content layer only
+
+## Recommended Storage Split
+
+### 1. Identity and topology store
+
+Responsible for:
+
+- account profile
+- robot/device registration
+- loop membership
+- person records
+- greeting/proactive presence metadata when it becomes durable
+
+This is the seam most likely to become Azure SQL or Cosmos later.
+
+### 2. Personal memory store
+
+Responsible for:
+
+- names
+- birthdays
+- preferences
+- affinities
+- important dates
+- household lists
+
+This can remain in memory now and later move to a durable store keyed by account/loop/device/person.
+
+### 3. Session and short-lived orchestration state
+
+Responsible for:
+
+- websocket/session tokens
+- temporary skill state
+- active report/list/greeting interaction state
+
+This can stay in-process for now, but should be clearly separated from durable memory.
+
+### 4. Media and backup store
+
+Responsible for:
+
+- uploaded media metadata
+- backup manifests
+- binary references
+
+This is a good candidate for Azure Blob Storage plus a metadata table later.
+
+## Record Shape Guidance
+
+For durable records, prefer a small shared envelope:
+
+- `AccountId`
+- `LoopId`
+- `DeviceId`
+- `PersonId` when relevant
+- `RecordType`
+- `RecordKey`
+- `Value`
+- `CreatedUtc`
+- `UpdatedUtc`
+- `Revision` or `ETag`
+
+That gives us:
+
+- easy partitioning later
+- clear tenant boundaries
+- room for concurrency checks
+- a path to Azure Table, Cosmos, or SQL without changing behavior code
+
+## Adapter Plan
+
+### Phase 1
+
+- keep `InMemoryPersonalMemoryStore`
+- keep `InMemoryCloudStateStore`
+- make sure all callers use the interfaces only
+- add tests against behavior, not implementation details
+
+### Phase 2
+
+- introduce durable adapters behind the same interfaces
+- likely split:
+  - SQL or Cosmos for identity/topology
+  - Blob or table-backed store for media/backup metadata
+  - table/SQL-backed memory store for personal facts
+
+### Phase 3
+
+- add replication/sync primitives if we need multi-server state convergence
+- prefer explicit change records or versioned snapshots over hidden shared state
+
+## Non-Goals For Now
+
+- no Azure SDK types in application logic
+- no event-sourcing rewrite
+- no giant generic repository
+- no distributed transaction work before single-node semantics are stable
+
+## Immediate Next Step
+
+Before building durable adapters, tighten the store contracts around:
+
+- tenant/person scoping
+- record versioning
+- explicit load/save operations for durable state
+
+That lets us swap the backing store later without changing the personality, report, greeting, or list behaviors already built on top.

OpenJibo/docs/release-1.0.19-plan.md

@@ -102,6 +102,10 @@ The goal is to port these in small batches, capture the source-backed phrasing w
 - prefer explicit change records or versioned state snapshots over implicit last-writer wins when we outgrow a single node
 - keep cross-server reconciliation out of the hot path until the single-server semantics are stable
 
+Reference design:
+
+- [persistence-architecture.md](persistence-architecture.md)
+
 ## First Implemented Slice In `1.0.19`
 
 The first delivered slice in this release is persona expansion:

OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Application/Abstractions/IJiboExperienceContentRepository.cs

@@ -30,6 +30,14 @@ public sealed class JiboExperienceCatalog
     public IReadOnlyList<string> WeatherTodayHighLowReplies { get; init; } = [];
     public IReadOnlyList<string> WeatherTomorrowHighLowReplies { get; init; } = [];
     public IReadOnlyList<string> WeatherServiceDownReplies { get; init; } = [];
+    public IReadOnlyList<string> CalendarNothingTodayReplies { get; init; } = [];
+    public IReadOnlyList<string> CalendarNothingReplies { get; init; } = [];
+    public IReadOnlyList<string> CalendarOutroReplies { get; init; } = [];
+    public IReadOnlyList<string> CommuteNowReplies { get; init; } = [];
+    public IReadOnlyList<string> CommuteServiceDownReplies { get; init; } = [];
+    public IReadOnlyList<string> NewsIntroReplies { get; init; } = [];
+    public IReadOnlyList<string> NewsCategoryIntroReplies { get; init; } = [];
+    public IReadOnlyList<string> NewsOutroReplies { get; init; } = [];
     public IReadOnlyList<string> WeatherReplies { get; init; } = [];
     public IReadOnlyList<string> CalendarReplies { get; init; } = [];
     public IReadOnlyList<string> CommuteReplies { get; init; } = [];

OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Application/Services/PersonalReportOrchestrator.cs

@@ -292,17 +292,50 @@ internal static class PersonalReportOrchestrator
 
         if (toggles.CalendarEnabled)
         {
-            reportSections.Add(randomizer.Choose(catalog.CalendarReplies));
+            reportSections.Add(
+                RenderReportSkillTemplate(
+                    ChooseReportSkillTemplate(
+                        catalog.CalendarNothingTodayReplies,
+                        catalog.CalendarNothingReplies,
+                        "Looking at your calendar, I don't see anything scheduled today."),
+                    userName));
+            reportSections.Add(
+                RenderReportSkillTemplate(
+                    ChooseReportSkillTemplate(
+                        catalog.CalendarOutroReplies,
+                        [],
+                        "And that's it."),
+                    userName));
         }
 
         if (toggles.CommuteEnabled)
         {
-            reportSections.Add(randomizer.Choose(catalog.CommuteReplies));
+            reportSections.Add(
+                RenderReportSkillTemplate(
+                    ChooseReportSkillTemplate(
+                        catalog.CommuteServiceDownReplies,
+                        catalog.CommuteNowReplies,
+                        "Sorry, commute information isn't available right now."),
+                    userName));
         }
 
         if (toggles.NewsEnabled)
         {
+            reportSections.Add(
+                RenderReportSkillTemplate(
+                    ChooseReportSkillTemplate(
+                        catalog.NewsIntroReplies,
+                        catalog.NewsCategoryIntroReplies,
+                        "Here's today's news, from the associated press."),
+                    userName));
             reportSections.Add(randomizer.Choose(catalog.NewsBriefings));
+            reportSections.Add(
+                RenderReportSkillTemplate(
+                    ChooseReportSkillTemplate(
+                        catalog.NewsOutroReplies,
+                        [],
+                        "And that's what's new in the news."),
+                    userName));
         }
 
         reportSections.Add(
@@ -697,4 +730,33 @@ internal static class PersonalReportOrchestrator
             .Replace("  ", " ", StringComparison.Ordinal)
             .Trim();
     }
+
+    private static string ChooseReportSkillTemplate(
+        IReadOnlyList<string> primaryTemplates,
+        IReadOnlyList<string> secondaryTemplates,
+        string fallback)
+    {
+        var primary = primaryTemplates.FirstOrDefault(static template => !string.IsNullOrWhiteSpace(template));
+        if (!string.IsNullOrWhiteSpace(primary))
+        {
+            return primary!;
+        }
+
+        var secondary = secondaryTemplates.FirstOrDefault(static template => !string.IsNullOrWhiteSpace(template));
+        if (!string.IsNullOrWhiteSpace(secondary))
+        {
+            return secondary!;
+        }
+
+        return fallback;
+    }
+
+    private static string RenderReportSkillTemplate(string template, string userName)
+    {
+        return template
+            .Replace("${speaker}", userName, StringComparison.OrdinalIgnoreCase)
+            .Replace("${speaker}'s", $"{userName}'s", StringComparison.OrdinalIgnoreCase)
+            .Replace("  ", " ", StringComparison.Ordinal)
+            .Trim();
+    }
 }

OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Infrastructure/Content/LegacyMimCatalogImporter.cs

@@ -154,6 +154,46 @@ public static class LegacyMimCatalogImporter
             return LegacyMimBucket.WeatherServiceDown;
         }
 
+        if (fileName.StartsWith("CalendarNothingToday", StringComparison.OrdinalIgnoreCase))
+        {
+            return LegacyMimBucket.CalendarNothingToday;
+        }
+
+        if (fileName.StartsWith("CalendarNothing", StringComparison.OrdinalIgnoreCase))
+        {
+            return LegacyMimBucket.CalendarNothing;
+        }
+
+        if (fileName.StartsWith("CalendarOutro", StringComparison.OrdinalIgnoreCase))
+        {
+            return LegacyMimBucket.CalendarOutro;
+        }
+
+        if (fileName.StartsWith("CommuteNow", StringComparison.OrdinalIgnoreCase))
+        {
+            return LegacyMimBucket.CommuteNow;
+        }
+
+        if (fileName.StartsWith("CommuteServiceDown", StringComparison.OrdinalIgnoreCase))
+        {
+            return LegacyMimBucket.CommuteServiceDown;
+        }
+
+        if (fileName.StartsWith("NewsIntroCategory", StringComparison.OrdinalIgnoreCase))
+        {
+            return LegacyMimBucket.NewsCategoryIntro;
+        }
+
+        if (fileName.StartsWith("NewsIntro", StringComparison.OrdinalIgnoreCase))
+        {
+            return LegacyMimBucket.NewsIntro;
+        }
+
+        if (fileName.StartsWith("NewsOutro", StringComparison.OrdinalIgnoreCase))
+        {
+            return LegacyMimBucket.NewsOutro;
+        }
+
         if (fileName.StartsWith("Weather", StringComparison.OrdinalIgnoreCase) ||
             string.Equals(fileName, "WetNowDryLater", StringComparison.OrdinalIgnoreCase))
         {
@@ -266,6 +306,14 @@ public static class LegacyMimCatalogImporter
             WeatherTodayHighLowReplies = Merge(baseCatalog.WeatherTodayHighLowReplies, importedCatalog.WeatherTodayHighLowReplies),
             WeatherTomorrowHighLowReplies = Merge(baseCatalog.WeatherTomorrowHighLowReplies, importedCatalog.WeatherTomorrowHighLowReplies),
             WeatherServiceDownReplies = Merge(baseCatalog.WeatherServiceDownReplies, importedCatalog.WeatherServiceDownReplies),
+            CalendarNothingTodayReplies = Merge(baseCatalog.CalendarNothingTodayReplies, importedCatalog.CalendarNothingTodayReplies),
+            CalendarNothingReplies = Merge(baseCatalog.CalendarNothingReplies, importedCatalog.CalendarNothingReplies),
+            CalendarOutroReplies = Merge(baseCatalog.CalendarOutroReplies, importedCatalog.CalendarOutroReplies),
+            CommuteNowReplies = Merge(baseCatalog.CommuteNowReplies, importedCatalog.CommuteNowReplies),
+            CommuteServiceDownReplies = Merge(baseCatalog.CommuteServiceDownReplies, importedCatalog.CommuteServiceDownReplies),
+            NewsIntroReplies = Merge(baseCatalog.NewsIntroReplies, importedCatalog.NewsIntroReplies),
+            NewsCategoryIntroReplies = Merge(baseCatalog.NewsCategoryIntroReplies, importedCatalog.NewsCategoryIntroReplies),
+            NewsOutroReplies = Merge(baseCatalog.NewsOutroReplies, importedCatalog.NewsOutroReplies),
             WeatherReplies = Merge(baseCatalog.WeatherReplies, importedCatalog.WeatherReplies),
             CalendarReplies = Merge(baseCatalog.CalendarReplies, importedCatalog.CalendarReplies),
             CommuteReplies = Merge(baseCatalog.CommuteReplies, importedCatalog.CommuteReplies),
@@ -347,6 +395,14 @@ public static class LegacyMimCatalogImporter
         WeatherTodayHighLow,
         WeatherTomorrowHighLow,
         WeatherServiceDown,
+        CalendarNothingToday,
+        CalendarNothing,
+        CalendarOutro,
+        CommuteNow,
+        CommuteServiceDown,
+        NewsIntro,
+        NewsCategoryIntro,
+        NewsOutro,
         ReportSkillTemplate
     }
 
@@ -365,6 +421,14 @@ public static class LegacyMimCatalogImporter
         private readonly List<string> _weatherTodayHighLowReplies = [];
         private readonly List<string> _weatherTomorrowHighLowReplies = [];
         private readonly List<string> _weatherServiceDownReplies = [];
+        private readonly List<string> _calendarNothingTodayReplies = [];
+        private readonly List<string> _calendarNothingReplies = [];
+        private readonly List<string> _calendarOutroReplies = [];
+        private readonly List<string> _commuteNowReplies = [];
+        private readonly List<string> _commuteServiceDownReplies = [];
+        private readonly List<string> _newsIntroReplies = [];
+        private readonly List<string> _newsCategoryIntroReplies = [];
+        private readonly List<string> _newsOutroReplies = [];
 
         public void Add(LegacyMimBucket bucket, string? condition, string text)
         {
@@ -438,6 +502,30 @@ public static class LegacyMimCatalogImporter
                 case LegacyMimBucket.WeatherServiceDown:
                     AddDistinct(_weatherServiceDownReplies, text);
                     return;
+                case LegacyMimBucket.CalendarNothingToday:
+                    AddDistinct(_calendarNothingTodayReplies, text);
+                    return;
+                case LegacyMimBucket.CalendarNothing:
+                    AddDistinct(_calendarNothingReplies, text);
+                    return;
+                case LegacyMimBucket.CalendarOutro:
+                    AddDistinct(_calendarOutroReplies, text);
+                    return;
+                case LegacyMimBucket.CommuteNow:
+                    AddDistinct(_commuteNowReplies, text);
+                    return;
+                case LegacyMimBucket.CommuteServiceDown:
+                    AddDistinct(_commuteServiceDownReplies, text);
+                    return;
+                case LegacyMimBucket.NewsIntro:
+                    AddDistinct(_newsIntroReplies, text);
+                    return;
+                case LegacyMimBucket.NewsCategoryIntro:
+                    AddDistinct(_newsCategoryIntroReplies, text);
+                    return;
+                case LegacyMimBucket.NewsOutro:
+                    AddDistinct(_newsOutroReplies, text);
+                    return;
                 case LegacyMimBucket.ReportSkillTemplate:
                     AddDistinct(_reportSkillTemplates, text);
                     return;
@@ -463,6 +551,15 @@ public static class LegacyMimCatalogImporter
                 WeatherTodayHighLowReplies = [.. _weatherTodayHighLowReplies],
                 WeatherTomorrowHighLowReplies = [.. _weatherTomorrowHighLowReplies],
                 WeatherServiceDownReplies = [.. _weatherServiceDownReplies]
+                ,
+                CalendarNothingTodayReplies = [.. _calendarNothingTodayReplies],
+                CalendarNothingReplies = [.. _calendarNothingReplies],
+                CalendarOutroReplies = [.. _calendarOutroReplies],
+                CommuteNowReplies = [.. _commuteNowReplies],
+                CommuteServiceDownReplies = [.. _commuteServiceDownReplies],
+                NewsIntroReplies = [.. _newsIntroReplies],
+                NewsCategoryIntroReplies = [.. _newsCategoryIntroReplies],
+                NewsOutroReplies = [.. _newsOutroReplies]
             };
         }
 

OpenJibo/tests/Jibo.Cloud.Tests/Content/LegacyMimCatalogImporterTests.cs

@@ -213,8 +213,10 @@ public sealed class LegacyMimCatalogImporterTests
         Assert.Contains("Looks like our weather service is offline. Sorry.", catalog.WeatherServiceDownReplies);
         Assert.Contains("Sure ${speaker}. Here it is.", catalog.PersonalReportKickOffReplies);
         Assert.Contains("And that's your report for the day. I hope you had as much fun as I did.", catalog.PersonalReportOutroReplies);
-        Assert.Contains(catalog.ReportSkillTemplates, reply =>
+        Assert.Contains("Looking at your calendar, I don't see anything scheduled today.", catalog.CalendarNothingTodayReplies);
-            reply.Contains("Checking your calendar, I see ${skill.calendar.numEventsToday} items today.", StringComparison.OrdinalIgnoreCase));
+        Assert.Contains("Sorry, commute information isn't available right now.", catalog.CommuteServiceDownReplies);
+        Assert.Contains("Here's today's news, from the associated press.", catalog.NewsIntroReplies);
+        Assert.Contains("And that's what's new in the news.", catalog.NewsOutroReplies);
     }
 
     [Fact]
@@ -260,6 +262,7 @@ public sealed class LegacyMimCatalogImporterTests
         Assert.Contains("Something's off with the connection to my sources. Maybe ask me again in a little while.", catalog.GenericFallbackReplies);
         Assert.Contains("For your weather.", catalog.WeatherIntroReplies);
         Assert.Contains("Today's high is {high}, and the low is {low}.", catalog.WeatherTodayHighLowReplies);
+        Assert.Contains("Looking at your calendar, I don't see anything scheduled today.", catalog.CalendarNothingTodayReplies);
     }
 
     private static string CreateSeedDirectory()

OpenJibo/tests/Jibo.Cloud.Tests/WebSockets/JiboInteractionServiceTests.cs

@@ -1727,6 +1727,10 @@ public sealed class JiboInteractionServiceTests
         Assert.Contains("Sure alex. Here it is.", decision.ReplyText, StringComparison.OrdinalIgnoreCase);
         Assert.Contains("First, your weather.", decision.ReplyText, StringComparison.OrdinalIgnoreCase);
         Assert.Contains("For your weather. In Boston, U.S., it's light rain and 61 degrees Fahrenheit. Today's high is 65, and the low is 54.", decision.ReplyText, StringComparison.OrdinalIgnoreCase);
+        Assert.Contains("Looking at your calendar, I don't see anything scheduled today.", decision.ReplyText, StringComparison.OrdinalIgnoreCase);
+        Assert.Contains("Sorry, commute information isn't available right now.", decision.ReplyText, StringComparison.OrdinalIgnoreCase);
+        Assert.Contains("Here's today's news, from the associated press.", decision.ReplyText, StringComparison.OrdinalIgnoreCase);
+        Assert.Contains("And that's what's new in the news.", decision.ReplyText, StringComparison.OrdinalIgnoreCase);
         Assert.Contains("alex that wraps up your report for the day. Hope you have a good one.", decision.ReplyText, StringComparison.OrdinalIgnoreCase);
         Assert.NotNull(decision.ContextUpdates);
         Assert.Equal("idle", decision.ContextUpdates![PersonalReportStateKey]);