207 lines
9.1 KiB
Markdown
207 lines
9.1 KiB
Markdown
# Feature Backlog
|
|
|
|
## Purpose
|
|
|
|
This backlog turns the current discovery work into a concrete implementation queue for the hosted `.NET` cloud.
|
|
|
|
Use it as the source of truth for the next feature slice instead of continuing the same investigation in chat each time.
|
|
|
|
## How To Use This Backlog
|
|
|
|
1. Pick one slice.
|
|
2. Confirm the target payload shape from captures and robot source.
|
|
3. Implement the smallest working parity path in `.NET`.
|
|
4. Test it live on stock OS `1.9`.
|
|
5. Update this file with results, regressions, and next guesses before moving on.
|
|
|
|
Status key:
|
|
|
|
- `ready`: grounded enough to implement now
|
|
- `discovery`: more robot-source or capture work needed first
|
|
- `polish`: behavior exists but needs cleanup
|
|
|
|
Parallel tags:
|
|
|
|
- `protocol`: websocket / turn-shape work
|
|
- `content`: provider or cloud content work
|
|
- `docs`: runbook / operator guidance
|
|
- `stt`: transcript reliability work
|
|
|
|
## Immediate Queue
|
|
|
|
### 1. Radio Resume And Genre Launch
|
|
|
|
- Status: `ready`
|
|
- Tags: `protocol`
|
|
- Why now: `@be/radio` is a real local skill and is the clearest low-risk expansion after Word of the Day.
|
|
- User goals:
|
|
- `open the radio` should resume the current or last station
|
|
- `play country music` should open a country station on iHeartRadio
|
|
- Current evidence:
|
|
- [index.js](C:/Projects/JiboOs/V3.1/build/opt/jibo/Jibo/Skills/@be/be/node_modules/@be/radio/index.js) resumes from `lastStation`
|
|
- the same file treats `menu` as a `play` launch and reads `result.nlu.entities.station`
|
|
- Implementation notes:
|
|
- add phrase routing for radio open/resume and genre launch
|
|
- inspect radio genre and station metadata before locking the outbound entity values
|
|
- prefer the same payload shape the menu path uses instead of a generic cloud speech reply
|
|
- Exit criteria:
|
|
- voice `open the radio` launches radio successfully
|
|
- voice `play country music` launches a country station
|
|
- no fallback cloud placeholder reply is spoken on success
|
|
|
|
### 2. ESML Apostrophe Encoding Bug
|
|
|
|
- Status: `ready`
|
|
- Tags: `polish`
|
|
- Why now: this is a small, high-confidence speech quality bug affecting many paths.
|
|
- Current evidence:
|
|
- [ResponsePlanToSocketMessagesMapper.cs](C:/Projects/JiboExperiments/OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Application/Services/ResponsePlanToSocketMessagesMapper.cs) currently escapes `'` to `'`
|
|
- the robot is pronouncing the encoded form instead of treating it as natural text
|
|
- Implementation notes:
|
|
- stop encoding apostrophes in spoken ESML text unless a capture proves a narrower escaping rule is needed
|
|
- keep escaping for `&`, `<`, and `>`
|
|
- Exit criteria:
|
|
- contractions and possessives sound natural again in live speech
|
|
|
|
### 3. Backup / OTA Yes-No Reliability
|
|
|
|
- Status: `ready`
|
|
- Tags: `protocol`, `stt`
|
|
- Why now: the update and backup prompts are real daily-use system flows and still feel fragile.
|
|
- Current evidence:
|
|
- `surprises-ota` is a real robot-side skill family in [index.js](C:/Projects/JiboOs/V3.1/build/opt/jibo/Jibo/Skills/@be/be/node_modules/@be/surprises-ota/index.js)
|
|
- we already improved constrained yes-no routing, but live tests still show some turns collapse into empty transcript or generic speech
|
|
- Implementation notes:
|
|
- keep local rules only on constrained replies
|
|
- improve empty-turn retry behavior for settings and OTA prompts
|
|
- capture whether stock OS uses a different yes-no prompt shape in backup versus update flows
|
|
- Exit criteria:
|
|
- spoken `yes` and `no` reliably work on backup and update prompts
|
|
- empty or missed turns retry locally without relaunching Nimbus
|
|
|
|
## Near-Term Queue
|
|
|
|
### 4. News Through Nimbus / Personal Report
|
|
|
|
- Status: `ready`
|
|
- Tags: `protocol`, `content`
|
|
- Why now: Nimbus already exposes a `news` cloud hook, so this is the next best cloud-first skill after radio.
|
|
- Current evidence:
|
|
- [ProcessCloud.ts](C:/Projects/JiboOs/V3.1/build/opt/jibo/Jibo/Skills/@be/be/node_modules/@be/nimbus/src/states/ProcessCloud.ts) checks for `cloudSkill === 'news'`
|
|
- Nimbus analytics and assets also reference `personal-report`
|
|
- Implementation notes:
|
|
- decide whether the first pass is a simple headline summary or a closer personal-report style payload
|
|
- confirm whether stock OS expects `news` as a dedicated cloud skill or under the broader personal-report family
|
|
- Exit criteria:
|
|
- `tell me the news` reaches a non-placeholder live path
|
|
- robot behavior feels Nimbus-native rather than generic chat playback
|
|
|
|
### 5. Clock Family Audit
|
|
|
|
- Status: `ready`
|
|
- Tags: `protocol`
|
|
- Why now: clock, date, timer, and alarm menu hooks are already visible in captures and the robot repo has a real `@be/clock` skill.
|
|
- Current evidence:
|
|
- [protocol-inventory.md](C:/Projects/JiboExperiments/OpenJibo/docs/protocol-inventory.md) already tracks menu intents for `askForTime`, `askForDate`, `timerValue`, and `alarmValue`
|
|
- `@be/clock` exists in the robot skill inventory
|
|
- Implementation notes:
|
|
- compare our custom time/date path against actual menu payloads
|
|
- decide whether timer and alarm should stay robot-local with cloud acknowledgement, or whether cloud needs to shape the launch and follow-up turns
|
|
- Exit criteria:
|
|
- time/date behavior stays correct
|
|
- timer and alarm launch or set correctly from both menu and voice where applicable
|
|
|
|
### 6. Photo Family Audit
|
|
|
|
- Status: `ready`
|
|
- Tags: `protocol`, `docs`
|
|
- Why now: photo confirmation improved already, and the robot skill inventory includes `gallery`.
|
|
- Current evidence:
|
|
- `@be/gallery` exists in the robot skill inventory
|
|
- current captures already show `snapshot` and related menu destinations
|
|
- Implementation notes:
|
|
- separate three flows:
|
|
- snap a picture
|
|
- photo gallery
|
|
- photobooth
|
|
- document whether each one is local-only, cloud-assisted, or upload-backed
|
|
- Exit criteria:
|
|
- known photo menu and voice phrases map to the correct local path
|
|
- capture storage expectations are documented for laptop versus hosted testing
|
|
|
|
## Discovery Queue
|
|
|
|
### 7. Weather As Cloud Report Plus Local Presentation
|
|
|
|
- Status: `discovery`
|
|
- Tags: `protocol`, `content`
|
|
- Why later: there is strong evidence for weather assets under Nimbus, but not for a standalone local skill package.
|
|
- Current evidence:
|
|
- Nimbus assets include personal-report weather content
|
|
- no standalone `@be/weather` package is present in the inspected Be skill inventory
|
|
- Questions to answer:
|
|
- is weather a dedicated cloud skill, a personal-report branch, or both
|
|
- what payload shape triggers the local animation / embodiment layer
|
|
- whether the first pass should be cloud speech only or forecast plus presentation metadata
|
|
|
|
### 8. Surprises Routing
|
|
|
|
- Status: `discovery`
|
|
- Tags: `protocol`, `content`
|
|
- Why later: `@be/surprises` is a router, not a single experience, so we should not wire this blindly.
|
|
- Current evidence:
|
|
- [SurpriseSkill.ts](C:/Projects/JiboOs/V3.1/build/opt/jibo/Jibo/Skills/@be/be/node_modules/@be/surprises/src/SurpriseSkill.ts) selects among surprise categories
|
|
- `surprises-date` and `surprises-ota` show category-specific branches already exist
|
|
- Questions to answer:
|
|
- should `surprise me` enter the top-level surprise router
|
|
- which categories still depend on cloud services versus fully local logic
|
|
- whether stock OS `1.9` differs materially from the `3.1` source snapshot here
|
|
|
|
### 9. Personal Report, Calendar, And Commute
|
|
|
|
- Status: `discovery`
|
|
- Tags: `protocol`, `content`
|
|
- Why later: these are already stubbed in `.NET`, but the robot-side ownership still needs clearer mapping.
|
|
- Current evidence:
|
|
- current `.NET` placeholders live in [InMemoryJiboExperienceContentRepository.cs](C:/Projects/JiboExperiments/OpenJibo/src/Jibo.Cloud/dotnet/src/Jibo.Cloud.Infrastructure/Content/InMemoryJiboExperienceContentRepository.cs)
|
|
- Nimbus has personal-report hooks, but the exact cloud contract still needs confirmation
|
|
- Questions to answer:
|
|
- should calendar and commute be independent feature paths or sections inside personal report
|
|
- what minimum provider data shape lets Jibo present these naturally
|
|
|
|
## Support Tracks
|
|
|
|
### 10. Hosted Capture And Storage Plan
|
|
|
|
- Status: `ready`
|
|
- Tags: `docs`
|
|
- Why now: repo-local zip bundles are fine for solo testing but not for group rollout.
|
|
- Implementation notes:
|
|
- define a clean boundary between local capture sinks and hosted archival/export
|
|
- document how group testers should submit sessions without touching repo paths directly
|
|
|
|
### 11. STT Upgrade And Noise Screening
|
|
|
|
- Status: `ready`
|
|
- Tags: `stt`
|
|
- Why now: feature work is moving again, but missed short replies still block otherwise-correct flows.
|
|
- Current evidence:
|
|
- local buffered STT still fails on some turns with `ffmpeg` / `whisper.cpp` issues
|
|
- low-energy or background-noise turns are still being sent down paths that should probably short-circuit earlier
|
|
- Implementation notes:
|
|
- evaluate lightweight waveform or energy gating before transcription
|
|
- compare a managed STT provider against the current local toolchain
|
|
|
|
## Suggested Order Of Execution
|
|
|
|
1. Radio resume and genre launch
|
|
2. ESML apostrophe fix
|
|
3. Backup / OTA yes-no reliability
|
|
4. News
|
|
5. Clock family
|
|
6. Photo family
|
|
7. Weather
|
|
8. Surprises
|
|
9. Personal report, calendar, and commute
|
|
10. Hosted capture/storage and STT improvements as parallel tracks
|