This document tracks the currently observed cloud surface area for Jibo and helps keep the .NET port aligned with real behavior captured by the Node prototype.
Current robot findings suggest the preferred OpenJibo bootstrap path is to inject a new region configuration rather than treat host overrides as the only integration seam.
The first two are the clearest current OpenJibo injection points. The others should remain on the audit list while endpoint and behavior mapping continues.
- fixture-backed joke-turn payload fidelity for `CLIENT_ASR -> LISTEN -> EOS -> delayed SKILL_ACTION`, including Node-like `EOS` envelope fields and the currently observed joke `SKILL_ACTION` metadata shape
The newest repo-root websocket capture at [captures/websocket/20260416.events.ndjson](/artifact-output/captures/websocket/20260416.events.ndjson) adds more grounded websocket discovery without implying broad protocol coverage.
Observed `CLIENT_ASR` transcript-bearing turns now include:
-`tell me a joke`
-`do a dance`
-`surprise me`
-`personal report`
-`tell me about the weather`
-`tell me about my calendar`
-`what does my commute look like`
-`tell me about the news`
Observed menu-driven `CLIENT_NLU` intents now include:
-`loadMenu`
-`askForTime`
-`askForDate`
-`start`
-`timerValue`
-`set`
-`alarmValue`
Observed entity/rule shapes from those menu flows include:
-`askForTime` with `entities.domain = "clock"` and `rules = ["clock/clock_menu"]`
-`askForDate` with the same `clock` menu rule family
-`timerValue` with timer duration entities
-`alarmValue` with alarm time entities such as `ampm` and `time`
Current `.NET` parity for that new slice is still intentionally partial:
- menu-side `CLIENT_NLU` replies now preserve the observed inbound intent/rules/entities in the synthetic outbound `LISTEN` payload
-`askForTime` and `askForDate` are now fixture-backed as mapped menu intents
-`do a dance` is now recognized as a distinct chat/dance intent in the current synthetic path
Still unknown:
- whether `surprise me`, `personal report`, weather, calendar, commute, and news should map to richer skill-specific websocket payloads
- whether menu-side clock/timer/alarm flows require additional websocket messages beyond the currently observed `LISTEN` and `EOS`
- how much of those flows are actually completed robot-side versus merely acknowledged by the cloud
### Buffered Audio / ASR Direction
The `.NET` hosted implementation now has two STT lanes:
- existing synthetic transcript-hint replay for fixture-driven parity work
- a new opt-in local buffered-audio path that preserves websocket Ogg/Opus frames and can invoke external `ffmpeg` plus `whisper.cpp`
That local tool-based path is intentionally experimental and disabled by default. Its purpose is to let us iterate on real buffered-audio decoding in `.NET` without changing the stable cloud-first architecture or claiming production ASR parity yet.
Future provider options still under consideration:
- local decode/transcribe in `.NET` using preserved websocket audio plus external tools
- Azure Speech as a hosted STT option for the long-term cloud path
- direct managed Opus decode later if a library proves stable enough for the hosted deployment target
Current raw-audio fallback behavior remains explicitly synthetic:
- when a buffered-audio turn can be resolved through the synthetic transcript-hint seam, `.NET` now auto-finalizes and emits `LISTEN` + `EOS` + `SKILL_ACTION`
- when the turn crosses the finalize threshold without a usable transcript, `.NET` now emits a fallback `LISTEN` + `EOS` + generic `SKILL_ACTION` rather than leaving the robot hanging on an unfinished turn
- that fallback is a compatibility measure inspired by the Node oracle, not a claim of real ASR understanding
That separation is intentional. The synthetic STT path currently exists only to support fixture-driven replay while parity work continues. It should be treated as an internal compatibility seam, not as the final production ASR design.
The first real `.NET` robot run has confirmed only an early startup slice so far:
-`api.jibo.com` startup HTTP requests are reaching the `.NET` cloud
-`Notification.NewRobotToken` is active in the robot startup sequence
-`api-socket.jibo.com/{token}` is being accepted live
The first live run has not yet shown full startup parity with the working Node server. In particular, the successful Node run continues into additional health/log cadence after token issuance and socket acceptance, while the current `.NET` run has not yet reproduced that full progression consistently.
Sanitized fixtures live under [src/Jibo.Cloud/node/fixtures](/OpenJibo/src/Jibo.Cloud/node/fixtures) and should be expanded as real traffic is captured.
