Chronicles v0.1.0

• Frontend integration with VRM (December 5)

  • VRM idle animation (December 6)
  • VRM blinking (December 7)

• Mouth (June 8)

  • Created the unspeech project (December 13)
    • Integrated TTS (June 8)
    • Integrated 11Labs
      • Encapsulated a standalone 11Labs package (December 3)

• Hearing (December 12)

  • Implemented a talk button (June 9)
  • Audio transcription
    • Frontend streams audio to the backend
      • Used socket.io for bidirectional communication via WebSocket Socket.IO (June 10)
        • Socket.io is not actually based on WebSocket
          • node.js - What is the major scenario to use Socket.IO - Stack Overflow
          • node.js - Differences between socket.io and websockets - Stack Overflow
        • Frontend uses the socket.io-client package, pnpm i socket.io-client
          • WebSocket has good support, and Nuxt's Nitro also supports it. How to use with Nuxt | Socket.IO
        • Backend uses the socket.io package, pnpm i socket.io
        • Nuxt 3 with socket.io
          • richardeschloss/nuxt-socket-io: Nuxt Socket IO - socket.io client and server module for Nuxt
          • javascript - Socket.io websocket not working in Nuxt 3 when in production - Stack Overflow
          • adityar15/nuxt3socket (github.com)
      • Used WebRTC for audio streaming, VueUse also supports this
        • Nuxt and Nitro do not yet support this, so skipping for now. Could consider it for group chats or Discord.
        • Tutorials:
          • Getting started with media devices | WebRTC
          • WebRTC | JavaScript Standard Reference Tutorial
    • Transformers.js + Whisper would suffice
      • Chrome / Edge now support WebGPU
        • There’s a demo available: Real-time Whisper WebGPU - a Hugging Face Space by Xenova (not open source for now)
      • Inference for Whisper can be done directly in the browser
      • WebGPU is not supported yet (now supported)
        • 🤗 Transformers.js + ONNX Runtime WebGPU in Chrome extension | by Wei Lu | Medium
      • Considering embedding Whisper.cpp via a Node.js CPP Addon
      • whisper.cpp
    • Tutorials:
      • Realtime video transcription and translation with Whisper and NLLB on MacBook Air | by Wei Lu | Medium
      • 🤗 Transformers.js + ONNX Runtime WebGPU in Chrome extension | by Wei Lu | Medium
  • Whisper WebGPU Demo (December 10)
  • MicVAD Demo (December 11)
  • MicVAD + ONNX Whisper real-time transcription (December 12)
  • dcrebbin/oai-voice-mode-chat-mac: Adds realtime chat for ChatGPT Voice Mode [Unofficial]

• Facial Expressions (July 9)

  • Frontend VRM facial expression control (December 7)

• Multilingual Support

  • UI multilingual support
    • feat: basic i18n (#2) (December 13)

• UI Optimization

  • Canvas scene mobile adaptation (December 5)
    • Improved Live2D Viewer (December 5)
    • Live2D model scaling and adaptive ratio (December 5)
  • Safe area for the screen (December 4)
  • Settings menu & overflow optimization (December 7)

Models #

• VRM
  • Thanks to kwaa for guiding me to @pixiv/three-vrm
  • Related tools and plugins:
    • VRM Add-on for Blender
    • VRM format — Blender Extensions
    • VRM Posing Desktop on Steam
    • Characters Product List | Vket Store
  • Animation support: VRM Animation .vrma
    • vrma Spec
    • 3D Motion & Animation popular doujin goods available online (Booth)
      • Seven VRM animations (.vrma) - VRoid Project - BOOTH
        • VRoid Hub introduces Photo Booth for animation playback! "VRM Animation (.vrma)" now listed on BOOTH, plus 7 free animation files!
        • malaybaku/AnimationClipToVrmaSample: Sample Project to Convert AnimationClip to VRM Animation (.vrma) in Unity