Drawing-Based Interactive Character

Auto-recognizing and rigging hand drawings, then applying body motion, facial expressions, and speech.

Overview

  • Period: Mar 2026 – Jul 2026 (K3I)
  • Auto-recognizes and rigs a user’s drawing, then applies motion, expressions, and voice to make it a moving character
  • Extended from body-motion animation → real-time expression transfer → a talking character

① Body-motion Animated Drawing

  • MediaPipe Pose extracts 33 body joints from the webcam → converted to BVH motion
  • AnimatedDrawings auto-detects the character and joints in the drawing → motion retargeting
  • Webcam, Rokoko, and Mixamo motions unified in a BVH library — one motion drives many drawings

Drawing recognition and rigging flow

Character detection → joint extraction → BVH motion

Motion retargeting

Simultaneous multi-character retargeting

② Real-time expression transfer — DrawFace Live

  • MediaPipe FaceLandmarker 478 landmarks → ARKit 52-channel blendshapes
  • Hand drawings use ARAP geometric deformation (original strokes preserved); standard-proportion illustrations use LivePortrait TensorRT, 27–37 ms (~30 FPS)
  • Runs entirely in the browser — no server, face video never leaves the device

Expression mirroring

ARKit 52-channel 3D expression mirroring

③ Talking Drawing Avatar

  • Local LLM (EXAONE 3.5) judges per-sentence emotion → applied to edge-TTS voice tone and facial expression together
  • JoyVASA + LivePortrait generate speech video (mouth interior pixels generated), fully local
  • TensorRT acceleration + fragment streaming + parallel emotion judging → playback start 4.8 s → 2.1 s

MediaPipe · AnimatedDrawings · BVH · ARAP · ARKit 52ch · WebGL · LivePortrait · TensorRT · JoyVASA · EXAONE 3.5 · edge-TTS · FastAPI

Code: github.com/ingon1026/drawface-live · github.com/ingon1026/talking-drawing-avatar · Demo: HF DrawFace Live