Home/GANDER docs/GANDER talking characters

GANDER talking characters

Preparing character models with face blendshapes for gander::face(): the two tools and what they output.

faces.mdllms-full.txt

gander::face() needs a model with face blendshapes. Meshy and Forge characters have none. Two tools give you one (both CC0-safe, both output plain GLB that gander loads; run them in the game folder):

F=/opt/gander/tools/face
python3 $F/make_human.py --sex female --age 30 --race asian --hair ponytail01 --out assets/models/hero.glb
python3 $F/add_face.py assets/models/soldier.glb assets/models/soldier_talking.glb
python3 $F/bake_lipsync.py assets/voice/*.mp3 --text "the exact line"     # one line at a time with --text

make_human.py: a realistic human that talks

MakeHuman body (CC0) with a Mixamo-named skeleton (mixamorig:Hips, mixamorig:Head ...: gander animation, IK, look_at and ragdolls find the bones), skin, eyes, eyebrows, eyelashes, teeth, tongue, hair, clothes, and 67 blendshapes on the head (52 ARKit + 15 visemes). Feet at y = 0, facing +Z, meters. About 4 s and 19 MB.

Option Values
--sex / --gender female male / 0..1 blend
--age years, 1..90
--muscle --weight --height --proportions 0..1, 0.5 average
--race caucasian african asian, or a mix african=0.5,caucasian=0.5
--skin auto (matches age / sex / race) or a skin name
--eyes brownlight (default) brown blue bluegreen green grey ice lightblue deepblue
--hair / --hair-color auto none short01..04 bob01 bob02 ponytail01 long01 braid01 afro01 / hex 3b2a1e
--eyebrows --eyelashes eyebrow001..012, eyelashes01..04, none
--clothes auto, none, or comma list: male_casualsuit01..03 male_worksuit01 female_casualsuit01..02 female_sportsuit01 shoes01..03
--texture-size skin texture size, default 2048 (1024 halves the file)

--style toon makes an anime character instead: large eyes and a small nose and chin on the same basemesh (the 67 face shapes still fit), one flat skin colour (--skin-color ffe2d0), painted anime irises (--eye-color 8a5cd6), clothes repainted in flat tones, and cel materials with ink lines (--outline 0.0022) that gander draws as material_toon + material_outline on load. --targets eyes/eye-scale-incr=0.5,chin/chin-triangle=0.4 adds any MakeHuman detail target (eyes, head, chin, nose, mouth, cheek, neck ...; a name without l- / r- shapes both sides). Pieces modelled elsewhere go on with --head-mesh head.glb --head-local head.json (an anime head with its own face shapes instead of the basemesh's head and neck; --head-scale 1.14), --hair-mesh hair.glb --hair-chains chains.json (hair; its swinging pieces get hair_<piece>_<n> bones for gander::spring_bones(anim, "hair_", 3.0, 0.25, 0.25)), and --outfit-texture garment=painted.png. --head-info head.json writes the head's measurements for the tool that models the hair.

--list prints everything installed. The model has no animations: retarget clips onto it in Blender (Mixamo-style names), or pose it with gander's bone_rotation / ik / look_at, and stand it idle with subtle look_at and the face's blinks and gaze.

add_face.py: give a generated character a face

Input: a rigged GLB (Meshy rigging, Forge rig) with a head bone (Head, mixamorig:Head, --head-joint NAME). It finds the face from the texture (skin tones), fits a CC0 template head onto it, and transfers the 52 ARKit + 15 viseme shapes onto the character's own head vertices. Mesh, texture, skeleton and animations stay as they were; the head moves to its own mesh node with the blendshapes. Helmets, hair and hoods do not deform. 20 to 50 s.

Limits, by design: generated characters have closed mouths and painted eyes, so the mouth talks with the lips and jaw but does not open a cavity (--jaw 0.7 default; lower for less stretch), and a blink narrows the painted eye. Expressions (smile, brows, surprise, anger) read clearly. For close-up dialogue scenes prefer make_human.py characters.

vrm_import.py: bring a VRM avatar

python3 $F/vrm_import.py avatar.vrm assets/models/avatar.glb: a VRoid / VRM 1.0 or 0.x avatar with its expression shapes renamed for gander::face (A I U E O, Blink, Joy, Angry, Sorrow, Fun, Surprised, eyeLook*) and its humanoid bones renamed to Mixamo names. Its cel shading and ink lines load as they are. Spring bones and constraints are not converted. It prints the avatar's licence terms: check them before shipping someone else's avatar.

bake_lipsync.py: the best sync for voice lines

Writes <line>.visemes.json (Rhubarb Lip Sync format) next to each audio file; gander uses it automatically instead of analyzing the audio live. With Rhubarb installed and --text given, the sync is phonetic; otherwise it bakes gander's own analyzer (still smoother than live). WAV directly, MP3 / OGG via ffmpeg. Bake every voice line you generate (the TTS text is the --text). A baked line also starts instantly: an unbaked one is decoded in full the first time it plays (a frame stall on a long line), so there is no need to pre-play lines at load.

In the game

import gander from "./.gander/gander";

var hero: gander::Anim;
var hero_face: gander::Face;
var line1: gander::Sound;

fn main() {
    gander::config("Talk", 1280, 720, 0);
    hero = gander::animator(gander::model("assets/models/hero.glb"));
    hero_face = gander::face(hero);                  // logs "52/52 ARKit shapes, 15/15 visemes"
    line1 = gander::sound("assets/voice/line1.mp3");
}

export "gander_frame" fn frame(dt: f32) {
    if gander::action_pressed("interact") && !gander::face_talking(hero_face) {
        let v = gander::face_say(hero_face, line1, 1.0);
        gander::face_emotion(hero_face, "happy", 0.7, 0.4);
    }
    let head = gander::position(gander::bone(hero, "Head"));
    gander::camera(head + gander::v3(0.0, 0.08, 0.7), head + gander::v3(0.0, 0.08, 0.0), 30.0);
    gander::face_look(hero_face, gander::camera_pos(), 1.0);
    gander::look_at(hero, "Head", gander::camera_pos(), gander::v3(0.0, 0.0, 1.0), 0.5, 40.0);
    gander::draw_animated(hero, gander::identity());
}

Data: the tools read CC0 MakeHuman data from $GANDER_FACE_DATA (installed by fetch_assets.py; see tools/face/LICENSES.md).