imagination XD

realizing dream


1 Comment

[Unity] Realtime Lipsync


Back on my manual lipsync using Tagarela. This one is faster to do and also easier, but of course, you’ll get only Anime lipsync (Anime lipsync only have 1 open mouth, there’s no O, U, I or any other mouth shape than A).

The method I’m using is just get Audio Source frequency. So basically it’s like music visualisation but for mouth.

What you need

  • Head mesh with 1 mouth blendshape
  • Voice
  • Audio Source

Here’s the script you should attach to head mesh.

using UnityEngine;
using System.Collections;

public class RealtimeLipsync : MonoBehaviour {

//LIPSYNC CORE
 [Header("Lipsync")]
 public AudioSource audioSource;
 public int mouthBlendShape;
 private float[] _samples = new float[64];
 private float clampedLipsync = 0f;

//PRIVATES
 private SkinnedMeshRenderer blendMesh;

void Awake(){
 blendMesh = GetComponent<SkinnedMeshRenderer>();
 }

void LateUpdate(){
 if(audioSource) DoLipsync();
 }

void DoLipsync(){
 if(audioSource.isPlaying){
 audioSource.GetSpectrumData(_samples,0,FFTWindow.BlackmanHarris);

clampedLipsync = Mathf.Clamp(_samples[2]*3000,0,100);

blendMesh.SetBlendShapeWeight(mouthBlendShape,clampedLipsync);
 }
 }
}

And here’s how to do it

  1. Attach this script to your head mesh.
  2. Also don’t forget to add audio source and assign audio source in this script. You can put Audio Source in whereever you want, but remember to assign audio source variable on this script.
  3. To play sound, use like audioSource.clip = <your clip name>; then audioSource.Play();
  4. Assembly-CSharp - VertexScriptCOREPlayerCORE.cs - MonoDevelop-Unity
  5. Here’s my example to use this. I put it on my player script. It’ll automatically works whenever you have audioSrc.clip = clipName; audioSrc.Play();
  6. That’s it, your mouth will move whenever the audio src playing sounds.