imagination XD

realizing dream

[Unity] Keyed Lipsync using Blendshape + Tagarela

3 Comments

edfedfe

I still research on how to make automated keyless lipsync. So, I found this plug in, it’s free, and it’s easy to use so I got that goin’ for me, which is nice.

Recipe (What you’ll need) :

  • PC with Unity 4.3 or above
  • Tagarela
  • Mesh that has blendshape (If you don’t know how to make blendshape, read this)
  • Blendshape that has at least A,I,U,E,O keys. (if you are too lazy to make E key, you can use I for every E)

Alright let’s start! I’ll use Kagamine Len by Animasa model as example.

  1. After finished download, open it in Unity.
  2. Select an object that has blendshape / skinned mesh renderer on it. Open Component > Tagarela. After that insert Audio Clip and then press Add.lt1
  3. Click Open Editor then press Create New Animation, choose Audio Sync, name it whatever you want, press save.lt2
  4. First, slide the tiny slider near audio’s start, then press + to add new keyframe, locate the new keyframe at the middle of audio, and adjust the blendshape below.
    lt3
  5. I use Unity chan’s konnichiwa (こんにちは私ユニティちゃん). 1 is all Zero, 2 is 100% 001 (O (I forgot to change name), 3 is also zero because it’s closing its mouth when saying kon’, and 4 is 100% i, when it says ni.
    lt4
  6. Set it till near perfection. Also remember to set keyframe to zero on all flat audio which is mute (does your mouth open when you say nothing ? unless you breathing with mouth, like me). For example, your voice was “Hello World”, 1 all zero keyframe was located before Hello, 2 keyframe was located in ‘He’ has 100% ‘E’, and 3 keyframe was located in the ‘llo” has 100% ‘O’ and 0% ‘E’, at the end of ‘Hello’ add 4 keyframe that is zero keyframe, and 5 keyframe was located at the beginning of ‘World’ which also zero. 6 keyframe repat like 2 keyframe but with different value.
    lt5
  7. After you done naturalising the lipsync, always remember to put zero keyframe at the start of audio not voice. and then save it. You may ask how to use it on gameplay, well, you’ll only need to put this GetComponent<Tagarela>().Play( <Your animation number (mine is 0 because it’s the first animation)> ); on a C# script. Also attach this script on the same place where your blendshape mesh.
    lt6
  8. Ta-da… Not bad I see. I also forgot to add some Mecanim animator, so it looks like stupid people says “Hello, I’m Unity-chan” but actually you are Len Kagamine.

Next, I’ll research much easier to use lipsync or Anime-like lipsync. Most anime I watch, they only use 1 key. Even they say “What the hell are you doing”, their mouth will be always open and close only, no wide or narrow. Also remember to make it jagged, so it’ll look like Anime. I also researching on how to combine cloth shader with Toon shader like cloth crumple shadow in usual anime.

Ah and Imagine-chan’s model has finished, and it’ll release probably this november. I think I’ll need to remodel, because her head looks ridiculous. However, morphless FBX and PMX download will be available. I’m kinda busy working. My supervisor wont let me to do this, so that’s why I wont upload this model soon.

Advertisements

Author: IMGVERTEX

Started making games since 13 year old. 3D CGI Weeaboo | VFX Otaku | Anime Researcher | Indie Game Dev | VR Chuunibyou

3 thoughts on “[Unity] Keyed Lipsync using Blendshape + Tagarela

  1. “Even they say “What the hell are you doing”, their mouth will be always open and close only..”

    Lol i already done it…
    here example nekomimi glados again

    its detect peak of the sound. Its got worst effect on android but good at window

    • yeah I also had a though on using it back then. I planned it to use AudioListener.GetSpectrumData for lipsync. Lipsync can be like Audio Visualisation but it’s for blendshape. But, I’m too lazy and busy leveling up my CG skills to develop that.

  2. Pingback: [Unity] Realtime Lipsync | imagination XD

FEED ME

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s