Lip Syncing with Fuse/Mixamo Avatars

Programs Needed:

Fuse (available for free if you have a Creative Cloud membership, or on Steam)

3DSMax

Cal3D Exporter (available to download from the WorldViz website)

1. Create your model in Fuse

 

2. Go to “File- Animate with Mixamo” or click on “Send to Mixamo”

 

3. Make sure “Facial Blend Shapes” is enabled and click “Finish”, then “Use this Character”

 

4. Download as an FBX model

 

5. Also, at this time you may want to select and download any animations you will wish to use

6. Open up 3DS Max and import your FBX model you just downloaded

 

7.  Carefully follow the steps for exporting using the Cal3D exporter, outlined in the Vizard documentation http://docs.worldviz.com/vizard/index.htm#cal3d_introduction.htm%3FTocPath%3DContent%20Creation%7C3D%20Models%7CCal3D%20Workflow%7C_____1

8. Create a new folder and name it whatever you want to call your avatar

9. In 3DS Max go to File- Export- Cal3D skeleton file

10. Make sure your units are set to meters in the export window, name your file and save it in the folder you created.

11. Next, select a mesh and click on the “material editor” and change the mode to “compact”

12. Rename each material so that it has sequential ID numbers in square brackets, e.g. body[0]

13. Now export your meshes, following the steps from the Cal3D exporter documentation

14. Next, select your head mesh, go to “Modify” and select “Morpher”

15. Click on where it says “Morpher” and you will see a list of all of your morphs show up. We will be using the one titled “Jaw_Down” for the lip syncing

16. Enter “100” in the value next to “Jaw_Down”

 

17. Now, export your head mesh once more and name this one something like “mSkel_m021_head_talk.CMF”

18. Next, export your materials using File- Export- Cal3D Material File

19. For the animations you will have to import each animation fbx model separately, then export using the Cal3D exporter and make sure to select the same skeleton you have been using. Save all your animations to the same folder as the other avatar files as well.

20. The next step is to create your .cfg file. Open up Vizard and select File- New Text file. Save your file as a .cfg file and give it a name. Make sure to save it in the same folder as all your other files.

21. Follow the steps in the documentation for setting up your .cfg file

22. You can test your Avatar’s morph and animations in Inspector by right clicking on your cfg file, select “Open with” and choose “Vizard 6 Inspector”

23. Once everything is setup, you can now add your Avatar to your scene, set its position, set its state, and then add the speech command by first importing the vizact module and using the code: speech = vizact.speak('jfk.wav') where “jfk.wav” is the name of any wave file you choose. You can now map this action to a keypress using vizact.onkeydown(' ', male.addAction, speech)

24. For a sample scene using a .cfg avatar and lip syncing click here . In the folder, open the FBX_lip_syncing.py in Vizard
If the lip syncing causes the head to distort in an unusual way, you may need to adjust the scale by adding vizact.speak(‘jfk.wav’, scale= [enter some value here])

 

 

How to have tracked FBX Avatars in vizconnect using 3DSMax and Mixamo’s Auto Rigging

The Autobiped script was created by Ofer Zelichover and Dan Babcock and it is freely released to Mixamo customers

The script will convert any character rigged using Mixamo auto-rigger into a Biped system in 3dsMax

  • If going from Fuse just send to Mixamo and Auto-rig

    • Go to mixamo.com

    • Select “Upload Character”

  • If coming out of other programs (such as Character Creator)

    • Bring avatar into 3DSMax and delete the skeleton (for some models may also have to merge meshes)

    • Export the mesh as an fbx

    • Upload to Mixamo and auto-rig then download

  • Import downloaded avatar into 3DSMax

  • Check size of avatar (Under Utilities- Measure)

  • May have to uncheck “Always Deform” on the mesh (except for feet) - Under Modify- Advanced Properties. This is if you get undesirable results on how the skeleton attaches to the mesh.

  • Add and run this script (AutoBiped) to convert to 3DSMax biped mesh object (Scripting- Run script)

  • Click on "Create Biped"

  • Rename all bones from Bip001*  to Bip01* in 3ds Max (Tools > Rename Objects). Expand all bones in the skeleton and select them all first.

  • Export with Cal3D

  • Use as an imported Complete Character in vizconnect

    • Start Vizconnect by going to Tools- Vizconnect in Vizard

    • Setup your trackers/ inputs/ etc. For help with vizconnect and avatars see the Vizard documentation. You can also choose from one of the common presets.

    • Click on “Avatars” tab and remove the current avatar if there is one there.

    • Add new avatar

    • Choose “imported Complete Characters”

    • In Settings under file, choose your exported .cfg file

    • Make sure to then go to the “Animator” tab under "Avatars" and map whatever trackers you are using to the avatar model

    • Drag the main camera under the head component in the scene graph, as well as any tools you may be using under the hand components.

    • Lastly, add gestures and map them to your input device

  • Afterwards you will want to manually edit the vizconnect file where the avatar is added to change the lines to use vizfx instead of viz.add :

  • import vizfx

  • avatar = vizfx.addChild(filename)

  • If wanting to change the avatar, you can just change the path in the code.

FBX Avatar to Vizard

In earlier versions of Vizard, Cal3D exports out of 3ds Max were the only option for a character workflow. As of version 6, Vizard includes the ability to use any FBX avatar. These can come from a variety of sources. In this case, we'll be using a .FBX avatar exported from a program called “Character Creator”.

Importing and Preview Lighting
First, import your avatar into Inspector. This can be done by using Open With... and selecting Vizard 6 Inspector from the list, or by running Vizard 6 Inspector and dragging the avatar's FBX file into the scene.


Read more