Mocap to Molehill, via 3D Studio MAX and Away3D Broomstick

After seeing David Lenaerts excellent Broomstick example, I've been eager to find a workflow to bring motion capture data into Flash. It took a while to work out, not being familiar with the MD5 format at all, but it turned out those nice guys at Away3D had already done all the hard work. I was thinking I'd have to map the bones of MAX's Biped to some other custom "MD5 scheme", but to my delight I discovered the parser is extremely generic and works with any bone structure I throw at it. I'd like to share my findings, not least in case I forget them before I need to do it again :O)

First a quick demo to show where we're headed...

(Click image to open, Flash 11 or incubator is required.) 

WARNING: The page loads around 14Mb of Motion Capture data, so be patient. When it's finished the "player" should start moving on his own. Click to put the flash window in focus, then use arrow keys to run, and the top row of keys (Q to I) to trigger different mocap files.  

You'll notice the motion is quite faithfully replicated, though there are some problems with the figure/bone structure around the shoulders. That could be fixed given the time, but I was satisfied with this as a proof of concept.

For this recipe you will need:

• Rigged model, downloaded free from TurboSquid 
• Copy of 3D Studio Max (My copy is Max 9, others should work just as well)
• MD5 Export Script http://www.doom3world.org/phpbb2/viewtopic.php?t=1914

If you use that particular model you will see that it comes already rigged using Biped.

Biped is one of the Bone systems that ships with Max. Some love it, some hate it. The good thing is there's alot of free motion capture data recorded in the ".bip" format, to get you started.

Normally you use Max's Physique modifier together with Biped. This is a quite advanced system using muscles and tendons. The MD5 format doesn't support that system directly, so we need to use a Skin modifier instead. 

(Click for larger image)

After you open the soccer player, select the model, right click on the modifier and select "Collapse All".

Now you should have an Editable Mesh. This is the only format the MD5 exporter can work with so it's worth remembering.

Now we have an Editable Mesh and a Biped lined up with each other, but not attached in any way.

With the player still selected, choose the "Skin" modifier.

Where it says Bones, press the "Add" button and assign all the bones of the Biped to the Skin modifier (this will be all the objects in the scene). Then you modifier stack should look like this:

Now it's time to load some Motion Capture (or Mocap) Data. Select Bip01 (easiest by pressing the H key and selecting it from the list of scene objects) and then under the motion panel, press the Load icon and find a .bip file from your harddrive. If you don't have any .bip files, here's one from the Max Tutorials.

You will find a folder full of .bip files on your Max DVD under Tutorials, as well as loads of free downloads on the net. 

So now we should have an animation when you press play. The Soccer player should move in perfect synch with your Biped. You might notice a few odd bulges, especially round the legs and feet.

This is caused be the legs "pulling" on each other, and it's now we need to understand a little bit about how Skin and Bones (in this case Biped) work. 

Each bone acts upon a group of vertices in the Skin. A vertex may have more than one bone acting upon it. So for each bone there is an array of indices for the vertices in the model that it effects, and not only that, but each of those vertices has a "weight", i.e. the amount the bone pulls on that vertex.

To make it easier in 3D Studio, and other 3D programs, the bones are assigned an Envelope. This simply means a sausage shape around the bone's core that encapsulates the vertices. If in 3D studio, we select the soccer player again and "Edit Envelopes", when we select a bone from the list, the envelope of that bone is shown in the viewport.

Note, your own viewport may not look exactly the same, depending on the settings under "Display".

In the illustration you can see the envelope of the Left Foot. There are highlighted in colour, the vertices it effects. Red means they're effected a lot, blue means they're effected a little. Notice that some vertices on the other foot are highlighted in blue. This has happened because in the models start position, the feet were close together, and therefore the vertices on one foot were within the envelope of the other. This is the cause of many issues when setting up Bone systems, also referred to as Rigging.

Max offers many ways of editing envelopes, but I'm only going to talk about one quick and dirty method for now. The darker red circles around the foot represent the limits of the effect area for that bone. By clicking on the small grey boxes where they intersect you can select that outer limit, and then adjust it's radius, either by dragging on it, or more precisely with the number spinner in the menu. By making it smaller, at some stage the vertices of the other foot are no longer effected. You need to adjust both feet before the model starts to look right. Then continue with the carves and thighs.

This subject is covered better in other tutorials on the net, but that should get you started.

NOTE: Make sure you have no zero weights in your model. If a vertex is connected to a bone with a weight of 0.000, it has the effect of attaching that vertex to the root of the model when imported into Away3D.

To avoid this, when you have finished rigging, set the "Remove Zero Limit" in the skin modifier palette to 0.01 and hit "Remove Zero Weights". You are then ready to export.

Finally it's time to export to MD5. Under Max Script, choose Run Script and find the MD5 exporter script you downloaded earlier. I tend to put my scripts in the script folder of Max, just because it's the default directory opened by Max. 

Choose export and you'll be prompted to choose a name for your file. Then you will be asked for another name for the animation file. These two files (.md5mesh and .md5anim) are needed for the mocap animation, and generally speaking need to come from the same model. However, when you have exported one mesh, you can then apply another bip file and tick "export md5anim only" so you use the same model with different motion files, as in the example.

NOTE: When you press "Export", your computer will most likely freeze. Max Script, much like other scripting languages, can paralyse your processor when running through big loops, and believe me there's a lot of number crunching and writing going on here. My advice, go make a cup of coffee. On my machine the exporter hasn't actually crashed yet, though I have sometimes lost patience with it and restarted Max. You can improve performance, by going into Max's preferences and increasing the allocated memory for script. (Customize>Preferences>Max Script> Initial Heap Allocation). I set mine to 50Mb. 

Now all that's left is to look at the source for David Lenaerts example from the Away3D repository and start swapping his files out with your own. I have included the entire source of my effort here, though admittedly it's not commented very well. The material has been made into a mipmap, i.e. resized to a square in Photoshop.
For some reason the texture coordinates get reversed between md5 and Broomstick. I guess this may change in future builds, but for now the work around was simply to reverse the number on the players shirt in Photoshop.

http://videometry.net/broomstick/assets/soccer/soccerExample.zip

Disclaimer: The Broomstick branch of Away3D is in constant development, as is the Flash player, and there is no guarantee these files will work with your installed library, without a little tweeking :O)
(They are compiled against the github version, 24th may.)

NOTE: A difference in the coordinate space between MD5 format and Flash results in all models and animations being mirrored on the x-axis when viewed in Away3D 4.0
To see animations and models as they appear in MAX, simply apply a scaleX = -1; after import to Away3D.

Enjoy! And please comment with your successes, failures, comments and tips!

UPDATE: In response to the high demand, (from MatOfLink at least) here's a method for putting guns in hands, shoes on feet, hats on head, etc.
When your MD5 mesh has loaded, add a Loader3D to it. This is where your prop will appear.


gunholder = new Loader3D();
gunholder.addEventListener(LoaderEvent.RESOURCE_COMPLETE, adjustGun);
_mesh.addChild(gunholder);


So far so good. When the gun loads it will likely be twisted in relation to the rest of the model, depending on which format it is and which software it was exported from. For obj files coming from Max I've found this simple routine aligns the object the same way as in Max. At the same time you can apply materials.


private function adjustGun(evt:LoaderEvent):void {

var i:int;
var gunMat:BitmapFileMaterial = new BitmapFileMaterial("assets/images/gun.jpg");
for (i = 0; i < gunholder.numChildren; i++) {
var m:Mesh = gunholder.getChildAt(i) as Mesh;
m.rotationX = 90; 
m.rotationY = 90;
m.material = gunMat;
}
}


Now your gun should be in the viewport. You need to know the index of the particular bone you want to follow. This you can either read in the debugger, by trial and error, or via the jointIndexFromName from name function on the skeleton class (later...)


Once you know the index of the bone (called "joint" in Away3d) you want to follow add this to your rendering loop:


gunHolder.transform = (_mesh.animationState as SkeletonAnimationState).globalPose.jointPoses[26].toMatrix3D();


And that's it! You basketball player is now armed and dangerous.
If I get round to it I'll post an updated source. Otherwise shout out below for help!


Now let's see some games...