Tuesday 17 May 2011

Mocap to Molehill, via 3D Studio MAX and Away3D Broomstick

After seeing David Lenaerts excellent Broomstick example, I've been eager to find a workflow to bring motion capture data into Flash. It took a while to work out, not being familiar with the MD5 format at all, but it turned out those nice guys at Away3D had already done all the hard work. I was thinking I'd have to map the bones of MAX's Biped to some other custom "MD5 scheme", but to my delight I discovered the parser is extremely generic and works with any bone structure I throw at it. I'd like to share my findings, not least in case I forget them before I need to do it again :O)

First a quick demo to show where we're headed...

(Click image to open, Flash 11 or incubator is required.) 

WARNING: The page loads around 14Mb of Motion Capture data, so be patient. When it's finished the "player" should start moving on his own. Click to put the flash window in focus, then use arrow keys to run, and the top row of keys (Q to I) to trigger different mocap files.  

You'll notice the motion is quite faithfully replicated, though there are some problems with the figure/bone structure around the shoulders. That could be fixed given the time, but I was satisfied with this as a proof of concept.
For this recipe you will need:
If you use that particular model you will see that it comes already rigged using Biped.
Biped is one of the Bone systems that ships with Max. Some love it, some hate it. The good thing is there's alot of free motion capture data recorded in the ".bip" format, to get you started.

Normally you use Max's Physique modifier together with Biped. This is a quite advanced system using muscles and tendons. The MD5 format doesn't support that system directly, so we need to use a Skin modifier instead. 
(Click for larger image)

After you open the soccer player, select the model, right click on the modifier and select "Collapse All".
Now you should have an Editable Mesh. This is the only format the MD5 exporter can work with so it's worth remembering.
Now we have an Editable Mesh and a Biped lined up with each other, but not attached in any way.
With the player still selected, choose the "Skin" modifier.
Where it says Bones, press the "Add" button and assign all the bones of the Biped to the Skin modifier (this will be all the objects in the scene). Then you modifier stack should look like this:


Now it's time to load some Motion Capture (or Mocap) Data. Select Bip01 (easiest by pressing the H key and selecting it from the list of scene objects) and then under the motion panel, press the Load icon and find a .bip file from your harddrive. If you don't have any .bip files, here's one from the Max Tutorials.


You will find a folder full of .bip files on your Max DVD under Tutorials, as well as loads of free downloads on the net. 
So now we should have an animation when you press play. The Soccer player should move in perfect synch with your Biped. You might notice a few odd bulges, especially round the legs and feet.


This is caused be the legs "pulling" on each other, and it's now we need to understand a little bit about how Skin and Bones (in this case Biped) work. 

Each bone acts upon a group of vertices in the Skin. A vertex may have more than one bone acting upon it. So for each bone there is an array of indices for the vertices in the model that it effects, and not only that, but each of those vertices has a "weight", i.e. the amount the bone pulls on that vertex.
To make it easier in 3D Studio, and other 3D programs, the bones are assigned an Envelope. This simply means a sausage shape around the bone's core that encapsulates the vertices. If in 3D studio, we select the soccer player again and "Edit Envelopes", when we select a bone from the list, the envelope of that bone is shown in the viewport.

Note, your own viewport may not look exactly the same, depending on the settings under "Display".
In the illustration you can see the envelope of the Left Foot. There are highlighted in colour, the vertices it effects. Red means they're effected a lot, blue means they're effected a little. Notice that some vertices on the other foot are highlighted in blue. This has happened because in the models start position, the feet were close together, and therefore the vertices on one foot were within the envelope of the other. This is the cause of many issues when setting up Bone systems, also referred to as Rigging.
Max offers many ways of editing envelopes, but I'm only going to talk about one quick and dirty method for now. The darker red circles around the foot represent the limits of the effect area for that bone. By clicking on the small grey boxes where they intersect you can select that outer limit, and then adjust it's radius, either by dragging on it, or more precisely with the number spinner in the menu. By making it smaller, at some stage the vertices of the other foot are no longer effected. You need to adjust both feet before the model starts to look right. Then continue with the carves and thighs.
This subject is covered better in other tutorials on the net, but that should get you started.

NOTE: Make sure you have no zero weights in your model. If a vertex is connected to a bone with a weight of 0.000, it has the effect of attaching that vertex to the root of the model when imported into Away3D.
To avoid this, when you have finished rigging, set the "Remove Zero Limit" in the skin modifier palette to 0.01 and hit "Remove Zero Weights". You are then ready to export.

Finally it's time to export to MD5. Under Max Script, choose Run Script and find the MD5 exporter script you downloaded earlier. I tend to put my scripts in the script folder of Max, just because it's the default directory opened by Max. 


Choose export and you'll be prompted to choose a name for your file. Then you will be asked for another name for the animation file. These two files (.md5mesh and .md5anim) are needed for the mocap animation, and generally speaking need to come from the same model. However, when you have exported one mesh, you can then apply another bip file and tick "export md5anim only" so you use the same model with different motion files, as in the example.

NOTE: When you press "Export", your computer will most likely freeze. Max Script, much like other scripting languages, can paralyse your processor when running through big loops, and believe me there's a lot of number crunching and writing going on here. My advice, go make a cup of coffee. On my machine the exporter hasn't actually crashed yet, though I have sometimes lost patience with it and restarted Max. You can improve performance, by going into Max's preferences and increasing the allocated memory for script. (Customize>Preferences>Max Script> Initial Heap Allocation). I set mine to 50Mb. 

Now all that's left is to look at the source for David Lenaerts example from the Away3D repository and start swapping his files out with your own. I have included the entire source of my effort here, though admittedly it's not commented very well. The material has been made into a mipmap, i.e. resized to a square in Photoshop.
For some reason the texture coordinates get reversed between md5 and Broomstick. I guess this may change in future builds, but for now the work around was simply to reverse the number on the players shirt in Photoshop.

Disclaimer: The Broomstick branch of Away3D is in constant development, as is the Flash player, and there is no guarantee these files will work with your installed library, without a little tweeking :O)
(They are compiled against the github version, 24th may.)

NOTE: A difference in the coordinate space between MD5 format and Flash results in all models and animations being mirrored on the x-axis when viewed in Away3D 4.0
To see animations and models as they appear in MAX, simply apply a scaleX = -1; after import to Away3D.

Enjoy! And please comment with your successes, failures, comments and tips!

UPDATE: In response to the high demand, (from MatOfLink at least) here's a method for putting guns in hands, shoes on feet, hats on head, etc.
When your MD5 mesh has loaded, add a Loader3D to it. This is where your prop will appear.


gunholder = new Loader3D();
gunholder.addEventListener(LoaderEvent.RESOURCE_COMPLETE, adjustGun);
_mesh.addChild(gunholder);


So far so good. When the gun loads it will likely be twisted in relation to the rest of the model, depending on which format it is and which software it was exported from. For obj files coming from Max I've found this simple routine aligns the object the same way as in Max. At the same time you can apply materials.


private function adjustGun(evt:LoaderEvent):void {

var i:int;
var gunMat:BitmapFileMaterial = new BitmapFileMaterial("assets/images/gun.jpg");
for (i = 0; i < gunholder.numChildren; i++) {
var m:Mesh = gunholder.getChildAt(i) as Mesh;
m.rotationX = 90; 
m.rotationY = 90;
m.material = gunMat;
}
}


Now your gun should be in the viewport. You need to know the index of the particular bone you want to follow. This you can either read in the debugger, by trial and error, or via the jointIndexFromName from name function on the skeleton class (later...)


Once you know the index of the bone (called "joint" in Away3d) you want to follow add this to your rendering loop:


gunHolder.transform = (_mesh.animationState as SkeletonAnimationState).globalPose.jointPoses[26].toMatrix3D();


And that's it! You basketball player is now armed and dangerous.
If I get round to it I'll post an updated source. Otherwise shout out below for help!


Now let's see some games...





15 comments:

Vic said...

Nyce!

Anonymous said...

Hi , I liked your tutorial :)

I have a problem concerning Away3D. Could you please help me ?

Here is how I could sum up my problem :

I use FlashDevelop, my models are in MD5 ( mesh + anims ) (and for your information i'm not the modeler, i'm just a developer).

So , let's say i'm trying to make a man holding a gun in his hand. The man is animated (running for example)

Unfortunately, I have no idea how to make the gun "bounded" to the hand; as I'm hearing it : the gun should be at the right position and following the moves of the hand.

Until now, all I'm doing is creating the 2 meshes and using the code " human.addChild(gun) " but the gun doesn't follow the hand.
Another important thing to be said : I must not make an animation on the gun so it follows the hand , because I want to use the gun on others models.

Thanks for reading :)

Videometry.net said...

Good question!
I haven't tried this, but my strategy would be to assign the position of a finger bone to the gun.
So in the render loop.

gun.position = myfingerBone.postion;

Just how you get a reference to a particular bone, I'd have to delve into the skeleton classes.

Anonymous said...

Alright, that's what I'm on ;)
But not easy digging in all this :/

If you ever work on that and find something , I hope you will post here :D

Thx !

Anonymous said...

Mhh I don't find the bones...

Videometry.net said...

There you go Mat! I'm afraid you didn't leave a mail address, so I hope you notice the update.

barathrumm said...

Thanks this is great info :)

Dennis Day said...

Is it just me or does your demo not work with Flash Player 11 Beta 2? I never could get David Laenert's demo to work with either Beta.

Videometry.net said...

Hi Dennis, now updated for RC1 :o)

the_drOne said...

Nice demo, not as Matoflink, i'm a modeler, and need to export MD5 mesh and anim, it's working with max 2008, but i cant find a working solution for max 2012 !

ANyone did find a MD5 exporter for max 2012 ?

Anonymous said...

The exporter does not working with 3dsm 2012.

mary_fisher said...

Thank you so much for getting us started on md5 with away3D! It's been incredibly useful (and still practically the only resource out there).

Concerning the gun: It is possible to add the gun's subgeometry to the geometry of the guy. That way you don't need to update the gun on every frame. It's tricky to get it working right with the exporter, as the md5 mesh needs to have had at one time in it's existence some kind of submesh that effects the skinning (or so I understand it from my boyfriend who is the modeler in our team). After that either you export the gun as md5 with a skeleton or you export it as whatever you want and then create the jointIndexData and jointWeightData yourself (if it should just follow the hand, it should be very easy, two lines of code, really). You can find me here http://www.janeaustengames.com/blog/ if you need more information on how to go about. Or if anyone has an even better solution, I'd love to learn it!

Unknown said...

Hi,

what do you make out of Error #1125? I kept having that error when I export my file. Is there a specific way my bones need to be setup? Can I have more than 3-5 bone influence on each vertex of the mesh?

Videometry.net said...

i seem to remember 4 bones is the limit

Anonymous said...

Hi,

I'm trying to export using your example files with the soccer player and biped. But everytime I export out of 3ds max with your script I get "Unknown system exception" and it highlights:

md5vertindices[i] = self.checkforvertex vertindices[i] tvertindices[i]

Any ideas?

Thanks!

Post a Comment