There and Back Again: Converting MetaHumans to UE Mannequin using a Blender Pipeline
In this post I’ll walk through a process for exporting Unreal Engine 4 MetaHumans with facial expressions to Blender and then back to UE4, so they can be used efficiently with UE Mannequin animations. Motivation for this was to have them work as drop in solution for human characters for a mid-poly multiplayer game.
First I’ll share a bit of background, but if you’re here just for the technical bits, please skip the next part.
History & Motivation
So, hi! My name is Teemu Polvi and I am currently 50% of the Snow War development team. I am the lead designer / artist / graphics person, with my friend Jan helping out with a lot of code & server setup.
I’ve been working on my game Snow War for seven years now. It’s a “remake” of Snow-War: a Half-Life 1 mod I developed with a few other amazing people back in the early 2000s. Snow War, as it exists today, started in 2019 as a hobby-level project, then — fueled by some freelance software development money — grew into a professional project. I hired a graphics artist and (then) Junior Programmer. But unfortunately we did not get funding, and when freelance work dried up, I got back to doing real work. Since 2023 it’s been developed in full hobby mode.
When the shift back to doing 99% of the art assets happened, I knew I had to do things differently. It had taken us over a year to get four playable characters. With spare time as my only resource, that wasn’t viable.
When MetaHumans were first announced, we were already working on Snow War.
Our graphics standard was to flirt with that modern realism, but steer closer to games you played maybe 10–15 years ago. Not an easy path to walk on, but essentially this meant being super polycount and texture size conscious, keeping shader counts to minimal and so on. This was one of the main reasons we stuck with UE4 when everyone else seemed to upgrade to UE4 (with mixed results, unfortunately).
For this the MetaHumans provided seemed to fit, but after just a few initial tests, it became obvious we couldn’t use them — as is. The rigs and materials were just too complex to run efficiently on the hardware targets we were aiming for.
Fast forward couple years later, and now when I was down to just one graphics person (me) and one game-coder (also me), there was no way I could be crafting enough custom characters anyway.
Based on the testing we had done previously with real players, it became obvious we’d need a lot of characters so that we wouldn’t just have clones running around. So I started on the somewhat insane task of providing some sort of character customization.
While I believed customizing facial features was off the table, I wondered if I could get like 10–20 proper human characters to choose from — maybe that was enough? Maybe I should take a second look at the MetaHuman thing?
Some helpful tools
The things I used the most with Blender were two key plugins developed by Epic Games (by the awesome James Baber): Send to UE and UE to Rigify (https://epicgamesext.github.io/BlenderTools/). If you are using Blender for making content for UE and not using these — you are missing out. With Send to UE, I only have to press CTRL + U (my selected hotkey) and all of the meshes and animations I have selected will be sent to UE4, and I can view them instantly. This is the workflow Unity 3D users will recognize well. UE to Rigify allows one to use Rigify to animate characters.
First steps
So the first thing is of course to make your MetaHuman(s). I made a separate Unreal project for this. The source MetaHumans take about 17GB in space, when in the actual game project for the 16 characters, they take up only 200MB. This is substancial file size save for other people who use the main game version control. Also this allows me to quickly iterate and experiment without polluting the main editor.
So we fire up the MetaHuman editor with one the presets, fiddle around, and import our creation to the Unreal. And it looks kinda nice!
As a tech artist, we immediately start to ponder how these are made and if we could optimize them. As they are now, even with lower LOD:s, they just have too much stuff going on.
Let’s browse to the _FaceMesh asset, right click it and Export to FBX so we can open this up in Blender.
Then hit up import FBX in Blender — and you are done! End of part1… errr.. wait. There is really a lot going on here.
So not only are there bazillion bones, but 7 different LODs, and 14 materials (which are not always the same, by the way). This was where I immediately gave up before. But necessity is a great motivator.
The manual way
My first test was to clean this up, remove the armature, align the mesh with the default UE Mannequin rig I was using. Then maybe 45–60 minutes later I was done. If I was going to do 20 of these, it would be a lot more. And once I tried different models, I kept making mistakes. Also I still needed to dig for the textures.
I realized I would need to make this automatic, or at least semi-automatic, so I could make more characters, and also later if I make changes to the MetaHumans, run the same pipeline again.
My goals were:
- Export the Facemeshes in a batch
- Export & Import the textures
- Cleanup the unnecessary LOD:s
- Cleanup the unnecessary materials and details
- Remove the Armature, reposition and weight paint to UE Mannequin
Mesh & texture export
My first piece of automation was to enable batch exporting the faces with textures and place them in nicely organized directory structure. One thing you learn quickly in game dev and especially with Unreal Engine — you need to maintain proper directory structures or you will be in a world of pain.
This I mostly acchieved with an Editor Utility Blueprint. Instead of using Asset Actions → Export, I could select multiple _FaceMesh assets, right click and select Scripted Actions → Export Facemesh.
This exported all of the meshes. In UE4 (AFAIK) exporting multiple things always creates a directory structure matching the content locations. I wanted them in a specific hierarchy, where textures and other data can be organized in neat folders. AFAIK there was no way to get that with Blueprints alone. So I added a C++ helper class MyBlueprintFunctionLibrary, which contained this nifty code for running command-line commands via Blueprints. BTW, this is quite risky to do, so one more reason not to do this in your main game .uproject!
void UMyBlueprintFunctionLibrary::RunCmd(FString cmdDir, FString workingDir, FString cmd, FString cmdParams)
{
FString exePath = cmdDir.Append(TEXT("\\")).Append(cmd);
uint32 processID = 0;
FProcHandle ProcHandle = FPlatformProcess::CreateProc(
*exePath,
*cmdParams,
false,
false,
false,
&processID,
2,
*workingDir,
nullptr
);
FPlatformProcess::WaitForProc(ProcHandle);
UE_LOG(LogTemp, Log, TEXT("RunCmd done"));
}The main “Export cleanup” BP looks like this. We export all of the meshes, then RunCmd to move the files to the folder structure we actually want.
So now I have a neat folder structure:
Source\A1\SK_A1.fbx
Source\B1\SK_B1.fbx
Source\B2\SK_B2.fbx…and so on. Next I wanted to get the relevant textures exported.
For this you look up the textures in the Material Instance of the Skeletal Mesh.
I only wanted the Albedo and Normal maps, so I search for the texture referenced as Normal_MAIN and Color_Main from the head_LOD1_shader_shader
Then we load the Texture Assets and export them. Finally we do the same RunCMD acrobatics to rename the assets in a subfolder under each character. As an extra step, I run ImageMagick on command line to scale the textures to an uniform size.
Importing
So now we’re back in Blender, with a fresh FBX import and a bunch of problems: too many LODs, too many materials, and a rig that’s great for cinematics but not what I want to ship in a multiplayer game.
I ended up making a Blender add-on panel. This panel has a button for batch importing all specified FBXs from the correct folders. After this, I no longer needed to do that File → Import routine. I like to keep things so that I can run the same command for multiple objects. So I can export all of my MetaHumans at once, then run the cleanup for all of them and finally export to Unreal on one step.
Keep one LOD, delete the rest
First step was to only keep one of the LODs. For Snow War, I decided to use LOD3, which is was about 17k polygons.
In startMetahumanCleanup() the script walks the imported hierarchy, finds the mesh ending with that suffix, renames it to the character name, and deletes everything else. The order of deletion is important here. You want to delete the child objects first.
LOD_TO_KEEP = "LOD3"
foundLodGroup = False
rootObj = None
bpy.ops.object.select_all(action='DESELECT')
for child in curobject.children:
if child.name.startswith("root"):
rootObj = child
for rchild in child.children:
if (rchild.name.endswith(LOD_TO_KEEP)):
rchild.name = charName
foundFace = rchild
else:
toDelete.append(rchild)
else :
if child.name.endswith("LodGroup"):
foundLodGroup = True
toDelete.append(child)
toDelete.append(curobject) # Add root
if not foundLodGroup:
showErrorMessage("Could not find LOD group")
return
for obj in toDelete:
obj.select_set(True)
print("Delete : " + obj.name)
bpy.ops.object.delete()Build a material set
MetaHumans come with a pile of materials, and they are not always consistent across characters. For our purpose I wanted this to be a super simple set, with only the skin material changing between characters.
So I collapse them into a small common set (FACE, eyes, teeth, eyelashes), and delete the rest.
MATERIALS_TO_COMMON = {
"M_EyeRefractive_Inst_L": "MI_Eyes",
"M_lacrimal_fluid_Inst": "MI_Eyes",
"M_TeethCharacterCreator_Inst": "MI_Teeth",
"M_EyeOcclusion_Inst": "MI_Eyelashes",
"M_HeadSynthesized_Baked_LOD3": "FACE",
"Body": "FACE",
}There are also dictionaries for MATERIALS_TO_DELETE and MATERIALS_TO_DELETE_VERTICES. These are materials that are not really being used in our current LOD and materials for which we don’t even want to keep the vertices: M_Cartilage and M_lacrimal_fluid_Inst… I don’t even know what they did, but sounded too disgusting so decided to delete them entirely ;-)
The python script loads the FACE texture from the characters own directory, while common textures (teeh, eyes, eyelashes) are shared between the characters. This means I don’t end up with MI_Eyelashes.008 in my Blender file, or in UE.
Align the head
Since Metahuman and UE Mannequin do not share the same exact size we need to align them. I had a naming convention for the genders, ‘A’ being female and ‘B’ being male, so I could already make adjustments based on that. I also added support for both Tall and Medium variants, as I kept accidentally saving with incorrect height. Better to fix it in the pipeline than write a post-it saying “Remember to save as tall” — as I never did.
HEAD_TRANSFORM_VECTOR_MALE_TALL = mathutils.Vector((0, 3.2, 2.3))
HEAD_TRANSFORM_LIMIT_MALE_TALL = 177.0
HEAD_TRANSFORM_VECTOR_MALE_MED = mathutils.Vector((0, 3.2, 9.24))
HEAD_TRANSFORM_VECTOR_FEMALE_TALL = mathutils.Vector((0, 2.32, 14.06))
HEAD_TRANSFORM_LIMIT_FEMALE_TALL = 163.0
HEAD_TRANSFORM_VECTOR_FEMALE_MED = mathutils.Vector((0, 2.32, 22.14))These offsets are not scientific; I eyeballed the eyeballs into roughly the same position. Perhaps more robust way would be to detect the dimensions more closely and then adjust solely based on that.
Auto-weights
Since we got rid of all the unnecessary bones, we also want to strip the vertex groups and rebuild weights based on the UE Mannequin. There are some Blender plugins to help with this, but decided to built my own script that uses a Reference Mesh. findReferenceWeights() returns a Barycentric weights based on the nearest polygon from the reference mesh. These are then blended to the MetaHuman mesh.
def assignVertexGroupsBasedOnRefMesh(curobject: bpy.types.Object, refObj: bpy.types.Object):
print("assignVertexGroupsBasedOnRefMesh: " + curobject.name + " ref: " + refObj.name)
refObj_bvh = BVHTree.FromObject(refObj, bpy.context.evaluated_depsgraph_get())
for vertex in curobject.data.vertices:
weights = findReferenceWeights(refObj, refObj_bvh, vertex)
#print("weights " + ", ".join(f"{group_name}: {weight:.3f}" for group_name, weight in weights.items()))
for group_name, weight in weights.items():
if weight > 0.001:
vertex_group = curobject.vertex_groups.get(group_name)
if vertex_group is None:
vertex_group = curobject.vertex_groups.new(name=group_name)
vertex_group.add([vertex.index], weight, 'REPLACE')One button then ready for “Send to Unreal”
There we have it, a plain 17k LOD0 for Snow War, with four simple texture sets. After this I can hit Pipeline -> Send to Unreal and it will pop-up, rigged and ready to go.
The Uncanny Valley
While the pipeline was set and I had my 16 characters set up in the game, there was that very strong sense of Uncanny Valley. The eyes would stare in to the far horizon, and there were no facial expressions.
I did the ol’ trick of moving the UV-offset of the eye shader to have the characters look around. And yeah, it made it worse.
Get Teemu Polvi’s stories in your inbox
Join Medium for free to get updates from this writer.
Most of us have watched other humans our whole lives, so we get drawn to these tiny details. Like how this character looks at the camera. Subtle, but noticeable (to most, not all) the eyelids move quite a bit when looking around. As it happens, MetaHuman does a pretty good work on this.
Even when in our case we are making a fast moving multiplayer game — having those micro-facial expressions is key on selling they are life like. Especially if players are spending time tweaking them for their own style and liking.
One way of achieving these both, without a complex facial rig, is to use Morph Targets, which are called Shapekeys in Blender.
So I got this idea: Since MetaHuman face seems to be mostly driven by bones, we could export the bone positions, and before destroying the original MetaHuman Armature in our cleanup script, bake them as Shapekeys.
This was again a combination of Blueprints, C++ and Python.
A face library Level Sequence
The first thing I did was to create a Level Sequence that’s basically a catalogue of expressions. Using the MetaHuman control rig, I did a bunch of expressions I thought were useful.
I’m not trying to export a performance capture clip or real animation here. I just want a stable set of expressions that I can later mix & match to make the face look more alive.
Export sockets
I was looking for a way to query the exact position of the facial bones in runtime, but I could not get it to work. However, I noticed that I could just query for Sockets. As it happens, at least in MetaHuman skeleton, there are sockets for all of the bones we care about.
We can sample all of the socket transforms at chosen frames, and write those to JSON, which we can then (hopefully) read back in Blender.
The exporter actor
I implemented the exporter as an AActor: ALevelSequenceExporter. I was hoping to have this a full editor script, but the socket info was not getting though without paying in play mode.
The actor looks up the face skeletal mesh component by name, then it seeks the sequencer playhead to each “expression frame”, waits a bit (as I noticed there was delay for the sockets to update), and samples all sockets in world space.
void ALevelSequenceExporter::SaveSocketsForFrame(USkeletalMeshComponent* SkeletalMeshComponent, FName name)
{
TArray<FName> SocketNames = SkeletalMeshComponent->GetAllSocketNames();
UE_LOG(LogTemp, Warning, TEXT("Exporting %d sockets to %s"), SocketNames.Num(), *name.ToString());
TMap<FName, FTransform> SocketTransformMap;
for (int32 i = 0; i < SocketNames.Num(); i++)
{
FName SocketName = SocketNames[i];
FTransform SocketTransform = SkeletalMeshComponent->GetSocketTransform(SocketName, RTS_World);
// Check if the socket transform is the same as the default
FTransform DefaultSocketTransform = DefaultFrameInfo[SocketName];
if (SocketTransform.Equals(DefaultSocketTransform))
{
continue;
}
// Store the socket name and transform in the map
SocketTransformMap.Add(SocketName, SocketTransform);
}
ExportedFrameInfo.Add(name, SocketTransformMap);
}We then form this as JSON.
void WriteMapToJSON(TSharedRef<TJsonWriter<>> Writer, TMap<FName, FTransform> map)
{
for (auto& SocketElem : map)
{
Writer->WriteObjectStart(SocketElem.Key.ToString());
FVector Location = SocketElem.Value.GetLocation();
Writer->WriteValue(TEXT("LocX"), FString::Printf(TEXT("%f"), Location.X));
//.. Y, Z
FQuat worldRotation = SocketElem.Value.GetRotation();
Writer->WriteValue(TEXT("RotW"), FString::Printf(TEXT("%f"), worldRotation.W));
//.. X, Y, Z
// Only save scale if not default
FVector Scale = SocketElem.Value.GetScale3D();
if (Scale.X != 1.0f || Scale.Y != 1.0f || Scale.Z != 1.0f)
{
Writer->WriteValue(TEXT("ScaX"), FString::Printf(TEXT("%f"), Scale.X));
//.. Y, Z
}
Writer->WriteObjectEnd();
}
}(Just noticed the values are all stored in strings. Nevertheless, it works.)
We write this out with FFileHelper::SaveStringToFile(). If running in Perforce version control, it is good practice to check if file is Read only and throw a helpful warning so the developer (me) remembers to checkout the files. I like to keep the expression-json’s in the version control even when they are derivative. That way if things go wrong, I at least have some data stored.
// Save the JSON string to a file
if (IFileManager::Get().FileExists(*ExportPathFull) && IFileManager::Get().IsReadOnly(*ExportPathFull))
{
UE_LOG(LogTemp, Error, TEXT("Cannot write to %s: file is read-only."), *ExportPathFull);
return;
}
if (!FFileHelper::SaveStringToFile(JSONString, *ExportPathFull))
{
UE_LOG(LogTemp, Error, TEXT("Failed to save JSON to %s."), *ExportPathFull);
return;
}This results in a JSON:
{
"LookLeft": {
"FACIAL_R_Eye": { "RotW": "0.99", "RotX": "0.01", "RotY": "0.08", "RotZ": "-0.01" },
"FACIAL_L_Eye": { "RotW": "0.99", "RotX": "0.01", "RotY": "0.08", "RotZ": "-0.01" }
},
"Mad": {
"FACIAL_C_Brow": { "LocX": "0.12", "LocY": "-0.04", "LocZ": "0.01" }
}
}So that is is it. This surely was the hard part… right?
Reading the expressions in Blender
Now comes the part that most likely took me the most time. A lot of this was experimenting and looking at results that looked like Garrys Mod gone wrong.
The main trick here is to temporarily recreate each expression as an armature pose, and then bake the resulting deformation into the mesh.
Load the JSON and loop expressions
The main loop is straightforward: for each expression, read the JSON, create an Action, pose the bones, bake a shape key.
with open(json_path, "r") as f:
data = json.load(f)
if not mesh.data.shape_keys:
mesh.shape_key_add(name="Basis") #Important to add the Basis key
for frame_name, frame_data in data.items():
action_name = f"{mesh.name}_{frame_name}"
action = importAnimation(armature, action_name, frame_data, frame_name)
armature.animation_data.action = action
createShapeKeyFromCurrentAction(mesh, armature, frame_name)Inside importAnimatin() we setup a new action.
bpy.ops.object.select_all(action='DESELECT')
armature.select_set(True)
bpy.context.view_layer.objects.active = armature
# Change the selected armature to POSE mode
bpy.ops.object.mode_set(mode='POSE')
# Delete action if already exists
if action_name in bpy.data.actions:
bpy.data.actions.remove(bpy.data.actions[action_name], do_unlink=True)
action = bpy.data.actions.new(name=action_name)
print("Create action: " + action_name)
bpy.context.object.animation_data_create()
bpy.context.object.animation_data.action = action
# Select all bones and clear their transform + set keyframe
bpy.ops.pose.select_all(action='SELECT')
bpy.ops.pose.transforms_clear()Then we iterate through the Armature data from the JSON and position our bones.
process_bone(armature, "pelvis", frame_data)And then it just works!
Ok.. it took me a couple of tries.
The coordinate conversion (the part that hurt)
UE4 and Blender do not agree on axes, and if you get this wrong you’ll know immediately: eyes rotate the wrong way, jaw opens sideways etc.
In the JSON, the bones were defined in a world-space transform. The importer builds a matrix from the JSON, then converts it into pose space.
For someone else this might have been super basic stuff, but for me this was a lot of trial and error.
First we build the Translation.
bone_data = frame_data.get(bone_name)
base_rot = curobject.convert_space(pose_bone=bone, matrix=bone.matrix, to_space='WORLD', from_space='POSE').to_quaternion()
w = float(bone_data.get("RotW", 1))
x = float(bone_data.get("RotX", 0))
y = float(bone_data.get("RotY", 0))
z = float(bone_data.get("RotZ", 0))
ue_import_rot = mathutils.Quaternion((w, -x, y, -z)).normalized()
t = (
float(bone_data.get("LocX", 0)),
-float(bone_data.get("LocY", 0)),
float(bone_data.get("LocZ", 0)),
)Then we apply it.
mat = bone.id_data.convert_space(pose_bone=bone, matrix=bone.matrix, to_space='WORLD', from_space='POSE')
loc, r, s = mat.decompose()
M = (
Matrix.Translation(t) @
ue_import_rot.to_matrix().to_4x4() @
Matrix.Diagonal(s.to_4d())
)
bone_matrix = bone.id_data.convert_space(
pose_bone=bone,
matrix=M,
to_space='POSE',
)
bone.matrix = bone_matrix
bpy.context.view_layer.update()We then do this iteratively for the entire Armature.
bpy.context.view_layer.update()
bone.keyframe_insert(data_path="location", frame=1)
bone.keyframe_insert(data_path="rotation_quaternion", frame=1)
for child_bone in bone.children:
process_bone(curobject, child_bone.name, frame_data)After this we use the operator which applies a modifier as a shape key. As we know, rmature/Skeleton is an modifier!
bpy.ops.object.modifier_apply_as_shapekey(
keep_modifier=True,
modifier=modifier.name,
report=True,
)We run the script and we are ready to preview the different expressions we have and try combinations. Some of them don’t look 100%, as MetaHuman does other stuff than just alters the Armature. But I am still feeling like this bloke in the pic.
Send to Unreal and…
Now with Shapekeys done, I can hit the Send to Unreal and see how everything works!
I can’t describe the excitement I felt once this started working. This was a hobby project, and in calendar terms I had worked on this in the period of 8 months. At points I was just working towards an idea for weeks on end, without real knowledge if any of this doable.
Finishing touches
Finally it was up to me to make more minimal Materials for the character. I won’t go on saying I succeeded making AAA-quality work here, but I was happy with I got with the resourced I had (me).
One of the things you might have noticed is missing from the past screenshots: eyebrows & hair. For eyebrows I drew up an atlas-texture and tweaked the face Material to use them (TIP: eyebrows are usually mirrored). This enabled the possibility to tweak their style, color and placement later.
For hair I purchased a bunch of haird-card-based assets. Then I spent a lot of time tweaking them to fit the overall headsizes and utilize the re-coloring Master Material I had made.
Overall I like the way the characters ended up looking. Just having them able to blink and do tiny micro expressions makes them look, if not alive, then at least more like what one would expect from a video game character.
Conclusion & extra info
So there you have it! As it’s sole creator and the machinist who runs it, I get by, even while it is still a very particular process. But every step was also done so that it prepared for (maybe) eventually running this as complete automated pipeline. Remember that RunCMD from the very beginning? Ideally I could easily run the whole Blender side through it, without the need to even open Blender. This is actually what Unity 3D does (at least used to) every time you modify a .blend file under it’s structure.
But for me, who uses Blender almost daily, I don’t mind running the process in Blender so I can also examine the results midway through.
If not an as a complete inspiration for making exactly this, I hope this serves at least as a thought provoker to some aspiring indie developer. You can get stuff done more by just “tiny” bit of automation.
Other pipeline tools I have done during the development on Snow War:
- Texture channel remapper: Re-configures AO, Roughness & Metallic — remaps Texture files to the format I use in Snow War so everything uses the same couple master shaders, no matter which asset pack they are from). This is currently run on command line, but my next step for it would be to make it run directly from UE4.
- Blender Animation cleaner: Removes unwanted tweak bone transforms when importing motion capture etc stuff for the UE Mannequin Rig)
- Material Instance creator: Based on a UE Marketplace asset, I reworked it heavily to create MI from a patch of textures, again use full for when importing assets.
Thank you
Thanks for reading it through — or at least scrolling to the very bottom.
If this gains interest I will definitely write more stuff like this in the future about other pipeline/game-dev topics. Might be the next one is something of completely different nature… let’s see.
But before I get too carried away…
I’m going to go back and finish our Early Access!
