MocapNET BVH compatible 3D Pose Estimation from RGB

Discussions about MHX2 and other plugins that are developed outside the scope of MakeHuman. Note that bug reports should go to the respective code projects and might go unseen here.

Moderator: joepal

MocapNET BVH compatible 3D Pose Estimation from RGB

Postby AmmarkoV » Fri Nov 15, 2019 7:45 pm

Hello world,

This is my first post here! First of all thank you for the excellent tools you have provided. I am a PhD student from Greece and since I have been using makehuman as my primary human model generator I would like to contribute something back to the community in the form of my 3D Pose estimation software that can derive BVH poses from RGB images and videos.

The code is written in C++ and uses the Tensorflow C-API as well as OpenCV for the visualization.

The idea is to be able to convert a single person video to a BVH file that can then be used to animate a makehuman model. Ideally you could sit in front of the camera and bring a makehuman model to life by performing poses yourself.

You can both render the animation using the MakeHuman/MakeWalk plugins of blender ( as seen in the Qualitative results on the following video )
Image

However I have also baked a model as a visualization mode inside the demo application ( the project needs to be built with the CMake ENABLE_OPENGL flag and then it is seen if you start the demo application with --opengl )

Image


The repository is the following
Image
https://github.com/FORTH-ModelBasedTracker/MocapNET

Looking forward to your comments, and I hope someone will find it useful.!
Feel free to ask any questions you may have. I am using the github version of makehuman 1.2.0 and the CMU Default Hybrid rig, but scaled to millimeters instead of centimeters.
In order to animate the model with OpenGL I export the model in .dae format which I then load using Assimp convert to my own vertex container file format called .tri


The output quality is not perfect, and hands and face expressions are currently not tracked, however I am working full-time to improve it..!

Ammar
User avatar
AmmarkoV
 
Posts: 5
Joined: Fri Nov 15, 2019 7:20 pm
Location: Greece

Re: MocapNET BVH compatible 3D Pose Estimation from RGB

Postby RobBaer » Tue Nov 19, 2019 8:25 pm

Haven't had a chance to try it, but it looks really interesting. Tensorflow is everywhere these days :D ...
User avatar
RobBaer
 
Posts: 1208
Joined: Sat Jul 13, 2013 3:30 pm
Location: Kirksville, MO USA

Re: MocapNET BVH compatible 3D Pose Estimation from RGB

Postby AmmarkoV » Wed Nov 20, 2019 1:04 pm

RobBaer wrote:Haven't had a chance to try it, but it looks really interesting. Tensorflow is everywhere these days :D ...

:D

Turns out the youtube video in the previous post also does not show the OpenGL visualization, so this is a screen recording of it and me doing stuff in front of the camera
User avatar
AmmarkoV
 
Posts: 5
Joined: Fri Nov 15, 2019 7:20 pm
Location: Greece

Re: MocapNET BVH compatible 3D Pose Estimation from RGB

Postby Shaba1 » Fri Jan 03, 2020 5:30 pm

If this works "out of the box" this would be great. I have the whole carnage-mellon bvh library and many time there is just not an animation that does exactly what I want. So I end up cutting and pasting thing which leads to fitting problems and frame rate problems ands other things that eventually lead to me just abandoning the whole project. If this program works I could do that posing myself and just import the bvh to my makehuman character. I will leave comment after I test it out.
Shaba1
 
Posts: 164
Joined: Wed Mar 05, 2008 3:21 pm

Re: MocapNET BVH compatible 3D Pose Estimation from RGB

Postby AmmarkoV » Tue Oct 20, 2020 8:25 am

Image

A new and improved version of MocapNET is out..! This time it is even more accurate..!

For example with it you can convert it from this youtube to this BVH and then directly load it to your Blender.
The produced BVH is compaitble with the CMU+Face armature I have commited here

Image => Image => Image

Image
User avatar
AmmarkoV
 
Posts: 5
Joined: Fri Nov 15, 2019 7:20 pm
Location: Greece


Return to MHX2 and other plugins developed outside MakeHuman

Who is online

Users browsing this forum: No registered users and 1 guest