Help in implementing conditional soft-symmetry

[Sroy20]

Hi,

First of all, thanks a lot for this awesome python port. This is helping me a lot in my project which is age classification from facial images. I am using your code to frontalize the images before giving them as input to a Conv Net.

The dataset I am using has been collected by my teammates here in NTU, Singapore. Some of the images of this dataset have non-facial occlusions such as hands, etc. As per the original paper "Effective Face Frontalization in Unconstrained Images" (Sec. 3.3) these are taken care off by doing conditional soft-symmetry.

However, I find that this is not implemented in this Python port. As a matter of fact, this is also not implemented in the original MatLab codes.

Can you please let me how to implement this for my project? This will be very helpful.

[dougsouza]

Hi @Sroy20,

I'm glad to hear that this port is being useful to you. Well, unfortunately I don't know how to help in this case. Perhaps Dr. Hassner (@TalHassner) can give an answer to your question.

Cheers,

Doug

[TalHassner]

Dear @Sroy20,

Indeed this was not implemented in the MATLAB code we released. It was messy, last minute code and I always intended to write it better, but never got around to it.

The good news is that what we did was not necessarily optimal and you could use the same reasoning to develop your own (possibly better!) conditions for the soft symmetry, pretty easily.

All you need to do is evaluate the final, symmetric face to see if it "looks like a face". If not, revert to the non-symmetric version. Now, checking if something looks like a face can be done in lots of ways, many of them better than what we did.

One way to go is to rerun the face detector on the frontalized face. If it fails, then it's a bad frontalization. Another, more delicate approach is to run the landmark detector on the output (equivalent to what we did). If the landmarks are not exactly in their ideal positions (allowing for some wiggle room) then the face was not frontalized correctly and again you can revert back.

There are other things you can do here, but that's the gist of it.

I hope this helps,

Tal

[Sroy20]

Dear Dr. Hassner (@TalHassner),

Thanks for your reply and the ideas. I am going to implement them.

Thanks
Subhrajit Roy