Overview of the initially provided prototype

An overview of the prototype made by Sjaak Verwaaijen.

The current prototype, hereafter referred to as VibeTest was created to get an overview of what OpenFace has to offer and what can be done with the data it provides. In short, the current feature set is as follows:

• Generated avatars (or other images) can be added and analysed using OpenFace

• Their AUs (Action Units) are registered and displayed in a table.

• Landmarks (LMs) are used to trace the face.

• Emotion data can be analysed using two methods: logically or based on machine learning. The logical method looks at specific AUs and determines the displayed emotion based on where they are located. For example: if AUs representing the corners of one's mouth are in an upward position compared to the centre of the mouth, this means that the person in question is likely smiling. Machine learning, on the other hand, takes a set of AU values with their correct emotion, and then uses this data to determine the emotion based on AUs.

• The difference between two faces can be measured not only in AU values, but also represented through the use of vectors.

Technical overview

VibeTest is a windows form application written in C# using .NET Framework 4.5.2. OpenFace is used for facial analysis, while the added C# code is there to create the GUI and perform some post-analysis calculations (such as calculating vectors). This approach is similar to what I had envisioned, though it would be preferable for the application to be headless as it will be used as a cloud service as opposed to something a user would interact with directly.

The functionality of the software encompasses OpenFace with added difference/vector calculation and emotion determination.

Difference/vector calculation

These functionalities have been grouped together as their functionality has the same base. The core is that the difference in AU values between two images can be calculated. This is done by comparing the two tables and determining the difference for each specific AU value. Vectors take this a step further by giving the user a visual indication of what has changed in the form of vectors.

Emotion Determination

VibeTest adds two ways to determine the emotion on display in a given image: logical determination and machine learning.

Logical determination comes down to a decision tree based on AU values. By following the tree, it is possible to make a guess as to the emotion on display. One of the more simple ways to do this is using FACS (Facial Action Coding System), as was implemented in this prototype --VERIFY THIS--. The benefit here is that it is very testable and legible. The downsides, however, include lower accuracy. Further research will be necessary to determine whether the determination of emotion lends itself toward this type of analysis. It is possible that the matter at hand is not suited to a decision tree, or that the tree would become so large as to be cumbersome and too advanced to develop for one not deeply familiar with the subject matter.

Emotion determination with machine learning, on the other hand, infers the emotion based on previously analysed images. By training a machine learning algorithm using a set of manually verified images it is possible to make an estimation as to what the emotional state on display is. Though it is theoretically more accurate, it is also more troublesome to implement (being technically more challenging and requiring extensive data sets). Another issue is that the actual decisions made by the algorithm are very opaque.

Conclusion

While most the features present are necessary in the end product, some additions will be necessary: exporting data, more accurate emotion recognition and general restructuring of the application. Modules such as the GUI will simply not be necessary in the end product. More on this in the Deployment to the Cloud report.