Comparison of the available facial analysis frameworks
Comparing what is on the market and what is best suited to the project.
Introduction to the libraries
As was previously mentioned, the prototype currently makes use of OpenFace. This is an open source library offering various computer vision services such as facial landmark detection and facial action unit recognition. Though the application is free to use and implement, the license does not allow commercial use without an annual royalty fee. The open-source nature of the project does, however, mean that it should be relatively simple to convert into a microservice (or at least take parts of it to use for such purposes). Previous groups at Fontys have already attempted to do so, with great success.
FaceReader is the only commercial option included in this comparison. This brings various benefits with it, such as extended support and increased ease-of-use. The obvious drawback is the cost associated with a commercial product. Alongside this, the SDK is not automatically included with a license. Not having the SDK would make it harder to turn into a microservice.
Automated Facial Affect Recognition (AFAR) Toolbox is, similarly to OpenFace, an open-source library. It is aimed at analysing faces to provide face tracking, face registration, action unit (AU) detection and visualization services \cite{AFAR}. One of the biggest differences from OpenFace is that it is aimed at analysing videos. As can be seen in the figure below, it also differs in the pipeline used to analyse videos and images. Most of the code is written using MatLab. Though MatLab is also used in OpenFace, it is mostly interfaced with using .NET. This is a drawback when it comes to AFAR Toolbox due to my being unfamiliar with MatLab.
Comparison of library efficacy and functionality
Based on "Assessing Automated Facial Action Unit Detection Systems for Analyzing Cross-Domain Facial Expression Databases" it seems the two open source libraries (OpenFace and AFAR Toolbox) provide more accurate results on average. The results were summarised and scored for an easy overview.
When it comes to ease of installation, OpenFace is the most user friendly of the open-source options. It required Visual Studio, some C++ libraries and a PowerShell script or two. From there it was smooth sailing: simply open in Visual Studio and run the GUI. It can also be run from the terminal with a bit of extra work. FaceReader shines in user friendliness as features such as these are generally where commercial interests shine. AFARToolbox is, based on what has been tested so far, the least user-friendly to set up. It requires various libraries, MATLAB installation (which is not free, though in my case a student license sufficed) and MATLAB modules. Furthermore, its interface left a lot to be desired. It is also not clear whether AFAR can be accessed from the terminal.
How each library works differs somewhat. In their technical functionality, OpenFace and AFARToolbox are the most similar in scope and focus, while FaceReader offers a more vast suite of services. Though the results of FaceReader are less accurate, its goals are different from the other options. It aims to offer a comprehensive overview of emotions, AU and LM values etc. in an admin-dashboard style data overview. It is far more plug-and-play than the open-source options.
Conclusion
Due to the easy set-up of a headless OpenFace instance, this is what I focused on for my own prototypes. The choice of library is a means to an end, and OpenFace allowed me to remotely generate .csv reports with very little set-up. Furthermore, it is open source and has sufficient documentation. The prior experience in implementing it in the prototype will hopefully also prevent mistakes that could otherwise have been made.
Last updated
Was this helpful?