Ground Truther & Clustering

Project Goal

Ground Truther: Build a Docker-based tool that uses face clustering to automatically generate reliable ground truth data for unseen video.

Clustering: Perform cluster analysis on extracted faces in the context of latent feature capacity.

Project Progress

  1. [DONE] Data Collection (9.1.2019)
  2. [2/3] Feature Extraction
    • [DONE] 2.1 facial_recognition library (9.3.2019)
    • [DONE] 2.2 RetinaFace+ArcFace (9.10.2019)
    • [YTBD] 2.3 OpenFace
  3. [curr/3] Clustering Performance
    • [curr] 3.1 facial_recognition library
    • [YTBD] 3.2 RetinaFace+ArcFace
    • [YTBD] 3.3 OpenFace

1. Data Collection

The data set includes 644 short videos (<15 seconds) of subjects walking towards a poster, reading the text (we see face profile from camera), then exiting the scene through the front.

Here’s a quick sample video featuring my advisor, Pat Flynn – what an excellent subject.

2. Feature Extraction

2.1 face_recognition library

This repo from ageitgey includes an easy-to-use, quick, off-the-shelf face detector and robust feature/encoding generator. It is built on top of dlib’s SOTA face recognition toolkit and achieves an accuracy of 99.38% on the Labeled Faces in the Wild data set.

Generating face_encodings with this face_recognition tool is as easy as calling a single command; this command inputs a list of bounding boxes (4-tuples) and outputs 128-D features for each face box in the supplied list. I used a GPU-enabled Dockerfile (below) for this and applied my script to the data set mentioned earlier. The results were saved as a dictionary of dictionaries via pickle files.

Example: dict[frame_counter] = {‘bboxes’: list of face_locations, ‘feature_vectors’: list of face_encodings}



# This is a sample Dockerfile you can modify to deploy your own app based on face_recognition on the GPU
# In order to run Docker in the GPU you will need to install Nvidia-Docker:

FROM nvidia/cuda:9.0-cudnn7-devel

# Install face recognition dependencies

RUN apt update -y; apt install -y \
git \
cmake \
libsm6 \
libxext6 \
libxrender-dev \
python3 \

RUN pip3 install scikit-build

# Install compilers

RUN apt install -y software-properties-common
RUN add-apt-repository ppa:ubuntu-toolchain-r/test
RUN apt update -y; apt install -y gcc-6 g++-6

RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-6 50
RUN update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-6 50

# Install dlib 

RUN git clone -b 'v19.16' --single-branch
RUN mkdir -p /dlib/build

RUN cmake -H/dlib -B/dlib/build -DDLIB_USE_CUDA=1 -DUSE_AVX_INSTRUCTIONS=1
RUN cmake --build /dlib/build

RUN cd /dlib; python3 /dlib/ install

# Install the face recognition package

RUN pip3 install --upgrade pip
RUN pip3 install face_recognition
RUN pip3 install opencv-python
RUN pip3 install opencv-contrib-python
RUN pip3 install argparse
RUN pip3 install pandas

COPY . /app

2.2 RetinaFace+ArcFace

Most of the following work was inspired by this francoisruty repo.



FROM mxnet/python:gpu

RUN apt-get update --fix-missing
RUN apt-get install -y build-essential cmake
RUN apt-get install -y make cython python-numpy vim python-setuptools python-opencv git

RUN pip install --upgrade pip
RUN pip install opencv-python
RUN pip install numpy
RUN pip install future

RUN pip install pandas 
RUN pip install scipy 
RUN pip install scikit-learn 
RUN pip install scikit-image
RUN pip install easydict

RUN git clone
RUN cd insightface_mod/deploy/ && make

WORKDIR /mxnet/insightface_mod/deploy/

docker build --rm --no-cache -t insightface .

NV_GPU=0,1 nvidia-docker run --rm -it \\ 
-v /home/ptinsley/Research/models/model-retinaface/:/model-retinaface/ \\
-v /home/ptinsley/Research/models/model-r100-ii/:/model-r100-ii/ \\
-v /home/ptinsley/Research/data/:/data/ \\
insightface bash

Most of this is straightforward except the RetinaFace mount. My local directory /home/ptinsley/Research/models/model-retinaface/ contains 2 files: model-0000.params and model-symbol.json. These have since been renamed from the original download where the word ‘model’ was originally ‘sshb’. I don’t recall why I changed it really. These were mounted to the /model-retinaface/ directory inside the Docker container. However, when calling the constructor for RetinaFace, one does not simply pass the directory containing the 2 files. Rather, one must also specify the prefix of these files, ‘model’ in my case, ‘sshb’ in the original case. Consequently, my (in the RetinaFace directory) was changed to…

detector = RetinaFace('/model-retinaface/model', 0, gpuid, 'net3')
detector = RetinaFace('/model-retinaface/model', 0, gpuid, 'net3')
detector = RetinaFace('/model-retinaface/sshb', 0, gpuid, 'net3')

Note: At this point, I have forked the original deepinsight/insightface/ repository, renamed it on my GitHub to pgtinsley/insightface_mod/, cloned it down locally, and copied the contents of insightface_mod/RetinaFace to insightface_mod/deploy. For this reason, I have and, which came from the respective repositories. This process has turned out to be a bit tedious as every change I make has to be added, committed, and pushed… and then the rebuild with the –no-cache option has to restart with the bash call. Small changes can be dealt with rather easily for testing (just use vim in the Docker container to fix small things), but they ultimately have to be reflected in the git repo itself.

It’s 3:22 in the AM, and I think I’ve got a container that will ultimately work. I’m going to run it on a GoT clip to see how it goes. There are a lot of faces throughout the scene, the lighting is dark, and there are about 5600 frames, which is the exact opposite of the data we have collected so far… perfect. I imagine it will have to run overnight. Side note: this re-building process takes 5ever. Docker needs to come up with a solution to having only one line in a Dockerfile that we want to make sure does not get cached. Against the idea of an ever-consistent container, but, boy oh boy, would that make this a whole lot faster. Solution to explore later: just mount the git repo instead (duh).

Here’s an example of the RetinaFace+ArcFace recognizer in action when applied to the GoT clip that I mentioned.

Now, that I know it works, there’s not as much a reason to keep with the videos/gifs. Rather, I’ll just save the features and call it a day. Before moving on to OpenFace, I’m going to look at the features produced from the face_recognition library (section 3).

2.3 OpenFace


3. Clustering

3.2 face_recognition

3.2 RetinaFace+ArcFace

3.2 OpenFace

Patrick Tinsley (PT)

Patrick Tinsley (PT)

My name is PT, and I am a graduate student at the University of Notre Dame. I am part of the Computer Vision Research Lab and work on surveillance video analytics. I also <3 sports, music, and Netflix/HBO shows.