PT

StyleGAN Tour Pat Tinsley


# 2.3.2020

# 1.27.2020

Daily Update

Here are some post-meeting ideas, and current goings-on through the mind:

# 1.22.2020

Daily Update

df_mean

df_binary

Idea

# 1.21.2020

Meeting Notes

Daily Update


Goal

The goal of this project…

is to… is to…… is to……… Huh, that’s funny! -Peter Pan


Re-creatable Pipeline


# Create the environment
conda env create -f sg.yml
# Activate the environment
conda activate sg 

Note: In our case, the filename of the raw image is tied to the subject’s identity. For example, ‘04572d16.JPG’ is the raw image associated with subject id 04572. For the sake of clarity, we will use subject 04572 to demonstrate how the nomenclature continues between experiments.


# Crop raw images to aligned images with dimensions (1024, 1024).
python align_images.py raw_images/ aligned_images/
# Encode aligned images as latent vectors and save corresponding generated image.
python encode_images.py aligned_images/ generated_images/ latent_representations/

The bulk of the work and time required for the ensuing experiments stems from the StyleGAN_tour_2.py script. However, before we dive into the implementation, let’s first explore what we really want to get out of the experiments.

We want to study how perturbing a vector along a certain axis in the StyleGAN latent space affects the ability of an off-the-shelf face recognition system to match a supplied identity.

These results will ultimately be extended into the context of data augmentation, but that will come later. Just in reading this objective above, we already know there are variables/dimensions to be explained.

  1. Firstly, per the original StyleGAN paper, StyleGAN latent vectors have dimensions (18, 512); a single image in the StyleGAN universe can be represented via 18 axes, each of which are 1-D arrays of 512 floating point values. The first four axes (0-3, inclusive) affect coarse features, such as pose, general hair style, face shape, etc. The next four (4-7) affect medium features, such as hair style, eyes open/closed, etc. The remaining ten (8-17) affect color scheme (eye, hair and skin) and other micro-features. Just based on this prior information, we can guess that perturbing coarser features (along lower numbered axes) will change the face structure more drastically (meaning decreased performance for the face recognizer). Contrarily, perturbing finer features (along higher numbered axes) will likely change micro-features that do not detract from the recognizer’s matching performance. So far, this is merely conjecture; we will see the results on the other side.

  2. Secondly, we need to explain how we propose to ‘perturb’ an axis: a 1-dimensional array of 512 floats. We decided to ‘perturb’ an axis by adding a scaled standard-normally distributed random sample (of length 512) to the original latent vector along the specified axis. Below is a breakdown for clarity/simplicity:

    1. Getting a sample from the univariate normal/Gaussian distribution of mean 0, variance 1, and length 512
       rnd.randn(Gs.input_shape[1]) # Gs.input_shape[1] == 512
      
    2. Scaling this sample element-wise by a shared magnitude
       magnitude * rnd.randn(Gs.input_shape[1])
      
    3. Adding the scaled sample back along the original (specified) axis
       original_vector[axis] + magnitude * rnd.randn(Gs.input_shape[1])
      
  3. We needed to determine the scale for the magnitude variable in the above code. We settled on a list of magnitudes containing five values: [0.05, 0.1, 0.5, 1.0, 5.0]; this list was determined while exploring results qualitatively in earlier stages of experiments. Drawing inspiration from the Monte Carlo method, for each subject, we perturbed each axis at each given magnitude 100 times, recording the performance of the face recognizer for each sample. Similar to above, we can conjecture that smaller magnitudes/perturbations will not drastically affect the face recognition system’s performance while larger perturbations might.
    # Random samples following N(mu, sigma^2)
    mu + sigma * np.random.randn(...) # FROM numpy-1.15.1 docs (numpy.random.randn.html)
    original_vector[axis] + magnitude * rnd.randn(Gs.input_shape[1]) # FROM StyleGAN_tour_2.py
    # Similarities?
    # mu = original_vector[axis]
    # sigma = magnitude
    # Perfect!
    

    With all these variables, the combinations are starting to add up. Just to keep track, let’s look at the case of our favorite subject: 04572. 1 subject (‘04572’) * 18 axes/subject * 5 magnitudes/axis * 100 samples/magnitude = 9000 samples per subject 9000 * 100 for all subjects… should take ~a long time.

  4. Which off-the-shelf face recognizer would we use? After reading literature and browsing available options, we chose the face_recognition package from ageitgey, which achieves state-of-the-art accuracy on the Labeled Faces in the Wild benchmark. The package sits atop davisking’s dlib library, which provides already-trained and easy-to-use CNN-based facial feature detectors and face encoding models.

# Given the data directories are linked up, this should run fine.
python StyleGAN_tour_2.py

Warning! Even with several GPUs, this process will take a very long time, especially in the case of larger datasets. To customize the script to your dataset, change lines 167 and 188-196. Additionally, differences in directory structure may need changing of some lines of code. Under the code is a glimpse of my directory structure. I have my ‘project data root’ directory in the same parent directory as my ‘stylegan_tour’ directory, fwiw.


# Wherever you see './data/FRGC/FRGC-2.0-dist/nd1/custom_100/', input your 'project data root' directory.
# Under your 'project data root' directory should be subdirectories for each subject
# Each subject subdirectory should be named appropriately and contain 2 things:
# - the latent_representation from the StyleGAN encoder
# - the subject's raw_image

### Line 167.

# dirnames = os.listdir('./data/FRGC/FRGC-2.0-dist/nd1/custom_100/') # CRC
dirnames = os.listdir('<path/to/project/data/root/directory>')

### Lines 188-196.

for dirname in dirnames:
    # vec = np.load('./data/FRGC/FRGC-2.0-dist/nd1/custom_100/' + dirname + '/' + dirname + '_01.npy')
    vec = np.load('<path/to/project/data/root/directory>' + dirname + '/' + dirname + '_01.npy')
    for a in range(18):
        # pkl_fname2 = './data/FRGC/FRGC-2.0-dist/nd1/custom_100/' + dirname + '/' + dirname + '_axis' + str(a) + '.pkl'
        pkl_fname2 = '<path/to/project/data/root/directory>' + dirname + '/' + dirname + '_axis' + str(a) + '.pkl'
        if not os.path.exists(pkl_fname2):
            print('mc_perturb-ing to create '+pkl_fname2)
            mc_perturb(vec, axis=a, pkl_fname=pkl_fname2)
        else:
            print(pkl_fname2 + ' already exists')

Directory Structure



Paper Template



Paper Title Here

Abstract

Introduction

Methodology

Results

Conclusions

Future Work


Extensions / Further Work