Speech2face Instruction

Notice

• This instruction is mainly written for the NAMZ & EMCS members. Therefore, the other users who would like to duplicate this tutorial might experience unexpected errors.

System Settings

• Mac OSX(Sierra)
• Matlab

Speech2face Introduction

---

• Speech2face is a plotting tool that visualizes dynamic lips movement in real time. This is trained with deep neural network which receives speech signals as inputs to predict lip parameters (as outputs). (Speech2face will be updated to visualize whole face movement.)
• By uttering a few sentences, the speaker is able to receive instantaneous feedback of his or her lip movement from the machine.

Pronouncing /a/,/o/,/i/

Install Feature Extractors

---

• Go visit this github and download speech2face to your local computer.
• We will use ‘FaceTracker’ and ‘OpenFace’ toolkit in order to retrieve 3 types of data: images, raw.pcm(audio file), and parameters.txt (pts data).
• Two toolkits are already donwloaded and manipulated for the task’s sake, you don’t have to visit the following github in order to download them again.
  
  
  1. FaceTracker
     
     • FaceTracker is a library for deformable face tracking.
       
       
       Video recording with FaceTracker
     • Installation
       
       • Install opencv3 and openal.
         

         $ # install opencv3
         $ brew tap homebrew/science 
         $ brew install homebrew
         $ # install openal
         $ brew install openal-soft

       • Open Makefile and add “(absolute/path)/openal-soft/1.18.0/lib/libopenal.1.dylib -lpthread” to LIBRARIES(line:14) variable.
       • Navigate to “speech2face/extractors/FaceTracker/src/exe” and open face_tracker.cc
       • In line 55, you will see “save_dir” variable. This path is where the data is saved. Fix the directory path.
         

         // Fix this line.
         const char *save_dir = "/Users/hyungwonyang/Google_Drive/C++/FaceTracker/extracted_data/;

       • I recommend you to change the path as follows. just simply fix this part: “path/from/Users/to”.
         

         const char *save_dir = "path/from/Users/to/speech2face/extractors/FaceTracker/extracted_data"           

       • After finishing all the procedures above, type “make” in the terminal.
     • Running a tool
       
       • Navigate to “./bin” and type ./face_tracker”
         
  2. OpenFace
     
     • OpenFace is an open source facial behavior analysis toolkit. more information
       
       
       Video recording with OpenFace
     • Installation
       
       • Install brew first but I believe all of you already installed it.
       • Install XQuartz
       • Install boost, TBB, and OpenCV
         

         $ brew tap homebrew/science
         $ brew install boost tbb opencv3

       • Navigate to “speech2face/extractors/FaceTracker” and open “CMakeLists.txt”
       • Find “find_package( OpenCV REQUIRED )” and change it to “find_package( OpenCV HINTS your/opencv3/path )” For example,
         

         find_package( OpenCV HINTS /usr/local/Cellar/opencv3/3.1.0_3/ )

       • Add following lines right after the boost packages.
         

         find_package( X11 REQUIRED )
         MESSAGE("X11 information:")
         MESSAGE("  X11_INCLUDE_DIR: ${X11_INCLUDE_DIR}")
         MESSAGE("  X11_LIBRARIES: ${X11_LIBRARIES}")
         MESSAGE("  X11_LIBRARY_DIRS: ${X11_LIBRARY_DIRS}")
         include_directories( ${X11_INCLUDE_DIR} )

       • Do build process by following commands in the terminal.
         

         $ cmake -D CMAKE_BUILD_TYPE=RELEASE .
         $ make

     • Data Process
       
       • Take a video with QuickTimePlayer and save it to “OpenFace/data/video” and run following lines in matlab. Extract images and audio signals from video file and save them to “OpenFace/data/inputs”.
         

         >> % matlab
         >> % Switch fileName as a audio file name.
         >> video_file = './Openface/data/video/fileName';
         >> v = VideoReader(video_file);
         >> i = 1;
         >> while hasFrame(v)
         >>     i = i+1;
         >>     video{i} = readFrame(v);
         >> end
         >> % save images.
         >> for i = 1:length(video)
         >>     f_name = sprintf('video_%04d.png',i);
         >>     imwrite(video{i+1},['./OpenFace/data/inputs/',f_name])
         >> end      

       • Run the following code in terminal, then you will get pts information.
         

         $ # inputs(folder where images are saved) 
         $ # outputs(folder where pts will be saved)
         $ bin/FaceLandmarkImg -fdir ./inputs -ofdir ./outputs

Speech2face Tutorial

---

• If you download “speech2face” from the github and finished the installation procedure, then run tutorial.m code block by block.

During tutorial, training set is checked whether it is recoreded correctly.

Developer

---

• 06.21.17
• Hyungwon Yang (NAMZ & EMCS Labs)
• hyung8758@gmail.com
• I mainly developed speech2face matlab code and borrowed two other toolkits(FaceTracker, and OpenFace)

Hyungwon Yang