Posted on

Team’s Status Update for 11/13/20

This week, we continued working on the various portions of iRecruit. We made a decision to change the help page into a tips page, where we give users interviewing suggestions. This was because we thought that our dashboard page covered the purpose and navigation of iRecruit, and it would be helpful to give users information about good interviewing techniques, how to practice/dress for interviews, and more. We also decided to change the initial setup phase back to 5 seconds after more testing, because we realized if a user is set up and ready to go (sitting at a desk against a bare background), 5 seconds proves a sufficient amount of time.

Jessica worked on implementing the initial setup phase and off-center screen alignment detection for the mouth, and updating the home, dashboard, and technical interview pages on the web application. The initial setup phase and off-center screen alignment detection was similar to how she did them for the nose last week, where the X and Y coordinates of the mouth are stored into arrays for the first 5 seconds. Then, the average of the coordinates is taken, which provides the frame of reference coordinates of what is “center.” While recording, if the user is not within range of the frame of reference, we alert them with a pop-up message box. The home, dashboard, and technical interview pages were updated to match the behavioral interview page. They are now complete with detail and navigation functionality. Next week, Jessica is planning on integrating the behavioral interview questions that Shilika wrote into the facial detection code, so users know what question to answer during their recording. She is also planning on implementing the tips page on the web application.

Mohini worked on the web application component of the project. She created a user database that will be used to store summary information about the completed interview practices for each individual user. A change that the team decided to make is to store the counts of subpar eye contact and screen alignment as a part of this summary information rather than the videos themselves. We believed that this would be the most informative to the user. Additionally, Mohini fixed a few minor bugs regarding the login process and made sure that the workflow of the entire web application is as smooth as possible. She made more progress on the technical interview page as she displayed the user’s chosen category and a possible question to ask the user. Next steps include completing this workflow and retrieving the user’s submitted answer to check for accuracy purposes and to store in the user database. 

Shilika continued her work on adding an additional hidden layer to the neural network. She realized flaws in her algorithm from last week and had to re-do and re-write the approach. She is in the process of coding the new algorithm and will continue this process throughout the week. Additionally, she will help improve the accuracy of the neural network by adding more training data.

Posted on

Shilika’s Status Update for 11/13/20

This week, I put my focus on improving the accuracy of the neural network and working towards adding an additional layer to the neural network. As mentioned last week, I worked on applying the stochastic gradient descent on the additional parameters for the additional hidden layer in-between the first hidden layer and the output. I realized there were flaws in my algorithm from last week as I had skipped over some variables while computing the gradient. . Because I have been having a hard time understanding how to get this matrix, I created two flow charts – one of the original neural network we had and one with the additional hidden layer – to visualize how the values are propagating to the output layer. 

After visualizing the neural network, I outlined the algorithm used to compute the current alpha and beta parameters. This gave me more insight into what the computation would look like for the second alpha layer. As mentioned, I am in the process of coding and debugging the new algorithm for the additional parameter and would like to have it finished early next week. In addition to the neural network, I worked on cleaning up the css and html in the technical page and the navigation bars. This included small details such as fixing the alignment of the username, positioning the side navigation bar, and altering the headings of the components in the technical page to make it more user friendly. 

Next week, I plan to continue working on the technical interview page and the completed technical interview page on the profile page. I will also continue to work on improving the accuracy of the neural network by creating more training data as this has shown to improve the accuracy by 5% for every 30-40 additional training samples we create.

Posted on

Jessica’s Status Update for 11/13/2020

This week, I worked on implementing the initial setup phase and off-center screen alignment detection for the mouth, and updating the user interface for the home, dashboard, and technical interview pages on the web application. I decided to change the initial setup phase time back to 5 seconds (the original amount), because after running the program multiple times, I realized that if the user is set up and ready to go, 5 seconds is enough time. 10 seconds required a lot of sitting around and waiting. The initial setup phase and off-center screen alignment detection for the mouth is similar to that of the nose that I worked on last week. The X and Y coordinates of the mouth are stored into separate arrays for the first 5 seconds. We then take the average of the coordinates, which will give us the frame of reference coordinates for what constitutes as “center” for the mouth. For each video frame, we check if the current coordinates of the mouth are within range of the frame of reference coordinates. If they are not (or the nose coordinates are not), then we alert the user with a pop-up message box. If the nose coordinates are not centered, then neither are the mouth coordinates, and vice versa. I wanted to have both the nose and mouth coordinates for points of reference in case the landmark detection for one of them fails unexpectedly.

I also updated the user interface for the home, dashboard, and technical interview pages on the web application to make the pages more detailed and increase usability. For the home page, I adjusted the font and placement for the login and register buttons. For the dashboard, I reformatted the page to match the behavioral interview page. The dashboard is the user home page, which gives them an overview of what iRecruit has to offer and the various options they can navigate to. For the technical interview page, I also reformatted the page to match the behavioral interview page. The technical interview page provides users with information about the different technical question categories and instructions to audio record themselves. 

I believe that we are making good progress, as most of the technical implementation for the facial detection and web application portions are complete at this point. Next week, I plan on integrating the behavioral interview questions that Shilika wrote with the rest of the facial detection code, so that users have a question to answer during the video recording. I also plan on implementing the tips page on the web application. This was originally a help page, but we realized that our dashboard provides all of the information necessary for the user to navigate iRecruit. We thought that it would be better to have an interview tips page, where we give users suggestions on good interviewing techniques, how to practice for interviews, etc.

Posted on

Mohini’s Status Report for 11/13/2020

This week, I worked on a multitude of things. I started with looking into the speech recognition algorithm and thinking of possible ways to increase the accuracy. I created more training data which helped increase the accuracy by about 5%. I also tested the workflow and integration of the algorithm significantly, making sure that the signal processing and machine learning components work well together. 

Second, I worked on the web application this week. I spent a little bit of time cleaning up the backend logic for users to create a new account and login into iRecruit. This included fixing a minor bug so now the user’s username appears on the dashboard and the navigation bar. I also created a user database to store the information of any user that makes an account with iRecruit. This database will be utilized in the profile page to keep track of each individual user’s completed behavioral and technical interview practices. Additionally, I worked on the tips page and researched and added in tips for technical interviewing for the user’s convenience. 

Majority of my time was spent continuing to work on the technical interview page. I finished creating the questions database so that each of our eight categories have a couple questions. I displayed the user’s chosen category (the output of our speech recognition algorithm) on the webpage as well as a random question associated with that category. I also created an input text box for the user to submit their answer in. Next steps include writing backend code in the Django framework to retrieve the user’s answer and check its accuracy. I also plan on displaying a possible correct answer on the screen, so the user can compare theirs to this sample answer if desired. I will be storing the user’s questions and answers in the database, so that a summary of their practices can be displayed on the profile page.

I believe I am on progress as the skeleton of the technical interview page has been completed. I will spend the rest of the semester trying to improve the speech recognition algorithm and formatting the technical interview page to incorporate the best UI practices; however, I feel that the core of the project has been completed.

 

Posted on

Team’s Status Update for 11/06/20

This week, we continued working on implementing our respective three portions of the project. We made a design decision for the facial detection portion to give users three options to account for different levels of experience with behavioral interviews. The first option is for users who are of beginner-level, and allows for them to practice with both eye contact and screen alignment. We thought this would be good for users who are unfamiliar with behavioral interviewing or the iRecruit behavior interviewing platform, to give them maximum feedback. The second and third options are for users who are of intermediate-level to advanced-level, and allows for them to practice with either only eye contact or only screen alignment. We thought this would be good for users who know their strengths and weaknesses for behavioral interviewing, and only wish to receive feedback on one technique. 

Jessica worked on implementing the initial setup phase and off-center screen alignment detection for the nose, and updating the behavioral interview page on the web application. She was able to store the X and Y coordinates of the nose into arrays for the first 10 seconds, and then take the average of those coordinates to calculate the frame of reference coordinates. If the current coordinates of the nose for the video frame are not within range of the frame of reference coordinates, the user is alerted with a pop-up message box. She updated the behavioral interview page to give the user an overview and provide them with the three different options. Next week, she is planning to work on the initial setup phase and off-center screen alignment detection for the mouth, and updating the dashboard and technical interview pages.

Mohini worked on integrating the signal processing and machine learning components together with the Django webapp. The output from the signal processing is saved to a text file which is the input to the machine learning algorithm which then outputs the predicted category into a separate text file. Then, Django reads from this text file to display the predicted category on the webpage. When the user records the category of questions they are interested in receiving, the webpage displays the category name through the speech recognition algorithm. The accuracy of this algorithm is quite low, so next steps would be fine tuning the model to increase the accuracy. 

Shilika worked on the css and html for the web application portion, and saving videos to the profile page of the web app. She also worked on the neural network portion of the speech processing aspect. She is researching on improving the accuracy of the current neural network and implementing one more hidden layer in the neural network. Next week, she will continue improving the accuracy of the neural network and in turn the speech recognition.

 

Posted on

Shilika’s Status Update for 11/06/20

This week, I worked on the web application components and the neural network portion. For the web application, I made the css and html for the login and register pages more user friendly and appealing. The design now properly integrates with the rest of the web pages, as well.

In addition to the css, I continued to work on saving the completed behavioral interview videos on the web page. I have not been able to properly display the video, as a blank video appears in every web browser that I have tried such as Firefox, Safari, and Chrome. 

In the neural network portion, I worked with Mohini to continue the code we are using for our baseline (the neural network homework code from the Machine Learning course at Carnegie Mellon). In the beginning, the neural network was predicting the same output for every training and testing data point we provided. After debugging and testing, we realized that the ordering in which the training data is provided has an effect on the final outcome of the predictions. Despite varying the order of the training data, our accuracy of our testing data is still low at approximately a 42% accuracy rate. To improve the accuracy, I decided to implement an additional hidden layer in the neural network. The changes that this will require in the neural network is integrating an additional layer of hidden layer after first hidden layer, initializing weights associated with the neural network, performing a stochastic gradient descent to optimize the weights/parameters, and connecting the hidden layer with the two output classes. I am currently working on applying SGD on the parameters and have been running into index out of bound bugs. 

By next week, I hope to have completed this layer and run it to test if the accuracy has increased. I will also continue to research other methodologies to improve the accuracy.  I also hope to figure out displaying completed behavioral interview videos in the django webapp. I am behind in this aspect because I intended to finish it this week. In order to get back on track, I will reach out to my team to help me with this portion as I have not been able to figure it out despite trying multiple possibilities that I found through online resources.

Posted on

Jessica’s Status Update for 11/06/2020

This week, I worked on implementing the initial setup phase and off-center screen alignment detection for the nose, and updating the web application for the behavioral interview page. For the initial setup phase, it is similar to the eye contact portion, where the coordinates (X, Y) of the nose are stored into arrays for the first 10 seconds. Then, the average of the coordinates are taken, and that gives us the coordinates that will serve as the frame of reference for what is “center.” For the off-center screen alignment detection for the nose, we check if the current coordinates of the nose for the video frame are within range of the frame of reference coordinates. If they are not, we alert the user to align their face with a pop-up message box. 

One change that we made this week was that we decided to split up the facial detection portion into three different options. We were thinking about it from the user perspective, and thought that it would be good to account for different levels of experience with behavioral interviewing. The first option is for beginner-level users, who are unfamiliar with the iRecruit behavioral interviewing platform or with behavior interviews in general. It allows for users to practice with both eye contact and screen alignment, so iRecruit will provide real-time feedback for both aspects. The second and third options are for intermediate-level to advanced-level users, who are familiar with behavioral interviewing and know what they would like to improve upon. The second option allows for users to practice with only eye contact and the third option allows for users to practice with only screen alignment. We thought this would be useful if a user knows their strengths and only wants to practice with feedback on one of the interview tactics. I separated these three options into three different code files (facial_detection.py, eye_contact.py, and screen_alignment.py).

I was able to update the web application for the behavioral interview page (see image below) to make the interface more detailed and user-friendly. The page gives an overview and describes the various options available. I was able to learn more about Django, HTML, and CSS from this, which was very helpful! I believe that we are making good progress with the facial detection part. Next week, I plan on working on the initial setup phase and off-center screen alignment detection for the mouth. This will probably wrap up the main technical implementation for the facial landmark detection portion. I also plan on updating the user interface for the dashboard and technical interview pages on the web application.

Posted on

Mohini’s Status Update for 11/06/2020

This week, I worked on integrating the signal processing and machine learning algorithms in order to create a complete speech recognition implementation. First, I finished creating the training data. This involved recording myself speaking a word and letting the algorithm run that results in the binary representation of the data being stored in a text file. I manually appended the contents in this temporary text file as well as the English representation of the word to my training data text file. I decided to record each of the 8 categories 8 different times for a total of 64 samples in the training data set. This process was quite tedious and took a couple hours to complete as I had to wait for the signal processing algorithm to run for each sample. I used a similar approach to create the testing data set. Currently, there are only 7 samples in it, but I will add more samples in the upcoming future.

Next, I used the training and test data sets as the input to the neural network implementation. I coded the baseline of this implementation from scratch last year for my 10301: Introduction to Machine Learning course. I had to tweak a few things in order to adapt the code for the purposes of this project. One challenge was formatting the datasets in the best way so that the reading and processing of those files is as simplified as possible. Another challenge was ordering the data in the training dataset in the optimal order as changing the order of the data had a significant impact on the accuracy of the model. For example, I noticed that the accuracy of the model decreased if the training dataset had multiple samples of the same word in a row. After overcoming these obstacles, I modified the stochastic gradient descent algorithm to work with these datasets and fine tuned the parameter matrices. Then I wrote a predict function using the optimal parameter matrices determined from training the neural network in order to predict the corresponding English word for each sample in the test data set. Currently, this accuracy is at 42%, but I will work on improving this in the upcoming future. 

Finally, I integrated the speech recognition implementation with the Django web app so that the user can record themselves from the technical interview page and the algorithm returns the predicted word. This word is then displayed on the technical interview page. Next steps will include improving the accuracy of this algorithm and picking a question from the corresponding category to display on the screen. 

The image below is a snapshot of the training data set. Each sample is of length 15000.

Posted on

Team’s Status Update for 10/30/20

This week, we continued working on implementing our respective portions of the project, making progress in the three main parts. Jessica worked on implementing a center of frame reference, facial landmark detection, and testing the eye detection portion. She thought it would be helpful to have a centered guideline for users to position themselves accordingly during the initial setup phase, so that they have a reference for the center of the video screen. She continued working on the facial landmark detection, and was able to get the coordinates of the center of the nose and mouth. The eye detection portion was also tested more, and the results seem to align with the accuracy goal. Next week, she will work on completing the initial setup phase for the facial landmark detection, and would like to complete the off-center screen alignment portion for the nose as well. She will also continue testing both the eye detection and screen alignment parts. 

Mohini finalized the signal processing algorithm and started making the training data set for the neural network algorithm. This week concluded the signal processing portion of our project, so I will be focusing on the machine learning portion as well as integrating the different components of our project together for the rest of the semester. Next week, I will be working on testing the neural network after finishing generating the rest of the training data set. 

Shilika began reviewing the neural network concepts, as this is the next technical aspect of the technical interview portion that she will help tackle. She also continued to work on the web application to improve the css and features that appear in the front-end to make the app more user friendly. Next week, she will continue working on the web application and the neural network. 

Posted on

Shilika’s Status Report for 10/30/20

This week after finalizing the output of the signal processing, I began to review the concepts of a neural network which will be the next technical portion of our project. I will be working with Mohini to improve the neural network that we created in a Machine Learning course we previously took. In this algorithm, we use a single layer neural network that uses a sigmoid activation function for the hidden layer, a softmax function on the output layer, and the cross-entropy loss function to gauge the accuracy of our model. I reviewed the concepts behind these activation functions and how the output layer is formed using the input layer and hidden layers. 

I additionally started working on the web application components of our project again. I worked on how to run the java code in django and used the “copy path” command to be able to run the code from a separate direction. I also began working on the profile page again which is where the user will be able to save their skill set and view previously recorded behavioral interviews. I improved the css for the profile page to make it more user friendly and began to look at saving the videos locally in django.

Next week, my goal is to be able to save the videos on django and allow the user to upload a profile photo to the profile page. Additionally as soon as our training data is ready, I start implementing ways in which our neural network can be improved to classify our 8 outputs.