Posted on

Team’s Status Update for 12/04/20

This week, we continued working on the various components of iRecruit. Jessica worked on integrating the eye contact and screen alignment parts together for option 1 of the behavioral interview options, and figured out how to store summary information for each video recording. She was able to combine the eye contact and screen alignment parts to alert the user of both subpar eye contact and screen alignment. There are setup phases for the eye detection and facial landmark detection parts, where the frame of reference coordinates for the center of the eyes, nose, and mouth are calculated in the first 5 seconds. If a user’s eyes, nose, or mouth are off-center, they are alerted with the appropriate category (eye contact or screen alignment). She also implemented the video summaries for the profile part of the behavioral interviews. There is a common text file that keeps track of all the video summaries. For each video recording, the interview number, timestamp, number of times the user had subpar eye contact, and number of times the user had subpar screen alignment are appended to a new line in the file. Next week, she plans on integrating this text file with Django, so that the video summaries show up on the profile page. She also plans on continuing to test the eye contact only option, and beginning to test the screen alignment only option and the integrated option. 

Mohini worked on a couple different things this week. Since she finished most of the implementation last week, she spent her time this week testing and refining. She refined the backend database model that kept track of each user and their corresponding questions and answers. Then, she tested the speech recognition algorithm using both automated and manual testing. Automated testing consisted of having a fixed testing data set of approximately 50 samples whereas manual testing consisted of retrieving the testing input through the signal processing algorithm. She will continue this testing next week to determine the true accuracy of the algorithm. Lastly, Mohini started working on the final report and incorporating some of the design changes that we made throughout the project. Next week, she will start recording her parts for the final video demo. 

Shilika worked on the web application and added finishing touches such as the Completed Technical Interview page and minor css changes. She also worked on testing the speech recognition algorithm. She ran the automated testing and collected data. Additionally, she ran the in-built gnb methods in python to compare the results of that against the neural network. Next week, she will continue testing the performance of the speech recognition model, and work on the final video and report. 

Posted on

Jessica’s Status Update for 12/04/2020

This week, I worked on combining the eye contact and screen alignment parts together for the facial detection portion, and implemented a way to store summaries about each video recording. I was able to integrate the eye contact part and the screen alignment part for option 1 to alert the user of both subpar eye contact and screen alignment. This required combining the two pieces of code I had written separately for options 2 and 3, so that there are setup phases for both eye detection and facial landmark detection, and that the coordinates of the center of the eyes and the nose/mouth were averaged during the initial 5 seconds. We then have separate frames of references for the eyes, nose, and mouth. In the respective parts of the code, if the current eye, nose, or mouth coordinates are off-center, the user is alerted of the appropriate one (eye contact or screen alignment). 

We were going to store the video recordings at the beginning of our project, and then allow users to view them in the profile section. However, we decided that it would be more helpful if we summarized the feedback from each video recording. There is a common text file (called behavioral_interview_output.txt) to store the video summaries. We calculate the interview number by counting the number of lines in the text file, and retrieve the timestamp of when the video practice took place using the Python datetime library. We keep track of the amount of times that the user had subpar eye contact and/or screen alignment during a video recording using count variables. The interview number, timestamp, subpar eye contact count, and subpar screen alignment count (for options 2 and 3, subpar screen alignment is “N/A” and subpar eye contact is “N/A,” respectively) are appended to the text file. This text file is to be displayed on the behavioral section of the profile page for the user to access.

I believe that we are making good progress for the facial detection portion, as we are wrapping up the technical portions and were able to accomplish a lot of the corresponding profile part as well. Next week, I plan on integrating the text file of the video recording summaries into Django for the web application. I also plan on continuing testing for the eye contact only option, and beginning testing for the screen alignment only option and the integrated eye contact and screen alignment option. I would like to get an idea of the current accuracy of the systems. 

Posted on

Mohini’s Status Report for 12/04/2020

Since I wrapped up completing the implementation last week, I spent most of my time this week testing and putting final touches to the web app. I updated the user model that keeps track of the user’s name and all their questions and answers. I tested this functionality thoroughly in order to ensure that if I logged out and then logged back in as a different user, then a new set of questions and answers are displayed on the completed technical page that correspond to this specific user. The completed technical page is a running record of all the questions and answers that belong to the logged in user. 

Second, I formally tested the speech recognition algorithm. This was accomplished in two different ways. First, I did automated testing with fixed testing data. I created a testing data file with approximately 50 samples representative of all 8 categories. I ran the speech recognition algorithm to predict the category of all 50 samples. The accuracy fell around 30-40% and varied significantly between different test cases. Next, I did manual testing that tested the integration of  the signal processing and machine learning components. Here, I created the sample of testing data through audio recording my voice and letting the signal processing algorithm decompose it. I created a spreadsheet to keep track of my manual testing. The spreadsheet consists of the true word and the predicted word, and has a similar accuracy of 30-40% as the automated testing. 

Lastly, I started working on the final report. I used our design report as a template and incorporated some of the many design changes that we made throughout the project. Some of the updates included word classification (rather than letter classification), the layout of the technical interviewing page, and the information that we stored in the profile page for the completed behavioral and technical pages. Next week, I plan on continuing to test the speech recognition algorithm through both automated and manual testing. I will also start recording my part for the final video demo. I believe we are making good progress as iRecruit is almost complete! 

 

Posted on

Team’s Status Update for 11/20/20

This week, we continued working on our respective parts of the project. We decided that instead of storing the behavioral interview videos, we will be storing a summary for each video with the amount of times the user had subpar eye contact and/or screen alignment. This is because we found out that it is difficult to display videos embedded in HTML from a local directory (for privacy and security reasons in HTML5), and it did not make sense to store all the videos in a public space. We also realized that it may be more helpful to the user if we provide them with a summary of how many times they had subpar eye contact and/or screen alignment for each behavioral interview practice they do. This way, they have concrete data to base their improvement off of.

Jessica worked on integrating the behavioral interview questions that Shilika wrote into the rest of the facial detection code and the behavioral portion of the tips page on the web application. She used the questions that Shilika researched at the beginning of the semester, and selected a handful of them to place into a global array. Using the Python random library, one question is selected at random and displayed at the top of the video screen for the user to refer to as they wish. She also implemented the behavioral section of the tips page to provide the user information about behavioral interviews and how to prepare for them. Mohini and Shilika will be working on the technical section of the page. Next week, she plans on combining the eye contact and screen alignment portions together for Option 1 of the facial detection part. This will provide users with alerts for both subbar eye contact and screen alignment. She also plans on figuring out how to keep track of the amount of times that the user had subpar eye contact and/or screen alignment during each video recording. This will be the summary available to users in their profile.

Mohini worked on improving the accuracy of the speech recognition algorithm through the addition of another hidden layer in the neural network and adding more training data. While both these attempts were not successful, she plans to continue trying to improve the accuracy by tweaking some of the other parameters of the neural network. Additionally, she continued working on the technical page. The chosen category, question, user’s answer, and correct answer are all displayed on the screen. She then stores all this information into a database so that a list of the logged in user’s questions and answers can be displayed on the completed technical interview practices page, which can be accessed through the profile page. Next week, Mohini will continue trying to fine tune the parameters of the neural network and start the testing of the speech recognition algorithm. 

Shilika worked on the neural network portion of the speech recognition algorithm. She was able to successfully add another layer and update all parameters and variables accordingly. This required changes to Mohini’s working neural network algorithm which had one hidden layer. The accuracy of the updated algorithm did not reach the expectations that we have set (60% accuracy). Next steps for Shilika include continuing the improve the accuracy of the speech recognition by changing variable parameters such as the number of hidden units and the amount of training data. She will also work on the profile page of the web application. 

Posted on

Shilika’s Status Update for 11/20/20

This week, I successfully added a hidden layer to the neural network. I worked with Mohini to gain a solid understanding of the back propagation algorithm. This gave me a clear understanding of how to update the three parameters – alpha, alpha2, and beta – of the new neural network. Alpha being the one between the input layer and first hidden layer, alpha2 being the one between the first and second hidden layer, and beta being the one between the second hidden layer and output layer. Using this knowledge, I incorporated the second hidden layer. In this algorithm, the two hidden layers have the same number of hidden units depending on the user input.

After this, I began testing the algorithm with our current training data to gauge its accuracy as compared to the algorithm with one hidden layer. The training data and testing data with the new neural network have an accuracy of 83.5% and 86.3% respectively, while the training and testing data with the previous neural network have an accuracy of 1.5% and 77.2% respectively. The results of the training data were as expected because an additional hidden layer reduces overfitting. The would result in a higher training error since the algorithm will classify the training data less correctly. However, I was disappointed to see the testing error slightly increase. I did not expect it to improve significantly, but at least marginally. This is because additional hidden layers bring more non-linearity to the neural network, and thus the classification. Therefore, it would be able to generalize the data better. 

My next steps will be to continue to improve the accuracy of the neural network. I will modify the variable values such as the number of training data and the number of hidden units in each hidden layer. I will continue to monitor how these changes affect the accuracy of the algorithm and try to find the optimal parameters to achieve the highest accuracy. Additionally, I will start working on the web application again to complete the profile page.

Posted on

Mohini’s Status Report for 11/20/2020

This week, I worked on various different things. I started with reviewing the neural net & back propagation algorithm with Shilika. We needed to take my working implementation for one hidden layer and apply it to determine the stochastic gradient algorithm for the addition of another hidden layer. After drawing out the feed forward algorithm with the different parameters, we derived the back propagation algorithm derivatives into a simplified form and coded the equations. Shilika was in charge of the coding aspect; I simply helped her think through the algorithm. 

Next, I continued working on the technical interview page. I focused on two things – improving the CSS of the page to make it more visually appealing and implementing the backend Django code for retrieving and saving the user’s answers to a database. Rather than tell the user if their answer is correct, I displayed the user’s submitted answer and the correct answer on the screen to let them decide if their answer is correct. I also created a database to store all the user information. This includes their name, email, and list of questions as well as their answers to the questions. For the CSS aspect, I played around with the layout of the page so that the chosen category and question banners appear under each other rather than next to each other. I changed the colors and positions of a few other elements to make the page, as a whole, more visually appealing. Additionally, I continued researching and adding tips to our tips page, as well as improving the CSS of that part. 

Lastly, I started testing the speech recognition algorithm. I created more training data, but the addition of this training data actually decreased the accuracy of the algorithm. I haven’t found the sweet spot of the optimal number of training data samples nor the optimal parameters to run the neural network. I will experiment to find the optimal ones for next week. 

 

Posted on

Jessica’s Status Update for 11/20/2020

This week, I worked on integrating the behavioral interview questions that Shilika compiled into the facial detection system and implementing the behavioral portion of the tips page. Shilika created a file at the beginning of the semester that contains a large number of common behavioral interview questions. I took a handful of these questions and placed them into a global array in the eye contact and screen alignment code. Then, using Python’s random library, I choose one of the questions in the array at random. This chosen question is displayed at the top of the video screen while the user records themselves, so that they can refer back to the question whenever they want to.

I also worked on implementing the behavioral portion of the tips page that we decided to change from the original help page last week. I used articles from Indeed and Glassdoor to provide the user with information about behavior interviews, as well as helpful tips and common behavioral interview questions. This gives background to the user on how to prepare for behavioral interviews and what to expect during a typical interview. The tips page will be divided into two sections, one for behavioral and one for technical. I implemented the behavioral section, and Mohini and Shilika will be working on the technical section for technical interview information and tips.

I believe that we are on track for the facial detection portion, as most of the technical implementation is complete and now we are working on improving and fine-tuning. Next week, I plan on working on combining the eye contact and screen alignment parts together for the first option on the behavioral page. I also plan on figuring out how to keep track of the amount of times that the user had subpar eye contact and/or screen alignment during their video recording, so we can provide this information to the user in their profile page.

Posted on

Team’s Status Update for 11/13/20

This week, we continued working on the various portions of iRecruit. We made a decision to change the help page into a tips page, where we give users interviewing suggestions. This was because we thought that our dashboard page covered the purpose and navigation of iRecruit, and it would be helpful to give users information about good interviewing techniques, how to practice/dress for interviews, and more. We also decided to change the initial setup phase back to 5 seconds after more testing, because we realized if a user is set up and ready to go (sitting at a desk against a bare background), 5 seconds proves a sufficient amount of time.

Jessica worked on implementing the initial setup phase and off-center screen alignment detection for the mouth, and updating the home, dashboard, and technical interview pages on the web application. The initial setup phase and off-center screen alignment detection was similar to how she did them for the nose last week, where the X and Y coordinates of the mouth are stored into arrays for the first 5 seconds. Then, the average of the coordinates is taken, which provides the frame of reference coordinates of what is “center.” While recording, if the user is not within range of the frame of reference, we alert them with a pop-up message box. The home, dashboard, and technical interview pages were updated to match the behavioral interview page. They are now complete with detail and navigation functionality. Next week, Jessica is planning on integrating the behavioral interview questions that Shilika wrote into the facial detection code, so users know what question to answer during their recording. She is also planning on implementing the tips page on the web application.

Mohini worked on the web application component of the project. She created a user database that will be used to store summary information about the completed interview practices for each individual user. A change that the team decided to make is to store the counts of subpar eye contact and screen alignment as a part of this summary information rather than the videos themselves. We believed that this would be the most informative to the user. Additionally, Mohini fixed a few minor bugs regarding the login process and made sure that the workflow of the entire web application is as smooth as possible. She made more progress on the technical interview page as she displayed the user’s chosen category and a possible question to ask the user. Next steps include completing this workflow and retrieving the user’s submitted answer to check for accuracy purposes and to store in the user database. 

Shilika continued her work on adding an additional hidden layer to the neural network. She realized flaws in her algorithm from last week and had to re-do and re-write the approach. She is in the process of coding the new algorithm and will continue this process throughout the week. Additionally, she will help improve the accuracy of the neural network by adding more training data.

Posted on

Shilika’s Status Update for 11/13/20

This week, I put my focus on improving the accuracy of the neural network and working towards adding an additional layer to the neural network. As mentioned last week, I worked on applying the stochastic gradient descent on the additional parameters for the additional hidden layer in-between the first hidden layer and the output. I realized there were flaws in my algorithm from last week as I had skipped over some variables while computing the gradient. . Because I have been having a hard time understanding how to get this matrix, I created two flow charts – one of the original neural network we had and one with the additional hidden layer – to visualize how the values are propagating to the output layer. 

After visualizing the neural network, I outlined the algorithm used to compute the current alpha and beta parameters. This gave me more insight into what the computation would look like for the second alpha layer. As mentioned, I am in the process of coding and debugging the new algorithm for the additional parameter and would like to have it finished early next week. In addition to the neural network, I worked on cleaning up the css and html in the technical page and the navigation bars. This included small details such as fixing the alignment of the username, positioning the side navigation bar, and altering the headings of the components in the technical page to make it more user friendly. 

Next week, I plan to continue working on the technical interview page and the completed technical interview page on the profile page. I will also continue to work on improving the accuracy of the neural network by creating more training data as this has shown to improve the accuracy by 5% for every 30-40 additional training samples we create.

Posted on

Jessica’s Status Update for 11/13/2020

This week, I worked on implementing the initial setup phase and off-center screen alignment detection for the mouth, and updating the user interface for the home, dashboard, and technical interview pages on the web application. I decided to change the initial setup phase time back to 5 seconds (the original amount), because after running the program multiple times, I realized that if the user is set up and ready to go, 5 seconds is enough time. 10 seconds required a lot of sitting around and waiting. The initial setup phase and off-center screen alignment detection for the mouth is similar to that of the nose that I worked on last week. The X and Y coordinates of the mouth are stored into separate arrays for the first 5 seconds. We then take the average of the coordinates, which will give us the frame of reference coordinates for what constitutes as “center” for the mouth. For each video frame, we check if the current coordinates of the mouth are within range of the frame of reference coordinates. If they are not (or the nose coordinates are not), then we alert the user with a pop-up message box. If the nose coordinates are not centered, then neither are the mouth coordinates, and vice versa. I wanted to have both the nose and mouth coordinates for points of reference in case the landmark detection for one of them fails unexpectedly.

I also updated the user interface for the home, dashboard, and technical interview pages on the web application to make the pages more detailed and increase usability. For the home page, I adjusted the font and placement for the login and register buttons. For the dashboard, I reformatted the page to match the behavioral interview page. The dashboard is the user home page, which gives them an overview of what iRecruit has to offer and the various options they can navigate to. For the technical interview page, I also reformatted the page to match the behavioral interview page. The technical interview page provides users with information about the different technical question categories and instructions to audio record themselves. 

I believe that we are making good progress, as most of the technical implementation for the facial detection and web application portions are complete at this point. Next week, I plan on integrating the behavioral interview questions that Shilika wrote with the rest of the facial detection code, so that users have a question to answer during the video recording. I also plan on implementing the tips page on the web application. This was originally a help page, but we realized that our dashboard provides all of the information necessary for the user to navigate iRecruit. We thought that it would be better to have an interview tips page, where we give users suggestions on good interviewing techniques, how to practice for interviews, etc.