Category: Jake’s Status Report

This week I focused on completing a few of the remaining things from my schedule. The first of these was to convert the EC2 database to run MySQL. For the demo we were just running sqlLite as a proof of concept but to ensure we are optimizing for our design requirements a change to MySQL was preferred. I also worked on updating some of Angela’s old front end html and django code and implemented that on the ec2 so that it runs on our website. I think this upcoming week I am going to create seperate versions of both this front end and our esp8266 code so that we can have the compartmentalized demos that we discussed with Tamal this week. The webapp code for some of the demos will be able to display info on exact latency and capacity.

I additionally developed two of the tests/validations our team initially had planned to do. The first of these is tracking the aws -> website capacity and loading times. I’m utilizing GTMmetrix which is a pretty standard speed test tool that performs multiple tests across various browsers, locations, and connections. I have yet to run this as Angela is finishing up her finalized front end which is the element of our pipeline this seeks to stress test. The Second of these tests is a fleet of node mcu’s that all are publishing json packets. This stress tests the device node -> aws capacity. This is more of a verification/validation than a test as there isn’t a solid metric for this other than delay and packet loss.

Last week I mentioned that we needed to  convert all devices to access CMU-SECURE through a wifi access point as CMU-DEVICE was blocking certain things we were trying to do with NTP. Now that we have confirmed our final demo location as Wiegland Gym we have concluded an ethernet based wifi access point is not viable as there isn’t accessible ethernet access for our team. I spent a lot of time researching wifi repeaters as a circumnavigation and we found several possible options that would work for our final demo. We are going to test one of these options, using a Samsung Phone as a wifi repeater this upcoming week. This is probably our greatest issue for risk mitigation at the moment. Our initial demo proved that we can still have a working project without this solution however to realize the full vision of our project this will need to be solved.

In terms of our updated schedule I am right on pace.

This week was super busy in terms of preparing for demos. Due to Ryan’s illness and Angela being busy I had to do a lot more work than initially anticipated. Because of this I am going to break down all my progress and issues by platform

AWS IoT – Originally our design had our incoming data going to AWS IoT’s internal time series data base and then the platform’s rules engine would utilize a Lambda Rule to pass this along to a MySQL DB hosted by Amazon RDS. Django would access this directly. I fully implemented this but kept running into errors integrating this pipeline with Django’s models. Django models are intended to be a wrapper that sits over SQL and keeps the programmer completely removed from it, but because I was writing a bunch of SQL myself as part of the Lambda function Django really didn’t like that. This left me with two options either abandon Django models and go with straight py sql or integrate the DB on the EC2. Because as a team we wanted to go with Django b/c of the abilities of models I made the design pivot to move towards a database with the django MVC on the EC2. This simplified the AWS IoT portion of our project in that I completely cut out the Amazon RDS MySQL server and lambda function and I replaced them with an AWS IoT rule that sent data within a https post to a subdomain of our hosted website.

EC2/Django – So this clearly had trickle down effects within the EC2 Django deployment. For one I created a bunch of code that established Django models for tables with all their necessary features i.e. table_id, occupancy status, time of last update, etc.. I didn’t just create the templates as I also created a migration such that whenever the database is reinitialized  or the server boots up the database is prepopulated with instances of tables for all the devices we are currently supporting. In terms of data communication from AWS to django I talked briefly about how AWS IoT was sending a post request to a specific url, this is specifically a json dictionary that contains all data transfered off the esp8266 as well as a key to match for security so nobody can spoof send data.

Because of the change in overall design I performed some extensive load testing to ensure this could still meet our project requirements. I converted the domain to point not an the original server instance but rather an elastic IP to help with load. For testing I flooded the EC2 with 80 db updates and 100 get requests within a minute and it experienced no issues whatsoever so the change still meets our project requirements.

I additionally worked a bunch on the front end as nothing really existed for it yet. I built out a basic html file to hold table information (id: status) and then tied this in with a javascript, jquery, and Ajax set of functions that would via a url automatically refresh occupancy data at a set interval of time. This tied into some backend code I wrote in Django that would query the database for all Table model instances, package these in a dictionary and send them off in json as a xml request response to the browser.

hardware/esp8266 – Ryan hasn’t been able to do any programming on the esp8266 yet so I wrote a script to perform basic pir based occupancy detection for the demo. This consisted of two versions of detection. One that would connect to wifi only when detection had occurred as a preliminary power saving technique and one that would stay connected. This will slot into our final demo idea of having two detection algorithms running. One that will show purely how well we can detect occupancy and another that will show our in practice version designed around saving power through wifi use minimization strategies.  We ran into a new problem using CMU-DEVICE Wifi where we wern’t allowed to use NTP for any accurate time synchronization. In our demo testing we found that there isn’t a reliable delay between detection and when status updates reach our sql. Because of this as well as our desire to ensure we are operating our esp8266’s  during our desired running window, 8am-5pm, we wanted to use NTP for a variety of things on our nodes. Because of this we will be pivoting from CMU-DEVICE to using a wifi access point to connect to CMU-SECURE.

 

This week has been primarily focused on getting everything in order for the demo. I purchased the domain occupancymonkey.com. I started a new Ec2 as Angela was having difficulty and cloned our project there. The django development server is not an acceptable way to run our program as it doesn’t handle high loads /service restarts  so I pursued using apache2 to run our program via a wsgi mod. I successfully completed this and then using ip forwarding I tied this to our domain at occupancymonkey.com. From here I utilized ssl to allow for https inbound and outbound traffic. This serves two purposes one it will allow aws iot to send messages safely via https which will greatly simplify data matriculation. Secondly it will help keep data more secure following the ethics discussion realizations are team had.

We hit a slight schedule snag in that our esp8266s are no longer behaving with cmu-device. Whereas in the past we were able to connect to EC2 cmu-device is now refusing us the ability to use pub sub client. Our mitigation strategy that I’m working on before the demo is converting a raspberry pi to a router/wifi access point

 

This week I and the team worked primarily on the ethics side of the project. We participated in a very productive in class discussion over the potential ethical implications of our project and as a team we had the opportunity to discuss potential solutions/mitigation factors. Some of the potential implications were as follows:

• Currently we have no features that limit who can see the real time data on occupancy in the UC. The multiple problems people found with this ethically were that it displaying when there is a high concentration of people within the UC allows potentially nefarious actors to come in and abuse this concentration in a multitude of ways ranging from solicitation and advertising to potentially things more harmful. To address this as a time we are going to require an andrewid to access the data. As CMU students and faculty are the target users we don’t see this restriction harming our use case however it is important to note that Chartwells employees do use the tables and this change will discriminate against them but as team we decided this trade off vs the original ethical concern was worth it to secure the safety of CMU students.

• The second ethical concern raised was that people have no way of opting into having theirselves tracked. This was something our team hadn’t really considered the ethical implications of. However we concluded that ethically already having security cameras in the UC supercede the table occupancy tracking. That being said we also wanted to spread flyers notifying people of the occupancy trackers, just saying what they are, the way the hardware works and that no personally-identifiable data is being gathered as we advertise the app.

In terms of the hardware software integration I had hoped to finish the data transfer pipeline last week I still have not finished yet but I made some good progress. In terms of our schedule this all needs to be done for our demo so I will work to finish it up this week. Additionally this week, I plan to integrate my data transfer pipeline with Angela’s EC2 django instance.

This week was spent mostly on integrating the hardware software interface. To start off the week I registered all of our esp8266s with CMU-Device’s wifi registration. I worked through getting these new devices to connect to our aws iot accounts. I successfully started quick connecting the esp8266 to aws iot and pushing data packets with table occupancy. Code can be found here: https://github.com/ayzhang00/OccupancyMonkey/blob/main/Esp8266/aws_iot_main.

I designed the code such that all secure information i.e. keys and wifi information is entered into a small header file that is added at device flash so no outsider can peek at that information on our github. I additionally created our mySQL database hosted by Amazon RDS and am close to finalizing the AWS lambda rule to push the sent data to this mySQL database. The code for this is similar in structure to the esp8266 code where all sensitive information is placed in a small additional file. The code is found at: https://github.com/ayzhang00/OccupancyMonkey/tree/main/send_data_to_rds

I am about on schedule, there is a slight hiccup with AWS that I need to work through as it appears our EC2 credits do not count towards AWS IOT and Amazon RDS payments, as I initially hoped. I think the RDS free tier will suffice unless Django has some unanticipated quirks. The risk mitigation for this would be less reliance on Django models for the substitution of the pymsql library or something similar that gives tighter control over database reads.

Next week, will be spent completing the implementation of our hardware software integration by finalizing the Lambda function with the hopes of completing the hardware to software pipeline to the point that this code will not need to be touched for the remainder of the project.

This week I developed more code to get the esp8266 to connect over wifi. The big question at the moment is what “.ino” authentication library to go with for optimal power savings. Theres several options that are available and I have working, the difficulty however is that to truly determine which one is best for our low power solution we have to run them all on the device and measure power draw.

So far I am on schedule as this week we implemented a working embedded device to software interface that communicated with the cloud. This upcoming week + spring break week I plan to not only conduct the previously mentioned power draw experiments but also to establish an AWS rule (essentially the pipeline that directs incoming data) to push inbound data to an AWS db.