Posted on

Mitul’s Status Report for 3/26

This week I continued working on the server architecture local implementation. I was able to get a working version of the multi-node data flow that can be redirected at different stages. I do this by having a specific node (in this case, the front end node) relay information to one of two back end/database nodes in order to begin a new data flow. In Node.js, it was even possible to have a fragmented video stream in which the data is sent in small chunks of about 1 MB that are relayed from database to web server to browser. There remains some debugging to do in the case of scrubbing (moving the current time instance to another).

In working on this locally deployed application, I noticed that for our intended architecture, there is exceedingly little additional functionality for our back end server tier when compared to our database tier. On that end, I observed that our intended virtual machine specification for the database nodes will likely the demands for both back-end and video storage functionality at once. Thus, I proposed to the group that we condense the back end and database server tiers into one comprehensive database access tier the idea was well liked. This change would streamline our server for greater speeds and reduced deployment costs.

Posted on

Team Status Report for 3/19

We met as a team several times to better detail our respective programming roles for the application as well as our integration plan. On implementation, we narrowed down and began work with NodeJS for our web framework, React.js for front-end served content, and Django Python for user simulation. We also worked together on some initial debugging of some aspects of the application that differ from tutorial bases.

We ran into some design concerns regarding the role of the front-end tier for the video data stream that goes from database to user (whether it is part of the chain or is circumvented). For now, we will keep it part of the chain as this will keep our data flow more regularized between nodes. However, we recognize a potential to change our VM choice for frontend nodes during the deployment stage accordingly. If front-end nodes handle a comparable load to the back-end, they may need to better match the capabilities of the backend.

Over the next two weeks, we plan to integrate our application-side work. We expect the minimum viable product will be a sufficiently complete environment to start implementing and creating testing modules for the traditional load balancing algorithm nodes.

Posted on

Mitul’s Status Report for 3/19

Over the past three weeks, I worked on an implementation prototype for our server architecture. I researched and found more implementation tutorial content, in both video and website form, on NodeJS over Tornado to use as a web framework technology. I presented this info to the team and we collectively agreed to switch our intended stack to NodeJS for convenience.

While Nakul worked on the high-level aspects of the application (e.g. the video player), I delved into the internal structure of how our server architecture can send and receive data. I learned about Express, NodeJS’s internal back-end framework, which was useful in creating data streams that can send small pieces of data in sequence. I experimented with this feature to create simplistic data streams across nodes that can be rerouted manually. I am now working on a smoother process for ensuring the correct data can be sent to specific nodes.

After that Nakul and I will work on integrating our current work into a unified codebase. We expect to create an application that can deliver a functional, locally deployable multi-tier video stream. Though the tiers will likely be nominal, featuring roughly two nodes within each, they can show a proof of concept for the load balancing task. The small app can function as an early, local test for our traditional and more easily implemented load balancers. The data stream could be rerouted between the few nodes randomly or in order (RandomLB and RoundRobin respectively).

Posted on

Team Status Report for 2/26

This week we met up as a group twice to discuss some additional potential risks in our project. In particular, we discussed some external advice regarding the pitfalls of video streaming servers and how they may not work well our accustomed technology suite (Django). Therefore we looked into alternatives for both the application area as well as overarching web server frameworks that can be good for us.

Over the next week, we plan to consolidate that understanding into our initial AWS deployment and connectivity tests. Furthermore we will complete the majority of the design review report that was started this week. We expect to have a tentative deeper technology suite for the overall project in the design report; we will adjust that plan with further testing.

Posted on

Mitul’s Status Report for 2/26

This week we got our AWS credits so we are ready to start initial deployment and connectivity tests for the following week. In preparation for said tests, I watched some tutorial videos and read some articles regarding multi-tier server deployment on AWS. In particular, I found this technology suite regarding AWS article insightful: https://aws.amazon.com/vpc/ . Virtual private cloud would allow for various high level tasks such as creating and integrating new nodes as well as tiers for said nodes; this would effectively give us centralized control and some preliminary monitoring for our overall server architecture.

I also did some further research on our possible database implementation. AWS offers specialized node instances for database storage and our naive solution (for the MVP) is to have several nodes that each store our full database archive. We plan to make the video catalogs on the databases all read-only during active deployment as we find this will make any distributed system more straightforward to implement. We will consider a more dynamic database solution if this leads to storage overruns during later testing.

I also created a shared document for the design report imported our presentation content to act as a starting template. We will continue to work on this report for the upcoming week.

Posted on

Mitul’s Status Report for 2/19

This week I spent most of my time working on the design presentation slides. This required adding furthering specification and numerical detail to our design plan. I specified our target problem being the lack of publicly available knowledge on load-balancing algorithms and their relative performance. Though enterprise solutions are widely available, the inner programming is not visible to users, so a stronger public knowledge can support more open-source software building and research. Our solution plan is to test many load balancing algorithms for insights on how they perform on the real-world scenario of video streaming.

I also further narrowed down the four load balancing algorithms we plan to test for our minimum viable product. We would have two purely parameter-based decision makers and two machine learning models. The parameter-based options will take into account either processor utilization or relative input/output volumes. The first machine model will be a low parameter unsupervised learner that follows the UCB1 algorithm based on local response time (https://jeremykun.com/2013/10/28/optimism-in-the-face-of-uncertainty-the-ucb1-algorithm/). The second is not yet finalized but will take into account request sizes as well as server loads, inspired by the insights of the following paper: https://hal.archives-ouvertes.fr/hal-02570549/document

I also standardized more of our final testing environment (several geographically different AWS EC2 instances) and how our internal server metrics will be calculated. There are further specifics on the testing end that we plan to experiment with once we receive our AWS credits for server connectivity trials.

Posted on

Mitul’s Status Report for 2/12

This week I spent some time preparing for the proposal presentation I gave on Wednesday (going over the slides, practicing speaking points, and monitoring the time).

I read more about the standard AWS multi-tier server building process and found that an interactive suite can build most of the architecture at a high level except for the LBs. AWS’s LBs are not directly programmable so we will need to replace those with separate ones that will be programmed to EC2 instances.  Thus, I applied for AWS credits at the start of the week as we plan to test the multi-tier architecture’s server connections with the separate LBs as soon as possible. The building process is detailed in this link: https://medium.com/the-andela-way/designing-a-three-tier-architecture-in-aws-e5c24671f124

I also did some further research on how a load balancer’s performance can be analyzed. The paper linked below was of particular interest: https://www.ijarcce.com/upload/2013/october/62-O-priyesh_kanungo_-MEARURING_PERFORMANCE.pdf

The metrics of mean response time and mean slowdown are of particular interest since the parameters can be easily obtained from LB-adjacent nodes and they are consistently insightful regardless of the situation. On the other hand, processer utilization seems only useful in low load server situations but still worth considering for enhancing analysis.