Team Status Report 04/27/2024

This week, the team made significant progress on both the web app and machine learning model integration. We addressed several backend issues, including package dependencies and storage for user-uploaded files, and resolved integration challenges. Enhancements to the microgrid visualizer now include a slider for hourly predictions and expanded input fields for user uploads. Despite these advancements, issues with the parser persist and will require further attention.

On the machine learning front, efforts focused on fully integrating forecasting models and enhancing the web app’s interactivity and design. New demo features, such as preset scenarios for different weather conditions and holidays, are being developed to showcase the capabilities of our tool in various environments.

Moving forward, the team will continue refining the front and backend, conduct extensive testing across different browsers and devices, and prepare materials such as a poster, video, and final report. Additional enhancements will be made to the forecasting statistics tab to incorporate more dynamic visualizations.

 

List all unit tests and overall system test carried out for experimentation of the system. List any findings and design changes made from your analysis of test results and other data obtained from the experimentation.

  • Unit Tests
    • Web Application:
      • Test to ensure all forms on the web application validate input correctly and handle errors gracefully.
      • Test the file upload functionality to ensure only the correct file types and sizes are accepted.
    • Optimizer:
      • Test the optimizer with various input ranges to ensure it handles all expected inputs without errors.
      • Test specific algorithms within the optimizer to verify that they return expected results for given inputs.
    • Machine Learning Model:
      • Test the preprocessing pipeline to ensure that data is cleaned and transformed correctly.
      • Test the training process to ensure the model fits without errors and handles overfitting.
  • System Tests
    • Integration Testing:
      • Test the complete flow of data from the web app through the optimizer to the ML model to ensure that data passes through the system correctly and triggers the appropriate actions.
      • Test integration with API to ensure that the system interacts with them as expected.
    • End-to-End Testing:
      • Simulate complete user scenarios from end to end, including logging in, uploading a file, receiving optimization and model predictions, and logging out.
  • Change:
    • To enhance the system’s performance and user experience, several key improvements were made based on testing feedback. The optimizer’s algorithms were refined and additional checks were implemented to ensure robustness. User feedback indicated that the file upload interface was confusing, prompting a redesign to improve usability and provide clearer instructions. Additionally, the machine learning model’s variable accuracy was addressed by adopting a more dynamic training approach to better adapt to data fluctuations throughout the day. Finally, to handle increased user demand and prevent delays, third-party API interactions were optimized and service plans were upgraded as needed.

Yuchen’s Report for 04/27/2024

Accomplishments This Week:

  • Fixed some previous problems such as package dependencies,  backend storage for user input files, etc
  • Solved some integration issues we experienced last week
  • Tested the website using more files
  • Optimized microgrid visualizer with slider for hourly prediction
  • Allowed more input fields while user upload

Challenges Encountered:

  • We’ve solved most problems but the parser issue still persist

Next Steps:

  • Keep improving frontend/backend and test more thoroughly with different browsers and devices.

Yuchen’s Status Report 04/20/2024

Accomplishments This Week:

  • Resized Instance and set up the remote desktop with Alby
  • Optimize Frontend for the upcoming demo
  • Try different display schemes for microgrid visualization (use icons, colors, scaling, etc.)
  • Solved some integration issues we experienced last week

Challenges Encountered:

  • The parser stores previous information and leads to merged file contents, which is caused by how the parser read/hold information

Next Steps:

  • Test the website using more files
  • Solve the ‘two file problem’ of the parser
  • Add minor functionalities like download forecasting results

Individual Question: Since you’ve designed, implemented and debugged your project, what new tools or new knowledge did you find it necessary to learn to be able to accomplish these tasks? What learning strategies did you use to acquire this new knowledge?

  • Deepened understanding of Django for robust web application development, focusing on its ORM, template system, and class-based views for a scalable project architecture.
  • Enhanced skills in JavaScript, especially AJAX, for asynchronous web requests to improve UI responsiveness and data interaction without reloading web pages.
  • Learned about various performance optimization techniques for both the front-end and back-end. For example, I got hands-on practice with Bootstrap Studio.

I engaged in project-based learning supplemented by online courses and tutorials, extensively used documentation and community forums for troubleshooting, and iteratively developed the application with continuous feedback to solidify my learning.

Additionally, the most important lesson I learned is not about new tools but to always leave more time for integration. People use different systems and for our group of three, we used three different systems which caused many unexpected issues. So integrating as early as possible to discern the problems to avoid last-minute issues is important.

 

Yuchen’s Status Report 04/06/2024

Accomplishments This Week:

  • Integrated Forecasting time series charts templates with current ML models of wind, solar, and load
  • Added a template for adjusting the hours we want to show for the hourly output from the optimizer
  • Solved some integration issues we experienced with the git submodule

Challenges Encountered:

  • There are still some integration issues when running SUGAR 3 locally. We suspect that it’s something very specific to Windows. We have decided to enable a remote desktop for our EC2 instance and switch to work on that later on.

Next Steps:

  • Set up a remote desktop locally for us and remotely for the AWS EC2
  • Get the optimizer running on EC2 and start user testing for basic functionalities
  • Add slider/panel for adjusting hourly forecasts

Yuchen’s Status Report 03/30/2024

Accomplishments This Week:

  • Added a side panel to show the optimizer results when the user clicks on the nodes/paths
  • Added time series charts for forecasting outputs display

Challenges Encountered:

  • There are some integration issues when I try to run SUGAR 3 locally. Probably because it’s not very compatible with Windows.

Next Steps:

  • Meet with Alby to solve integration issues. If it turns out that it’s not very compatible with Windows then I will switch to using EC2 for future development.

Team Status Report 03/23/2024

Risks

The team has successfully progressed for each component of the project. However, the ongoing development and integration of the side panel and ML forecasting model could introduce complexity. Additionally, the challenge of overfitting in the load data forecasting model indicates a risk in the accuracy and reliability of ML predictions.

System Changes

Significant system changes include the initialization of an AWS EC2 Ubuntu instance, marking a pivotal shift to a more robust and scalable deployment environment. On the ML front, the refactoring and modularization of the forecasting codebase represent substantial improvements, enabling more efficient code management and model training processes.

Schedule Changes

Currently, everything aligns with the project timeline. Our team plans to begin component integration next week and try our best to meet our deadlines.

Yuchen’s Status Report 03/23/2024

Accomplishments This Week:

  • Finished the Microgrid Visualizer and integrated it with the current webpage
  • Initialized AWS EC2 Ubuntu instance for deploying the website

Challenges Encountered:

  • N/A

Next Steps:

  • Add a side panel to show the optimizer result and a panel display information when the user clicks on the nodes/paths
  • Integrate with current ML forecasting model for demo

Team Status Report 3/16/2024

Risks

  • There is a compatibility risk with the parser integration into the Django web app due to operating system limitations, which necessitates a shift to a Linux environment.
  • There are also some convergence issues in the battery lambda gradient, indicating a risk in the mathematical modeling and computational efficiency of the multi-period solver.

System Changes

  • Plan to transit to an AWS EC2 Ubuntu instance to overcome the compatibility issues encountered with Windows
  • Plan to refine the mathematical equations and introduce specific modifications to renewable PQ loads
  • Shift from linear regression to LSTM models for forecasting introduces a change in the analytical approach.

Schedule Changes

  • The integration of the parser is a bit behind due to the incompatibility

Yuchen’s Status Report 3/15/2024

Accomplishments This Week:

  • Finished the overall frontend of the website
  • Added card-style dashboard page for displaying statistics, forecasting, etc
  • Added profile, login, and upload file pages

Challenges Encountered:

  • When integrating the parser with the Django web app, there was something incompatible with Windows, and turns out that the parser needs to be set up in a Linux environment.

Next Steps:

  • Initialize AWS EC2 Ubuntu instance for deploying the parser
  • Integrate the updated frontend and the parser code into current Django web app
  • Start to integrate ML models into web app

Yuchen’s Status Report 03.07.2024

Accomplishments This Week:

  • Users can now specify a zip code and upload customized GridLAB-D files
  • Finished most parts of power flow and dynamic load visualizations. Each node in the microgrid will display information such as the node type, scale, power flow direction, distance, and node paths.

Challenges Encountered:

  • There were some issues with setting up the parser so the next step is to integrate it with the parser to finish the overall pipeline. It’s a minor issue and will be fixed next time we meet.

Next Steps:

  • Moving forward, we aim to fix the parser integration, finalize the pipeline, and ensure its functionality with various file types and sizes.
  • Meet with team members and discuss user registration and logins necessary for our apps, figuring out whether we should store users’ inputs into the database or run everything locally based on our needs.