Team’s Status Report for 04/12

Risks

  • We are training LSTM/CNN models based off of data polled from sensors.  Retraining the model every request (if the cache fails) can cause lag

  • Home Assistant API latency spikes can delay forecasts. Plotting charts via matplotlib adds overhead (200–300ms)

  • Caching logic failing silently if timestamps are misaligned
  • Infeasible LP results due to overlapping constraints or underforecasted solar

  • LP objective may optimize cost but ignore user experience (e.g., clustering devices)

  • Forecast errors can propagate into LP, yielding poor schedules

Changes

The initial approach used an LSTM network to forecast solar generation based on time-series data from sensor.maya_solar_panel. Main challenge was high latency due to recurrent nature and sequential computation and overfitting on short term trends rather than daily cycles. Transitioned to a 1D Convolutional NN. CNNs capture short-term temporal correlations ( sunrise to peak curve) better.

Progress

  • Successfully polled data from all sensors into backend
  • CNN model produces realistic solar forecasts aligned with production hours

  • Performed testing on API response times

Verification/Validation

  • Used curl requests to profile API timing in the terminal while polling data from sensors

  • Unit tested LP constraints to ensure feasibility and logic correctness
  • Compared cached vs. uncached outputs for CNN model for correctness and stability

  • Compared forecast vs. actual solar power to assess prediction alignment
  • To validate the 15% energy savings, we’ll compare historical baseline energy usage (from fixed schedules) against SmartWatt’s optimized schedules using the same input conditions. Using sensor data (from sensor.maya_fan_power, sensor.maya_solar_panel), we’ll simulate both scenarios over a 1–2 week period and calculate daily energy costs. The optimized case will use CNN-based solar forecasts and LP scheduling. If the average cost reduction exceeds 15% compared to the baseline, the savings target is met.
  • To verify the robust assembly of the house, I will carry the house around and apply reasonable force to ensure it will remain intact during the demo.
  • To validate the integration of the hardware components with the house, I will perform a fit test with the interior components. I will ensure that all feedthrough holes are wide enough and that the components can be neatly and securely placed inside their respective rooms.
  • For validation, we plan to conduct user testing sessions where participants interact with the model house and dashboard to simulate their household energy consumption. This will help us evaluate whether SmartWatt creates a coherent and immersive experience. We will analyze user feedback to determine if the system effectively communicates the intended purpose of the project, the scaled energy usage and passage of time, and feels intuitive to interact with.

 

Anya’s Status Report for 04/12

Accomplishments

  • Integrated real-time sensor polling and displayed it on the dashboard

  •  Developed a responsive FastAPI + HTMX dashboard to display live sensor values

  • Implemented Chart.js charts that auto-refresh every few seconds without full page reloads

  •  Latency about 10s for the API calls, page refreshes every 30s

Progress

Reduced latency by introducing in-memory caching for the a CNN solar forecast endpoint

  •  Forecast results are now cached and refreshed every 10 minutes, eliminating redundant model retraining on every request

  • Slightly behind progress, need to integrate all the charts in the same app with routing
  • Verification

Performed basic verification of the CNN forecasting model to ensure it meets the intended design requirements for solar prediction accuracy.

  • Input Feature Sanity Checks:
    Verified that time-series input includes correct power, sin_hour, and cos_hour features, resampled to 15-minute intervals.
    Resampling was validated via debug logging (len(df), delta distributions).

  • Window + Horizon Coverage:
    Model was trained on sliding windows (e.g., 48 steps = 12 hours) with a 24-step forecast horizon. These parameters were confirmed to provide full diurnal coverage and enough context for trend learning.

  • Envelope Alignment Verification:
    The daylight envelope was implemented and centered around peak solar generation (13:00–14:00) to prevent unrealistic output at midnight. Forecasts before and after masking were compared to confirm correction.

  • Polling Interval Consistency:
    The system pulls data using HA/api/history/period endpoint. I confirmed that sensors such as sensor.maya_solar_panel, sensor.maya_fan_power, sensor.maya_fan_voltage, and sensor.maya_fan_current are polled with sufficient granularity — typically every 5–10 minutes.

  • Each response is parsed to validate:

    • state is numeric and finite

    • last_changed timestamps are increasing and consistent

    • There are no large gaps in time , > 1 hour)

  • Resampling Debug Logs:
    Added internal logging to capture the number of raw and resampled datapoints. This ensures the model always receives enough sequential history for the sliding window.
    Used interpolation (.interpolate()) on the resampled time-series to ensure continuity in data, even if occasional values are missing from Home Assistant.

  • latency tests via time curl http://localhost:5051/api/endpoint to make sure the dashboard refresh updates meet design and usecase requirements

Next Steps

  • Test LP behavior under forecast uncertainty

    • Run LP with low vs. high solar predictions

    • Assess how schedules shift and whether they violate any constraints

  • Stress-test LP under edge-case constraints

    • Zero solar availability

    • Multiple high-power devices requiring overlap

    • Very short vs. very long runtime constraints

Test LP reactivity to live data

  • Feed in updated solar forecast every hour

  • Ensure LP returns updated schedules quickly (<2 sec)

 

Team’s Status Report for 03/29

Risks

  • Backend performance under real-time load
    With multiple device schedules and real-time data polling, system lag or crashes may occur if not optimized.
  • Home Assistant integration 
    Integration with HA might face compatibility or API syncing issues depending on setup.
  • Edge-case handling for devices
    Unresponsive or offline devices could cause unexpected failures if not properly handled

Changes

  • System currently loads price data from CSV files, need to change it to connect to a SQLite database, for storing device state, energy usage, and forecasts is in place
  • Add some more loads such as speakers for an audio component in the model house

Progress

  • Frontend and backend frameworks set up
  • User preferences taken into the consideration when performing constrained optimization
  • Model house for demo set up
  • Sensors integrated into Home Assistant

Anya’s Status Report for 03/29

Work Accomplished

  • Implemented conversational AI interface allowing users to query the system about energy optimization at their home.
  • Created suggestion system that guides users with prompt examples for better engagement with the AI assistant
  • The optimization system was enhanced with a user preferences framework that allows the user to select custom timeslots in which to run devices.
  • The system incorporates earliest start times and latest end times for each scheduled device.
  • Then designed a new constraint-based optimization that respects user preferences while maximizing energy savings
  • A three-tier priority system was implemented:
    • Low Priority: Maximizes energy savings with flexible timing (priority weight: 0.3)
    • Medium Priority: Balances energy savings with user-preferred times (priority weight: 0.5)
    • High Priority: Strictly adheres to user time preferences (priority weight: 0.8)
  • Priority settings directly influence how the optimization algorithm weighs time constraints against cost savings

Progress

Frontend and backend up and ready.

Right now, I need to implement the API routes that will actually trigger device control — switching devices on or off based on user input or automated schedules. Slightly behind schedule with regards to the GANT chart.

These routes will act as the bridge between the UI actions and the actuation layer. Once the endpoints are set up and mapped to the appropriate device control logic, the system will be able to execute real actions, completing the loop from user interaction to physical outcome.

Next Steps

  • Tie each route to the code that interacts with the device (ESP 32GPIO pins/ Home Assistant API).
  • Then test with real devices to validate actual switching.
  • Refine ML or linear programming algorithms that decide when to turn devices on/off.
  • Incorporate feedback loops from usage data.

Anya’s Status Report 03/22

Work Accomplished:

  • Did some testing of a LSTM neural network architecture for time-series energy consumption prediction
  • Tracked self-consumption metrics for solar utilization, peak reduction and cost savings % according to a baseline.
  • The original cost is based on the predicted load with original device schedule. The optimized cost uses the new schedules with shifted loads

Mock backend for schedules : (I need to parse the results of optimization algo and figure out whats a good way to put that into device schedules). This is what the frontend should look like

Current display of predicted vs optimzied power. Need to scale this up accordingly once the power sensors are connected to Home Assistant and download the data from there.

Progress: I would say I am a little behind because a) the power sensor data is not logged in, need to do the integration of the backend with Home Assistant to feed that data, look at transience to analyze average power consumption b) The grid pricing data is fed in through Nordpool which has an integration  with Home Assistant, but for now I am training the LSTM by downloading a csv from Home Assistant rather than having it dynamically via an API (API calls are expensive)


Challenges and Next Steps:

  • Improve load shifting algorithm with better device priority handling [figure out a way to parse info about optimized loads from an aggregate to a device level]
  • Also right now, all devices have the same priority.  Need to assign weights based on device priorities (like running a fridge vs fan, fridge is way more important)
  • Use these weights in the objective function to favor high-priority loads when minimizing cost or peak demand.

  • Add detailed logging for performance metrics to validate optimization results

  • Integration between HA and backend would be just creating a docker container and adding that to HA vs API requests

Team’s Status Report for 03/15

Risks:

Gradient Step Update Issue:

  • In SGD and L-BFGS optimization algorithms, when forecasted energy demand closely matches actual load, the gradient of the cost function is near-zero.
  • This results in very small updates to optimization variables, reducing adaptability and making it harder for the model to improve scheduling decisions.
  • Mitigation: Introduce regularization techniques, perturbations in forecast data (white Gaussian noise)

Misinterpreting Transients as Long-Term Load Changes

Short-term power spikes (when a fan starts) can be mistaken for sustained high energy demand. The optimizer may overcompensate by reducing battery discharge or increasing grid imports unnecessarily.

Changes:

No major changes happened this week. We are thinking of adding a user tracking functionality to improve future recommendations based on past actions to make the system more interactive.

Progress:

  • Optimization Model Debugging: Refined solver selection and fixed SOC linearity issues.
  • Created CAD model of the model house
  • Algorithm Development: Partial success with SGD & L-BFGS, but linear programming constraints need debugging.
  • Recommendation Engine: Classifies insights by priority and impact, with user tracking integration.
  • Frontend: Built an interactive energy visualization dashboard.
    Pending:
  • Hardware Integration: Waiting for power sensor polling to complete before full backend integration.
  • Building the model house in which to house all of our electrical components.

Anya’s Status Report for 03/15

Work Accomplished:

Spent a significant amount of time debugging the optimization constraints and attempting different techniques to resolve DCP rule violations in the optimization model. To resolve it, I made sure the SOC dynamics equation is now linear in the decision variables, added fallback options (within CVXPy library) to try different solvers if the primary ECOS solver fails.

The SGD and L-BFGS-B methods are somewhat working, for the linear programming solvers its thinking the constraints are of conic form so I need to debug that.

Here is the output for SGD


Started working on a recommendations analyzer that processes energy data to identify patterns and anomalies. I developed a classification system that categorizes insights by priority and potential impact. Added functionality to track user actions on recommendations to improve future suggestions

Worked on the recommendations frontend.


For the frontend, I implemented a responsive dashboard that visualizes energy usage, production, and optimization results across devices

Challenges

One of the main challenges in SGD and LGBFS optimization algorithms is that the forecasted energy demand is very close to the actual load.

  • If the forecasted load closely matches the actual load, then the gradient of the cost function is near-zero.
  • This results in very small updates to optimization variables, reducing the ability of the algorithm to adapt or improve scheduling decisions.
  • Mitigation Technique

Progress

I am on track regards to the schedule. Still waiting on the hardware integration to be complete and polling data from the power sensors, which will be fed into a database and processed to be displayed on the frontend.

Next Steps

  • Power sensor data arrives at high frequency (every few seconds or milliseconds). A proper database in SQLite is required to be created for  logging and retrieval.
  • When calculating average power consumption, factor in transients when devices switch on/off
  • Develop interactive dashboards that display key metrics from the optimization algorithm, such as grid usage, battery state of charge, device schedules, and cost calculations.

  • Start working on the integration of the backend with Home Assistant and ESPHome firmware. I have created the clients and API requests (communication protocol library) but actually need to start debugging

Anya’s Status Report for 03/08/25

Work Accomplished :

  • This week, I focused on developing components of the system that interact with Home Assistant and Nordpool electricity pricing data. The primary tasks included designing and implementing the ESPHomeClient, HomeAssistantClient, and NordpoolClient while also working on the frontend and backend integration.
  • Completed the design report (System Architecture, Quantitative Design Requirements, System Implementation, Testing Methodology)
  • Started working on the frontend + backend functionality of the web app


Implemented an ESPHomeClient to enable communication between ESPHome-based IoT devices and Home Assistant. This module allows devices to send and receive sensor data while maintaining real-time connectivity with Home Assistant.

    • Designed a HomeAssistantClient to interact with Home Assistant’s API, enabling data retrieval and control over smart home devices.

    3. NordpoolClient

    • Built a NordpoolClient to fetch electricity prices from Nordpool via Home Assistant.
    • Integrated API calls to retrieve real-time electricity prices and structured the data for easy analysis.

    On the frontend side, I developed and integrated an energy flow chart that visually represents energy distribution and load optimization within the system.



    Progress:

  • On track with regards to the schedule and GANT chart
  • Currently waiting for integration with Raspberry Pi and  power sensors via ESPHome.
  • Once hardware integration is complete, live data from the sensors will replace simulated inputs in the ML and forecasting module

Tasks to complete next week

  • Conduct end-to-end testing of energy data collection, processing, and visualization.
  • Compare original vs. optimized energy loads to assess efficiency improvements.
  • Identify any bottlenecks in data flow between ESPHome, Home Assistant, and the dashboard.
  • Start integrating predictive analytics for energy consumption forecasting via some sort of inference API

Team’s Status Report for 02/22/25

Risks

  • Running ML models on Raspberry Pi could cause performance limitations due to limited computing power. To address this, I will optimize models for efficiency and consider offloading computations to an external server.
  • Additionally, compatibility between ML frameworks and Home Assistant might pose challenges.I will validate API integrations using Postman to address this.

Changes

No major changes in implementation/design yet. After trying to deploy the ML model on RPi, if the RPi has limited storage/processing power, will deploy the ML forecasting on a computer.

Progress

  • Automation setup with Home Assistant.
  • Initial implementation of optimization models.
  • Model training and prediction framework.
  • CSV data storage and Docker integration.
  • Nordpool grid price retrieval setup.

Anya’s Status Report for 02/22/25

Accomplishments this week

Home Assistant Automation Implementation [Integration Platform]

Defined shell commands and set up tasks in config yaml files for data publishing via the HA API

Grid Price Integration from Nordpool

  • Configured retrieval of real-time electricity prices from Nordpool.
  • Integrated Nordpool API into the optimization workflow to factor dynamic pricing.
  • Implemented logic to optimize energy consumption based on real-time grid prices.
  • Instantiated the grid price optimization problem using plp.LP
Challenges & Next Steps
  • Sensor Deployment: I am awaiting final integration of sensors to obtain real-time load consumption data
  • Performance Optimization:  I need to fine-tune the grid pricing model for higher accuracy.
  • Scalability: Need to expand the system to support multiple optimization strategies dynamically.
  • Progress and Schedule
  • Completed foundational tasks and progressing into real-time data collection and optimization refinements.
  • Minor delays in sensor delivery [due to snow storm] could impact the timeline for model training.
  • Action Items for Next Week
    • Deploy sensors and verify real-time data collection.
    • Train models using load consumption data
    • Conduct further tests on the dayahead optimization strategy.
    • Optimize automation workflows in Home Assistant.