Posted on

Team Status Report for 2.26.2022

We delivered our design review presentation on Wednesday. Overall, it went pretty well. Creating the presentation gave us a lot of insight into what the next few weeks are going to look like, and we discussed our schedule more in-depth after the presentation to cover ground in areas that we still need to work on.

The PCB board is being manufactured and will be shipped next week, and we’re currently testing individual sensors to see how they behave. Additionally, we decided on using MQTT communication since it was the most common IoT communication used, especially with the Photon Particle. We began testing the sensors and developed a visual interface using JavaScript and HTML to see how the sensors detect objects close by. For the most part, we’re able to tell what sensors are going off when a finger comes close by, and we can also almost guess which direction we are swiping. This is a good indication that our rotation gesture recognition algorithm may actually work.

We also started the web application portion of our project. We have figured out how to embed the Unity application onto our Django web app and communicate data between our Web application and Unity.

Together, we worked on brainstorming how the web interface will look like and how it will be implemented. Joanne will primarily be working on getting a demo of that working soon. Edward will test out the PCBs once they are shipped. Anushka will work on getting the Jetson to boot and run code. We seem to be on schedule this week.

An issue that arouse was that the VL6180X sensor can read data in a 25 degree cone, and when a finger gets too far away from the sensors (75-150mm range), all the sensors will say that they see an object. We have no idea how this will look like exactly, so we created a visualization to see what would happen if a finger gets too far away. Would the finger look like a line? Would it have noticeable dip? We will need to explore this once the PCB is fully done and verified, which might take a while and put us behind schedule.

Posted on

Anushka’s Status Report for 2/26

This week was heavily design focus. I was responsible for giving the Design Review presentation, so I spent most of last weekend researching and getting the presentation together. It was very helpful doing this presentation because it helped show me what parts of the project we have more planned than others. For example, the “middleware” part of the project, i.e. the Jetson and the communication part, is where we have the most questions surrounding the specs and how/if it’ll fit in the final scope of our project. However, we have the most information on the sensors, which we have already ordered. I have a clearer understanding of the direction of our capstone and what areas I need to explore more.

Speaking of areas I need to learn more about, I continued tinkering with the Jetson this week. It’s still behaving very spottedly, and I predict it has something to do with how I downloaded the image on the SD card. As mentioned before, this is the weakest part of our project, so I need to spend more time on next week. My goal before Spring Break is to get it set up, collect data from the sensors, and send information to the web application. It’s a lot, but if I make this my sole focus, it’ll be good for the project as a whole after we come back.

We ordered individual sensors to test out while we wait for the PCB board to be manufactured. We began testing it and decided to create a visual interface of it. The idea that I had was that the yellow dots represent the sensors and the blue dot represents the objects that are being detected.

I originally wanted to create this web application in Django, so I told Edward to code the MQTT communication in Python. However, since the CSS of the “objects” need to be changed, and the only time it would update is when the server would refresh, we decided to change our web application to be all on the client side. I asked Edward to change the code into JavaScript, and then we were able to get an almost instantaneous rendering of the sensor data. A demo is attached to the link below.

https://drive.google.com/file/d/1w0-5nBngHThfDe-A_Iem-NNt1k3NcuCe/view?usp=sharing

For the next week, I will focus on the Jetson and finishing the design review paper. If I have some time after these two, I will finetune our visualization so that it can be used as a dashboard to measure the latency and accuracy of the sensors. However, this may become a Spring Break or after Spring Break issue.

As mentioned before, the Jetson is the weakest part and the greatest risk. Although we have a contingency plan if the Jetson doesn’t work, I don’t see a reason why we have to remove it at the moment. I will reach out to my teammates and the professors on Monday so that we can get over this hump. We’re still pretty much on schedule, since we are getting a lot of work done before the actual sensors get shipped.

Posted on

Team Status Report for 2.19.2022

This week as a group we primarily discussed future design goals. Some major topics that we focused on was how we would write the algorithm for the gesture recognition, more precise testing measures, a more in depth flow chart of our device. More design details that we discussed will be on our design presentation slides. Afterwards we worked on the design presentation slides. We also got samples of the sensor that will be on our custom PCB so we can start working on getting sensor data to the Jetson and then to our web application this week. We placed the order for our custom PCB that Edward designed this week as well.

Individually we worked on 3 different areas of our project: the PCB board to collect sensor data, the Jetson Nano, and the 3d modeling interface (Unity Web application).

Joanne worked on getting the Unity section. She scripted one of the three gestures the model should perform when a user inputs our target gesture. We got the rotation gesture (which is the swiping gesture from user) to work. The translations work well when projected onto the holograph pyramid as well. She will be working on finishing up the other two gestures (zooming in and out) this week as well.

Edward worked on finalizing the PCB and testing out the VL6180 distance sensors. He made a small testbench using three breakout boards we bought. The sensors seem fairly sensitive and a pretty fit for our use case. They detect up to around 150-200mm, which fits on a human forearm. Honestly, maybe with some clever math, we can detect finger movements with just three sensors and might not need all 10 sensors. So if the PCB ends up not working out, it might be ok to just use the breakout boards? Will need to explore this further next week.

Anushka worked on setting up the Jetson Nano and looking into communication protocols to implement. This will be helpful later this week once we work on sensor communication with our test sensors. She will be using this information to guide the web application interface throughout this week. She will also be presenting the design review at the end of the week.

The schedule has moved around a lot since last week, and we’ve updated our current Gantt chart to reflect that, as seen below. The team will meet next week to discuss these updates.

Schedule Updates

We haven’t made any major design changes since last week, but we did flesh out what we want the user experience to look like. We will be discussing these during the design review this week. The most significant risk is how the board will work, but this is why we ordered it now so that we have some time if we have to make any changes to the PCB. We will test as soon as the PCB comes, but we purchased single sensors in the meantime so that we can test the general behavior of the distance sensors.

Posted on

Anushka’s Status Report for February 19

This week, my focus was on a myriad of tasks rather than focusing on a specific dimension. This was prompted by our earlier purvey into the hologram pyramids; even though we aren’t scheduled to work on it until much later, I wanted to look into web application interface and the Jetson communication.

With a quick look into the holograms last week, after discussions with the professor and the TA, we are considering making the hologram pyramid bigger so that there is more surface area. I had an idea of cutting square pieces of acrylic, decreasing the angles of the trapezoid, and testing.

Looking into the Pepper’s Ghost illusion, the original use of it was in magic show’s. The original projections are similar to the ones in the diagram shown below. That projection is horizontal, and there is only one glass available. I may brainstorm ideas on how to execute this, since most people have access to projectors and can easily replicate this instead of needing to use a big screen and a pyramid.

​​

Image from Wikipedia: https://en.wikipedia.org/wiki/Pepper%27s_ghost#/media/File:Peppers_ghost_low_angle.jpg

I also began looking into Jetson. Setting it up is similar to setting a Raspberry Pi, but I was having some trouble formatting the SD card. I also began researching different communication protocols, including MQTT. I want to follow this tutorial once I set up the Jetson to see if we can use it in our final project: https://www.youtube.com/watch?v=WPmzoYwXj00&ab_channel=AshutoshMohanty.

Image of Jetson

We have a design presentation due next week, and I volunteered to present this time. I want to focus on making our presentation more pitch-focused, so I began storyboarding how I imagine the user will be using it. Some screenshots are below. For the presentation, I also worked on a few slides on the wrist watch CAD file and the gesture algorithm, which can be seen under the Design Review tab above.

Screenshots of the user experience

Since we are moving around the different parts of the schedule, I don’t believe I am behind. If anything, starting on these parts means that we are ahead, but I have to continue working on them in order to continue to hold this position. Over the next week, I want to do more work on the Jetson, since this will be the most relevant once the sensors are completed. Since we ordered some sensors to test with, we can start with communication at the end of next week with the Jetson, so I’ll make that my deadline.

Posted on

Joanne’s Status Report for 2.19.2022

This week I worked more on the Unity portion of our project. I am working on mimicking the translations that would be applied to the 3d model after a user produces a gesture (i.e. zooming in/out, swiping). I initially wrote script to get 3d object to move on its x,y,z coordinates when a user does specific key presses. This was to ensure that the object responds to the key presses and that the projected image also translates well. Next I got the swiping motion which translates to rotation of the object to work. The script currently takes x,y coordinates from the mouse cursor location and will rotate the 3d object. I am planning to work on the last gesture (zooming in/out) this week. I have been discussing with Anushka (who is working on the jetson nano part) about how the data should be serialized (i.e. what kind of data should be sent and in which format) when sent from the Jetson to the web application.

I worked on some portions of the design slides and talked as a group about our future design plans.

Posted on

Team Status Report for 2.12.2022

Welcome to our project website! We’re so excited to share everything we are doing.

This week we focused on initial designs of different components. We focused on the PCB boards, Unity, and hologram pyramid. For the PCB board, we designed and are ready to order, but we also decided to place an order for two sensor breakout boards so that we can start testing our measurements as soon as possible since that order will arrive most likely before our boards do. The Unity environment was intuitive to work with, and we were able to set up an environment that we needed for the hologram quickly. We created the layout necessary to project our 3d model onto our hologram pyramid. However, our biggest challenge this week was with the results of the hologram pyramid. 

After tinkering with the hologram, we are concerned about the scope of the component. We aren’t satisfied with the projection of the 3D image and we are considering either improving it or replacing it with another medium altogether. We have placed a demo video link of the hologram pyramid: https://drive.google.com/file/d/1ip7AXfh7wN9jhmTKIRnzeADvgepjKqEY/view?usp=sharing. We will make a decision on what to do after our testing this week. Since we are working on this component earlier than planned, we have enough time to make modifications and it will not serve as a risk to our project at the moment. This may change our requirements, but we cannot make that decision at this time. 

Since we were prototyping with multiple pyramids, we had to incur the cost of the plastic. It isn’t a huge cost, since our overall projected budget is low, but we do not want to spend too much just cutting new pyramids after every modification. We will try to do more research ahead of time on what dimensions serve best before cutting more pyramids.

 

Next week, we want to accomplish several things, and below is our modified schedule. We decided that we’re going to parallely work on parts to prevent us from scrambling at the end.

  1. We’re going to be working on improving the hologram and seeing if we can get the image to be continuous across an axis instead of disjoint.
  2. We’re going to be placing the PCB order as soon as we get it approved.
  3. We’ll continue tinkering with Unity and exploring how external inputs can be applied to objects.
  4. We are planning to try out a different size design for the hologram pyramid to see if it improves the quality of 3d model viewing. 

Modified Gantt Chart with this week’s tasks and progress

Posted on

Joanne’s Status Report for 2.12.2022

This past week I worked on making the proposal slides with my team and I presented it during the proposal review. We got back feedback from our classmates and TA’s. One of the feedback was thinking about how we were going to implement the hologram visual portion of our project.

Thus for the latter part of the project, I worked on Unity to create the layout of the hologram. In order to project a hologram onto our plexiglass pyramid, we needed to place four perspectives of a 3d model around the base of the plexiglass pyramid. My main responsibility was to use Unity to create the four perspective view of a random 3 model and then export it to the web using Webgl. I placed a photo of the unity scene when it is running below. The white lines are drawn in by me to illustrate how the pyramid would be placed in respect to this view.

I exported the Unity scene onto a web browser so we can place the pyramid on top of my ipad (which holds now the link to the unity scene with the four views of a model). Edward and I helped Anushka in laser cutting the pyramid plexiglass she designed. An image of a basic hologram model that we completed is on our team status report page for this week.

Posted on

Anushka’s Status Report for February 12

This week was our first week working directly on the project. We spent the first half preparing for our proposal presentation. We discussed bringing in a hologram component into our project to fulfill our use case further in providing a novel experience to interact with 3D models. I contributed information on the slides related to this new addition and began coming up with requirements for this component. One of my main goals this week was to improve our explanation of our use cases; our TA gave good advice on thinking of our proposal as a business pitch, so I wanted to incorporate as many applications of our project in the presentation.

Going further on this, I began researching hologram pyramids. They are easy to create from everyday plastic items, but we decided to look into clear acrylic sheets in our first round of prototyping. I visited IDeATe and found a piece of acrylic, but by the time we wanted to use it, IDeATe was closed. I began by creating a model on SolidWorks and cutting it out as shown below.

Model of Initial Pyramid

Trial 1 Demo

However, this was a naive approach, as I didn’t consider any proportions of the sides or the angles involved. As a team, we found that the Pepper’s Ghost pyramid is a more accurate way to create the hologram, and I quickly remodeled and cut the plastic. Unfortunately, we still couldn’t achieve the desired outcome.

Trial 2 Demo

Even though the hologram becomes more relevant at the end of the project, it’s important that we start looking into it now. I wouldn’t say that I’m behind, but I do think that this does not require a lot of time and can be finished over the next two weeks. Over the next week, I hope to do more research into holographic illusions and see if there are either improvements or alternatives.