Category: Status Reports

There are no serious risks for the week, I think technically our project is going along smoothly. Nadine is finishing up her hole drilling, and integration between the CV/local and web client with Jonathan and Aria and integration between the local and RPi with Aria and Nadine are both going relatively smoothly. The main risk now is just being able to get latency down for the overall project, and also to reduce any issues with our actual demo. The demo on Wednesday went off well, and the technical aspects were ok. Getting a cohesive storyline should be doable as well. We should be able to finish up all integration hopefully by this coming week, and then start working on the report and demo early. This risk is largely mitigated because our initial report was pretty comprehensive, and the implementation mostly went off according to the report (except the forking of threads for CV), and so we do not anticipate having to really change our report too much.

The system has not changed from last week, the schedule has not changed yet either. We are excited to be in the last phase of our project 🙂 This week we will focus mainly on more integration, Jonathan will be meeting Nadine hopefully end of the week when the board is fully drilled to do a full testing, this would involve testing for sensitivity for the LEDs, and figuring out the best way to reflect invalid moves or invalid turns to the user, as well as doing full timed testing.

This week, I had 4 main tasks: Optimization for speed, Dealing with skew, Integration with sockets and Getting a camera stand.

For the camera stand, this is a newly added task after getting feedback about the demo setup, where we want a mount for the webcam so that it can point downwards. My current setup literally tapes the webcam to a piece of plank that is protruding from a table. The main difficulty here is that a webcam ordinarily would not need such a functionality, and the mounts that do do not seem high enough for our needs. I am looking at this and this currently. The problem with both honestly is thinking long term in terms of the demo. The reason why the current setup is sort of ideal, is because of the fact that I can easily just add an additional phone or something on top of it to provide a live video feed for the demo, which is something that would be nice to show the full storyboard/flow of the project. The first clamp is also a bit worrying because it seems the most ideal for getting the downward angle, but it is a ball clamp and I am not sure if the head is removable. Regardless, I will discuss with the team by this week to make a decision on it.

Integration with sockets took longer than expected, because Aria is using sockets, while I am on windows and it is not supported. Transitioning to winsock took a bit longer than expected due to a couple of transitioning issues, but it seems to be ok now. We will test it when we meet up this week.

Dealing with skew is done fully as well this time. This part of the code is basically an additional check on top of the code to see if the skew is more than 20 pixels from corner to corner, and if so to correct for it when scanning for pixels for correctness. This was done early in the week, and was just finishing up of what was mostly already done last week, and to deal with more obvious skew issues.

Optimization for speed is the last main task for the week, and is what I spent the most time on this week. I have tried many different ways to reduce the latency for the CV component. I think it makes sense to figure out where the latency is in the first place. My starting latency was at 280ms, which is higher than expected, and the main issues were due to doing 2 different full scans of the image. Basically a first overlay is required to figure out the black squares to detect the corners, and then a second overlay is required to figure out for each grid cell if it is of a certain color or not. Some early easy wins were reducing the number of pixels checks for each grid cell for correctness, and writing the output vector directly when figuring out the second overlay instead of doing a third pass. However, further optimizations proved tricky, as they tended to start causing some correctness issues. My final latency for the week is at 230ms, which is still very high. Will work further to bring it down next week.

Change of plans

I am again on track to finish on time, I am mostly done with the rough product, it is entirely working already and if the demo was this week it could work, but we are now in the refinement and optimization phase for my portion of the project 🙂

Deliverables for next week

I hope to cut down latency further, at least for my portion of the code. I will also be meeting Nadine for the next stage of integration.

Risks:

The construction of the board is more tedious than expected. We changed our approach so now we are drilling holes in each of the tiles of the Blokus board. The process is taking some time as there are 400 squares to measure and hand drill. The code to control the LEDs is currently still being debugged. We are currently solving this so that integration between the client software and the hardware can begin soon.

UPDATE: Nadine finished the LED Arduino code and it works. However, we seem to have encountered our dead pixel (17, 19) which will have to require soldering to replace.

Changes:

The LED game board has undergone some design changes. Since the LEDs got too hot resting right below the board we are now drilling holes in each tile of the board.

Overall progress:

Integration is done for the CV and the game software, and integration between the game software and the hardware is going to begin asap.

Schedule updates:

We have updated our Gantt chart.

The five main tasks I had to do this week were avoiding detection of external objects, dealing with skewed images, optimization for speed, testing for speed and integration.

Detection of external objects is done, basically I accounted for changes and large blobs, and ignored the frames if that was the case.

Dealing with skewed objects is partially done, so what I have done is taken an average of the corners for now, so that helps to alleviate slight skewing. I have not done any transformation of the given image yet, but I will work on that more after the demo.

Integration was the most major part for this week. I met up with Aria and did integration for the system, we are done integrating the local server side, and just have the web server left to complete.

Speed testing is a recurring part of the project, we are now pushing the boundaries slightly and taking about 180ms, and I will focus on trying to push that number down in the following weeks. One slight change I did was to detect the 4 corners first and if undetectable to skip the frame entirely, potentially saving time between invalid frames.

This week I had 3 main tasks: camera setup, camera integration with code, and camera testing on videos. Camera setup was pretty quick. Camera integration with code went smoothly as well. Initial timing analysis went perfectly, and in fact surprisingly showed that from video stream input to output takes only 0.055s, which is about 55ms, which is quick enough that we probably will not need to fork off child threads even. There is also a lot of optimization that can be done still, so I am optimistic that this time can be trimmed further. This would save the time required to fork and join threads as well, which is pretty sweet. Camera testing on videos is going well as well, the major obstacle that I faced this week was that the lighting in my room is not ideal, and so the pieces have portions that look black in the image. I am artifically using camera flash to help with this for now, and I can get a good processing result thus far.

My current setup is pretty makeshift, with a wooden plank hovering over a table taped to the webcam, but it works nicely so far. I have also managed to deal with slight skew in my image, by using all 4 corners for detection instead of just two. One other bug that came up was that the black rectangles identified in the image appears in different orders when restarting the webcam, and so I had to account for that by manually searching for which corresponded to which corners.

Hand detection came out to be easier than expected, because what a hand basically does is introduce a giant rectangle to the board state. I simply checked if there were not 4 black portions, and skipped to the next state if so. This was very robust, and I can add and remove pieces and the entire system responds well currently. 🙂

Next week is integration week! I will be working with Aria closely to merge our code nicely so that we can get everything in. I will then work on optimization and trying to make the code more robust.

Risks:

The LED matrix board is under still under construction, but we are working quickly to complete it today and tomorrow so that that some preliminary testing can be done.

Changes:

There were some minor changes in with regards to the client software (allowing use of OpenGL GUI) which was necessary for listening for pieces sent from the server. This change does not impact aspects of the CV or LED game board.

Overall progress:

We are finalizing our individual parts of the project and will be starting integration next week. Integration will start with sending coordinates from CV software to the game client software and also sending coordinates from the game client software to the microcontroller.

Schedule updates:

We have updated our Gantt chart. Schedule changes have been made to allocate time for LED matrix construction next week. We have added more specific tasks under web communications and allocated times for them. Additionally, we have hidden completed tasks.