Progress
Implemented video streaming from pi to unity on pc, which included writing a script in Python on the pi to connect to the pi matchmaking server and set up a socket server for the video stream, and finishing up the source filter for the video stream to Unity. Finding a suitable library for h.264 video decompression in C++ for the source filter was difficult, but luckily OpenCV has a high level VideoCapture class that can produce frames from a network stream, so I used that instead of the OpenH264 library.
For the optimization portion, I tried tuning various options on the pi and camera side, including resolution, quality, and bitrate, and on the pc side I tried various compiler options in Unity and different ways to process frames in the source filter. However, the best results I could achieve were 10 fps at 1080p, 24 fps at 720p, and 30 fps at 576p. Using Visual Studio’s profiler showed that the ffmpeg functions used in OpenCV were the bottleneck, so it seems like these are the best resolution/framerates we can achieve while maintaining a reasonable quality.
Schedule
This week I fell a bit behind schedule by not finishing the video streaming optimization because I have two labs due in other classes early next week. However, Bujji was supposed to help me with video streaming but instead took my Xbox controller task from me to better prepare us for the midpoint demo, so I can push my other tasks back a few days.
Deliverables for next week
Video stream latency measurement and optimization, race logic and communication.