{"id":125,"date":"2019-03-30T18:11:53","date_gmt":"2019-03-30T18:11:53","guid":{"rendered":"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama9\/?p=125"},"modified":"2019-03-30T18:11:53","modified_gmt":"2019-03-30T18:11:53","slug":"davids-status-report-week-6","status":"publish","type":"post","link":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama9\/2019\/03\/30\/davids-status-report-week-6\/","title":{"rendered":"David&#8217;s Status Report: Week 6"},"content":{"rendered":"<div class=\"inner-wrap\">\n<div id=\"primary\">\n<div id=\"content\" class=\"clearfix\">\n<article id=\"post-112\" class=\"post-112 post type-post status-publish format-standard hentry category-davids-reports\">\n<div class=\"entry-content clearfix\">\n<p><strong>Progress<\/strong><\/p>\n<p>This week I worked on tasks related to preparing for the interim demo on Wednesday, namely, the video stream latency optimization and controller integration.<\/p><\/div>\n<div class=\"entry-content clearfix\">For the latency optimization, the first thing I did was figure out how to measure end-to-end latency by displaying a timestamp on the laptop screen, recording the screen with the camera, and checking the difference in timestamps. After that, I began my optimization attempts<\/div>\n<ul>\n<li>PC side optimization attempts\n<ul>\n<li>Use a Unity WebcamTexture instead of Vuforia &#8211; 140ms faster<\/li>\n<li>Use async functions or threads to get frames from OpenCV &#8211; no noticeable change since the bottleneck is the library&#8217;s processing of frames<\/li>\n<\/ul>\n<\/li>\n<li>Pi side optimization attempts\n<ul>\n<li>Changing camera framerate to 30\/40\/60 &#8211; lowest latency (10-20ms faster) with 40fps, the fastest rendering rate on the PC<\/li>\n<li>Use raspivid and netcat as video server instead of picamera Python library &#8211; ~10ms faster but fewer config options<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>The final result was a latency of 200ms. Unfortunately, this might be the best latency we can achieve with OpenCV\/ffmpeg. An experiment I ran with a few minimal lines of OpenCV code that displayed frames as fast as possible still took 170ms to decode and display frames, and Unity is a little slower because it has to do other work.<\/p>\n<p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-medium wp-image-126\" src=\"http:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama9\/wp-content\/uploads\/sites\/24\/2019\/03\/unity_latency40hf-300x169.png\" alt=\"\" width=\"300\" height=\"169\" srcset=\"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama9\/wp-content\/uploads\/sites\/24\/2019\/03\/unity_latency40hf-300x169.png 300w, https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama9\/wp-content\/uploads\/sites\/24\/2019\/03\/unity_latency40hf-768x432.png 768w, https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama9\/wp-content\/uploads\/sites\/24\/2019\/03\/unity_latency40hf-1024x576.png 1024w, https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama9\/wp-content\/uploads\/sites\/24\/2019\/03\/unity_latency40hf.png 1920w\" sizes=\"(max-width: 300px) 100vw, 300px\" \/><\/p>\n<p>The other component I worked on was setting up the Xbox controllers in Unity and writing code on the PC to connect to Pis and stream controller inputs and data. We decided that Unity&#8217;s native support for Xbox controllers was enough for our purposes, so we didn&#8217;t use the XInput.NET library. For communication between the PC and the Pis, I used the .Net frameworks&#8217; TcpClient and NetworkStream.<\/p>\n<p><strong>Schedule<\/strong><\/p>\n<p>After receiving more information about the interim demo, we all shifted our tasks around to have a more demoable product by next week. I prioritized the video feed and control for a single car, and didn&#8217;t work on the race logic, but I should still be on track because communication with the Pi was brought forward and the game was pushed back.<\/p>\n<p><strong>Deliverables for next week<\/strong><\/p>\n<p>Game logic and communication. Should be able to walk through a race when the methods are called on the three PCs.<\/p>\n<\/article>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Progress This week I worked on tasks related to preparing for the interim demo on Wednesday, namely, the video stream latency optimization and controller integration. For the latency optimization, the first thing I did was figure out how to measure<\/p>\n","protected":false},"author":70,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[3],"tags":[],"_links":{"self":[{"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama9\/wp-json\/wp\/v2\/posts\/125"}],"collection":[{"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama9\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama9\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama9\/wp-json\/wp\/v2\/users\/70"}],"replies":[{"embeddable":true,"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama9\/wp-json\/wp\/v2\/comments?post=125"}],"version-history":[{"count":1,"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama9\/wp-json\/wp\/v2\/posts\/125\/revisions"}],"predecessor-version":[{"id":127,"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama9\/wp-json\/wp\/v2\/posts\/125\/revisions\/127"}],"wp:attachment":[{"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama9\/wp-json\/wp\/v2\/media?parent=125"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama9\/wp-json\/wp\/v2\/categories?post=125"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/course.ece.cmu.edu\/~ece500\/projects\/s19-teama9\/wp-json\/wp\/v2\/tags?post=125"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}