Blog posts

Frame Rending in Virtual Reality is Expensive on Mobile Devices

Following three research papers show that the key bottleneck in un-tethered VR experience on mobile devices is Rendering a Frame. And here is how they try to tackle this problem. These three papers are a nice series of work on practical VR experience on mobile devices.

FlashBack: Mobisys’2016

FlashBack is one of the first papers that studies the impact of running the VR applications on mobile devices. The authors say that if frame rendering is very compute intensive and the mobile cheap hardware does not meet this compute requirements. Instead, they argue that there is lot of storage often underutilized on mobile devices, and this storage can be used to memoize the pre-rendered images. They show the effectiveness of their method by pre-rendering all the possible orientations of images for Viking Village VR application offline and by storing in three tiered hierarchical cache (GPU memory, RAM, SSD).

Comments: Very specific to one VR application. Does not scale well.

Furion: Mobicom’2017

Furion is a fantastic paper that investigates how the current mobile devices can satisfy the rendering requirements of VR applications, in cooperation with a remote rendering server. The authors also argue that future mobile hardware alone is not going to satisfy the rendering compute requirements because the mobile device enhancements are almost saturated. Further, they show that the wireless networks are not capable if all the rendering is placed on a remote server and stream the raw frame data over the wireless link (A raw frame requires a data rate of tens of Gbps network which is way far from the existing and upcoming future future wireless technologies such as 802.11ad, ay or ax).

Instead, Furion characterizes different VR applications and figures out a way to split the rendering into foreground and background activities. They show that the foreground activities are light weight in load and are not predictable, and hence they can be placed at the mobile device. Whereas the background activities are heavy weight in rendering load and predictable, and hence they can be placed at a remote server. The remote server also does some sort of compression to optimize the data over wireless links, and the mobile device decodes the data and displays.

Cutting the Cord: Mobisys’2018

This paper follows Furion by offloading complete rendering computation to a remote server. The authors argue that the mobile device display rate should be in sync with the remote server so as to avoid frames missing and unnecessary frame display delays. They propose a parallel rendering and streaming pipeline where the remote server first renders the left eye image of VR application and starts streaming (encode, transfer, and decode) it while simultaneously rendering the right eye image. Further, it proposes remote Vsync driven rendering approach to synchronize the remote server frame rendering and mobile device display rate.


EnvironmentError: IOError: usb rx6 transfer status: LIBUSB_TRANSFER_OVERFLOW

If you are facing the following problem, that means the FPGA tried to load too much data into the DMA FIFO in the Cypress FX3 (USB chip) and got crashed. This might be resolved in some later versions of UHD, but you can use below workaround a suggested by Michael at here.

terminate called after throwing an instance of 'uhd::io_error'
  what():  EnvironmentError: IOError: usb rx6 transfer status: LIBUSB_TRANSFER_OVERFLOW
Aborted (core dumped)

use the command utils/b2xx_fx3_utils --reset-device if you ever have to Ctrl-C your application

One of the best books to understand the history of humankind

A favorite from the recent reads of my summer.
Sapiens: A Brief History of Humankind, by Yuval Noah Harari.
The book brings back the memories from the stone age to ice age to agricultural revolution to scientific revolution. I was fascinated to learn about the details of how the homo sapiens spread around the world during the ice age and got separated into virtual worlds until few hundred years ago. It was a one of the thought-provoking experience I have ever had.
I highly recommend this for each and every book lover. Happy #BookLoversDay 🙂 🙂

The Beginning of my PhD Journey!

The thought of PhD: After three years of fantastic experience in a startup, I realized that I am more interested in research and decided to pursue PhD. Now, the obvious question that appears to anyone is PhD in India or a foreign land. It was a hard choice. I have to leave my family, relatives and all of my friends If I have to choose foreign. To be honest, I was more worried about the food 🙂 Anyway, for whatever reason I decided to step foot on a foreign country. Now, where to go? how to go? what is the procedure to apply for a PhD program in foreign country(s)? A storm of questions in my head and have no clue for any of them.

 The immature happiness: Thanks to my friends, I got guidance from all along. At first, I decided to join one of the European institutes and started preparing for IELTS (a TOFL version of US). I got my GRE and IELTS scores (not so bad). So, now what to do with these scores? Got an advice that forward your resume with these scores to professors and ask if there is any research position in their group. Took that blindly and started approaching professors from TU Berlin, EPFL, INRIA Sophia, UCL and many others.  While I was not getting any response from any of these professors, I learned the importance of recommendation letters, statement of research purpose and a formal application procedure to the university. Meanwhile, I was also convinced that America is the place to go pursue my PhD (an obvious influence from your environs). But there is only so much of time that I had to figure this all out. It was already September 2015. I wanted to start my PhD as soon as possible. But, I have to wait for another year, If I want to apply formally and get an admission and join in Fall 2016. So, guess what, I again started my inquiry emails for research position to American university professors as well. Lucky this time, a professor from Missouri University of S&T, Rolla offered me PhD position. That’s it, I was so excited and instantly quit my job and started my preparation for the dream foreign land for almost everyone in the world.

The road not taken: America! The place where the earth was born, my mother says. It was Christmas, the 25th Dec the day I landed in Missouri. Silly me, was expecting full of people celebrating as if it were an Indian festival. Anyway, soon after I landed,  the first thing I noticed is that the roads are very clear and spacious. And of course the cold winter. Thanks to the social websites or apps, I made few friends already even  before landing in Missouri. I also got the comfort of speaking in my mother tongue and did not care enough to explore to meet new people. I have never met a foreigner before and have never talked to them either. Although, my new lab mates were Chinese, I could almost never get a chance to speak to them unless in our group meetings. It was almost no difference except I get to see few foreign people here and there but never talk. Things are normal until I hear the news from my adviser that she is going to move. I didn’t know what to do. Should I go with my adviser or should I stay. Friends suggest that I should consider some other better universities. For a moment, I thought I might be going back to India if I don’t get a research position anywhere. While I was talking to few professors, I got a last minute chance to apply for Stonybrook university in June, 2016 (For a Fall admission it was a rare situation). Believe me or not, I got an admission notification in two days of my application. I can confidently say that my admission to Stonybrook could be the quickest offer ever.

A new experience: Time to New York now, hah! This time I actually felt a difference from India. It was overwhelming to see so much diversity after I landed in New York. But, soon after I reached the campus, I realized that almost 90% of the graduate students are Indians and Yet I have almost no one to talk to in my mother tongue which is Telugu. Funnily enough, I am used to listen to more Hindi words in a foreign English speaking land unlike my hometown where Telugu is the only language I was exposed to. Admittedly, I got new friends with new cultures, new languages, and everything new. I can say now, I have friends from multiple countries — US, Brazil, Russia, Ukraine, China, Korea, Taiwan, Bangladesh, Pakistan, Turkey, Iran, Egypt, Greece and what not. Finally, with new energy and enthusiasm, I started my new career and hope to have a successful journey for next 5 to 6 years of my PhD study.