In case people are curious what it's like to work in VR, here's a demo of me using Simula: https://youtu.be/FWLuwG91HnI
I would argue that working in VR is fundamentally superior to working in PCs/laptops. It's basically 10x the screens, focus, and work immersion. Simula's text quality is also very good (getting around the eye fatigue present in older VR Desktops).
Well looking at the video, I mean, I can have dozens workspaces with hundreds of windows (if I need to) and navigate them with moving just my fingers, whereas you are apparently breaking your neck to navigate more than 3 windows. I think what you're doing is cool and has to be done for the sake of exploration, but I would really, really focus on navigation. Also, how are you supposed to focus on text, when reading or reading or writing when it's wiggling all the time. It seems to me that VR really is for games, but for working with text I think humans are better off with stationary screens.
On a second thought: if you would make it easier to "fix" window, such that one can focus on it and work. As when you are "placing" a new window in your video, that could make it a workable solution.
I haven't used VR devices too much but I believe the wiggling that you see in VR demos is not present when actually using the device. The device is simply adjusting the perspective to match your head motion and so the actual user is not seeing the wiggling but in the demo it appears that the view is wiggling to the watcher.
But I agree that wiggling is off putting. I think VR demos should low-pass filter the output to some degree to give a more realistic perception of what VR is like.
I wonder if a separate camera for pancake with a larger FOV would help, with the translation (e: and rotation) matrix lowpass-filtered from user orientation.
But I feel its mostly rotational motion that is perceptible in VR demos rather than translational motion. I suspect that's because a small rotational motion results in a large shift in perspective for objects in the world that are some distance away from the camera while small translational movements don't really result in a significant perspective shift.
Would be interested in seeing what research into human head motion while sitting/standing/walking/etc. shows though.
> It seems to me that VR really is for games, but for working with text I think humans are better off with stationary screens.
Have you actually tried VR? The fact that you think there's wiggling makes me think you haven't. Yes, videos make it look that way but I can guarantee you that's not reflected in the actual experience. VR is notoriously bad to watch on video and doesn't capture the experience at all.
There is no wiggling. That wiggling you see in videos is your head movements not being in sync with the wearer’s head movements. You’re doing it right now, your brain is just anticipating and compensating for it.
Actually the problem is low resolution and FOV of VR headsets. VR require at least 72Hz framerate at 4Kish which result in 2K 72Hz 90° ish stereo and that’s hard already even for moderate gaming PCs.
As for pinned windows: fixed HUD is a no-no in VR games because it burns in same spot of your eyesight. When a HUD is desired, players are usually given a virtual or symbolic helmet that allow display to lag behind head movement.
Navigation is one of the things we definitely need to improve upon.
I think the wiggling is actually kind of natural -- you don't notice your head or eye movements when you do them automatically, but it might be exacerbated in VR.
The fixing is an interesting idea, we'll try it out. I'm a bit worried it might feel weird in VR.
Yes, there is a thing called corollary discharge in the human visual system that keeps the world steady as one glances around (in the real world). Changing the direction one is looking causes the image to shift rapidly on the retina, yet perceptually we see a world that is not wiggling.
Corollary discharge[1] causes the brain to “subtract” out the motion of the image on the retina. It happens when our nervous system instructs the eyes to look in another direction. You can have someone else move your eyes or even do it yourself by pressing gently on the side of the eyeball with a fingertip on top of the lid. Wiggling one eye this way is not supported by the hardware of the eye and produces a very noticeable wiggly view of the real world.
I mentioned subtracting out the motion going on in the visual system. Experiments suggest that something like this is happening. If a muscle paralyzing drug is injected into the eye’s muscles it causes a wiggly world because the intent to glance somewhere causes both an ineffective command to move the eye, producing no real movement of the image in the retina while the triggered corollary discharge subtracts the intended motion from the still image and causes a perceived motion (in the opposite direction).
YouTube has support for VR videos. If you could record & render such that people with headsets could watch a session, that might help with interest and adoption.
I think navigation needs eye tracking, which sadly no headset currently supports. Focus-follows-gaze would be a game changer.
Uploading a YouTube VR video of a work session is a potentially fantastic idea. Thanks for the suggestion.
Right now, Simula uses "dumb" eye tracking, in the sense that windows receive keyboard and cursor focus when the user's forward eye gaze intersects a window. We also have it so that users can control the cursor focus with their forward gaze (presently binded to `Super + Apostrophe`); similarly, users can drag windows around by holding `Super + Alt` and looking around. The experience adds up to something quite productive once you learn all the keyboard shortcuts (your fingers don't need to leave the keyboard).
I bought a Vive Pro Eye specifically to try and develop eye tracking navigation for a system like this. I haven’t done any actual digging yet bc it turns out the NDAs for their SDK are...unfortunate. Definitely an opportunity for a free/open alternative.
Virtual Desktop already has many customizable features that allow you to work the way you're describing. I've worked in it before, and it was very immersive, though there was of course some fatigue.
You can have just one window at a time, and with a good headset, the wiggle is not significant. You can even adjust it so that your workspace follows you when your head turns.
I haven't used Windows' Virtual Desktop since the early days, but I suspect our text enhancements have improved upon the situation dramatically (when keeping hardware constant).
An interesting potential for VR would be alternative portable form factors. The laptop form factor is mostly dictated by the shape and size of the screen; since with VR the screen is moved to the HMD, this would no longer be a restriction.
The webcam view is also interesting; if it were "inverted" so that, instead of being confined to a window, it was outside of all windows (that is, the background "wallpaper"), it would work even better as a laptop replacement (that is, the screen doesn't cover your whole view, allowing you to keep aware of your surroundings).
If could contribute an idea, I wonder if one can make the camera watch your surroundings but with people recognizer that would notify you if someone is approaching with the intent to interact with you, and maybe show a notification or an approaching avatar. It would be neat in an office scenario. The next step would be a screen in the real world telling your visitor whether you're busy or interruptible, and a button the visitor can push to "knock on your door".
So that's the future, maybe: cubicle dwellers get their own VR offices...
This seems like the job for another program that you run simultaneously, not the window manager. Unless there are problems with having multiple programs make use of the camera or something.
I love the idea of a VR workspace. There are so many
limitations in our current workflows and tools that mainly just arise from being restricted to a small flat surface. Even just having your perfect setup anywhere you go would be amazing. All you need is a headset and a small portable cube computer - basically a laptop without a screen.
---
But after watching your video, I feel like this really needs some tiling window manager (i3 etc) inspired way of easily creating workspaces with tiles, and effortlessly navigating between them. With relatively minimal head movement.
Also, does Simula support curved displays? It would seem very natural to arrange different curved screens around you instead of flat ones, which also potentially waste quite a bit of useful space.
I use a curved monitor (Samsung CRG9) myself as my daily driver, and it appears curved when I use desktop view in SteamVR, so I definitely see the appeal. Just not sure how nice it looks on smaller windows.
I would not be able to work like that, it would lead to neck strain or wrist pain, if you think about it comes naturally to reposition your body when you are looking in a new direction there is a reason for this. This hurts me because the keyboard locks the shoulders and it's only my heavy head that moves, this is not ergonomical.
So being able to fetch up windows so it's perfectly centered above the keyboard would make it easier for me to use.
How is the eye fatigue in general? How long can you go in a VR work session?
For VR gaming (Oculus Rift), I start to experience mild discomfort after about 30min, which grows severe by 60min. It's hard to envision using VR for a full work day. (My eyes are fairly light-sensitive from laser-eye surgery; not sure how much is saccading, vs. bright light directly in front of the eyes.)
Many early VR Desktops (and VR games) didn't go to special effort to optimize text quality, and I think this has given people a lower than warranted impression of what is possible in VR today. There's no doubt that VR hardware is going to get exponentially lighter/sharper over time, and that these improvements will be welcome, but things are good enough now to put in 1hr+ long sessions without eye strain (at least in my experience as a daily user).
It really depends on the task - I (currently) have an OG Vive, and its absolutely fine for games with good anti-aliasing (no discomfort for me for basically any length of time, although I mainly play beatsaber and you don't really focus on the blocks beyond blinks of light and muscle memory), but it's hopeless if you want to watch a movie let alone program. The kind of programming I tend to do is generally fiddly enough that I've got a book open or some kind of datasheet etc. (I think - even good - VR would be a hindrance here)
I think the Index might be OK but I couldn't comment.
Productivity use for VR is way more compelling to me than any game I’ve ever seen. Maybe that’s why it’s never taken off. We probably need to have an email client people want to use in VR and an IDE before we worry about gamers.
Totally agreed. Our project's "secret": most people think that the future of VR is in games & entertainment, but it's actually in office work.
I never would have even purchased a VR headset had it not been for the promise to start working on a productivity environment. But before we can start building the killer VR office apps, we have to get the basic 2D apps working crisply/perfectly, IMO.
I came here to say this. Never in my life have I wanted a VR headset. I've used others' to play games (Beatsaber in particular is really fun) but I don't have time in my daily life to allow for a regular gaming habit or indulging that kind of money on it.
In 2017, I had a conversation with a co-worker, where we talked about how badly we wanted a VR desktop environment with infinite screen-space. There's VRDesktop, but that isn't this. And then again the same conversation last year with another friend.
After seeing the demo video, I was compelled to go look at Vive prices (~$1,000, ouch).
I feel like my productivity would skyrocket with this, mostly due to lack of background distractions. Imagine wearing noise-cancelling headphones. I want this so bad.
It’d be awesome if you could do this natively on the Quest (sideloaded). Native gesture control, inside-out tracking, wireless, almost twice the resolution, and wouldn’t even need to be connected to a PC at all so could be used anywhere (with Bluetooth keyboard and mouse).
Quest would be amazing because it would truly demonstrate a completely untethered, self contained portable full desktop system that you could take anywhere.
Having said that, getting even a standard desktop setup on Android to work well is iffy so doing it in VR does seem like a pretty immense challenge. But it would be a breakthrough in what it could demonstrate in terms of the concept.
Cool demo. But wouldn't it be better (for your neck) to move the screen instead of moving your head? Or both? I guess sooner or later it's going to hurt the way it is implemented now.
I would argue that working in VR is fundamentally superior to working in PCs/laptops. It's basically 10x the screens, focus, and work immersion. Simula's text quality is also very good (getting around the eye fatigue present in older VR Desktops).