Hello everyone! I am Owada, an XR (Extended Reality) engineer at NEXT-SYSTEM.
When it comes to my blog posts, there are quite a few introductions related to AR (Augmented Reality), but in reality, I take on quite a lot of VR development projects.
And outside of work, I’ve actually spent over 2000 hours playing VR games, so I consider myself quite the heavy user!
This time, I would like to talk about VR (Virtual Technology)!
Nowadays, many clients are visiting us for development discussions and consultations due to the growing interest in the Metaverse.
Especially when clients come for VR-related consultations, we demonstrate and let them to experience our development content as a demo. However, I have been concerned about the following things:
- The display only shows a first person perspective, so the viewer cannot see the overall picture of what is being observed.
- Since one’s own avatar is not displayed on the screen, the experience of the participant in the VR space is not conveyed.
- During VR demonstrations, it is difficult to explain the content because it is unclear what is being displayed on the screen.
I wanted to make the appeal of VR more clearly and easily understandable.
Therefore, I’ve created something like this!
You can view the 3DCG world of VR experiences from any location using mobile devices such as iPads!
Since the mobile device side is simply running a web application on the browser, there’s no need for any special app installation. In other words, you don’t need to prepare dedicated devices!
How can we do it? What can it be used for?
I’ll introduce the development of a viewer feature using the latest features of Unity, which allows for easier and more comprehensible viewing of the VR world from a third-person perspective!
System Configuration of the Entire Tool
In order to stream video within a local network, both the VR operating PC and the viewing mobile device (such as an iPad) should be connected to the same network, such as Wi-Fi.
The main VR application willl incorporate a virtual camera feature that tracks the tracker, enabling real-time rendering processing on the PC side.
When accessing the web application on the local server from a mobile device such as an iPad equipped with a tracker, the camera video rendered on the PC is directly streamed to the web application on the device. This allows users to view the video from the freely movable virtual camera.
We utilized the HTC Vive Pro VR headset, renowned for its high-resolution visuals, enhanced comfort, precise tracking, and seamless compatibility with professional applications.
Detailed Mechanism
The two core technologies required to implement this system are:
- Estimation of the position of a mobile devices in virtual space and realization of a free viewpoint camera viewable from that perspective.
- A system that allows video captured by the camera to be streamed and viewed on mobile devices in real time.
These are achieved using the following methods.
Virtual Camera with Vive Tracker
We developed a versatile tool suitable for PC-connected VR environments using a LightHouse system (which involves setting up base stations for sensor tracking). For position estimation, we utilized Vive Tracker.
By attaching the tracker to viewing devices like an iPad, we can detect the position of the viewing device within the VR space.
In Unity, we set up an additional camera for streaming alongside the VR headset’s projection camera. By aligning this camera object with the position of the iPad in the real world, we can generate a virtual camera capable of free movement.
Using a tracker that operates on the same tracking method as controllers and the VR headset allows us to identify the camera’s position based on the SteamVR room space reference. This enables us to move the camera’s position rapidly and accurately.
The purpose of developing this tool was to enable observers to view the experience of users wearing the VR headset from a third-person perspective. Therefore, by utilizing trackers that accurately reflect the distance between the real-world user and the observer. This allows the observer to continue capturing the user’s experience even as they move around the VR space with controllers.
Streaming via Unity Render Streaming
This system’s most crucial feature, the capability to transmit video to devices, utilizes an API called Unity Render Streaming, provided by Unity officially.
For more details👇
https://docs.unity3d.com/Packages/com.unity.renderstreaming@3.1/manual/index.html
This is a cutting-edge feature that enables direct streaming of video from any camera placed in the scene to the viewing device.
The core technology enabling the concept of WebRTC, which facilitates Peer-to-Peer (P2P) communication, allowing clients to directly send and receive data without the need for a server.
There are two devices: the Sender, which wants to send data, and the Receiver, which wants to receive data. By establishing a connection between these two clients via a server called the signaling server, direct data transmission between the clients becomes possible.
The data that can be transmitted includes audio, video, binary data, and more.
In this tool, a local server is set up as the signaling server on the VR execution machine. It implements the functionality within the virtual camera object in the VR scene to connect to the signaling server and stream video to the viewing devices.
When you open a browser on the viewing device (iPad) and access the IP address of the signaling server, the signaling server establishes a connection between the VR machine and the viewing device. Then, the VR machine streams real-time video from the virtual camera to the viewing device.
Unity Render Streaming provides these WebRTC systems as development packages, covering the implementation of P2P, signaling server, and web application parts.
With this package, we were able to implement everything from streaming to viewing relatively quickly, which enabled us to get it up and running in a relatively short period.
Strengths of this system
When a client visited our office the other day, we demonstrated VR content using this system. With the ability for viewers to freely observe the content without wearing an VR headset, it became much easier to convey the immersive experience to them.
To convey the allure of VR, there’s no doubt that allowing people to experience stereoscopic images through an VR headset is the most effective method. However, VR content is primarily designed for single-user experiences, and there are challenges such as difficulty in operation and motion sickness. Hence, it’s common for developers to demonstrate the content themselves.
In such cases, having a third-party perspective camera that allows viewers to observe from any angle, rather than a first-person view camera that moves vigorously allows VR users to objectively understand the experiences they can have.
Furthermore, even during experiences where participants engage directly rather than observing a demonstration, developers can guide operations and introduce content as operators while speaking and viewing through a third-person viewpoint camera. This approach enhances smooth communication and shared experiences among participants.
In Summary
This was an introduction to a virtual camera tool that allows you to peek into virtual spaces from a free perspective!
We plan to package and update this feature for future VR development as a versatile option.
If implemented, we believe it could enhance VR content by offering a service where you can peek into the virtual world by simply attaching a tracker to your smartphone or tablet!
Why not consider adding a little extra value to your contemplated VR services?
At Next System, we specialize in developing content and services using the latest xR technologies such as VR and AR!
If you’re considering new xR services, please don’t hesitate to consult with us!
▼NEXT-SYSTEM’s website
You can also find us on Twitter (X), Facebook,Website, Youtube, & Linkedin.
