Rory.games

BeamXR Enterprise is an extension of BeamXR InGame, building upon the streaming technology with enterprise focused features and tools such as autonomous remote viewing, session events, remote control, and two-way communications. Developers can implement their own portals and directly integrate within their own M2M environments.

I architectured, developed, deployed, and provided support for the SDK, both within Unity and through numerous Azure and AWS based services.

BeamXR InGame is a fully featured streaming SDK for Unity that allows developers to easily add streaming capabilities to their VR games. It includes a full native streaming tech stack, multiple helpful Unity interactions and advanced camera features, allowing users to stream from any possible camera angle they want.

I architectured, developed, implemented, documented, and maintained the SDK, working with several partners across different games to help implement the tech within their experiences. This included both an initial version and a full rewrite and upgrade to a newer, high-performance native tech stack. There were two elements to this, the main core SDK which included the streaming stack, and the Director Kit which includes advanced features and a pre-built in-game UI (using both Unity XRI and the Meta Interaction SDK) to help developers easily integrate the tech.

Project Helios was a ground breaking research project within Ultraleap focused around the use of event-based camera technology for hand tracking. Unlike conventional cameras, event-based cameras only ever register the changes in light intensity between samples for each individual pixel, rather than sampling the entire image. This means you can sample at significantly higher rates than a standard camera.

Helios consisted of two main projects: traditional hand tracking, and gesture recognition for smart glasses. I was focused on the UX and interactions within the later.

I developed and produced both an SDK and demo applications for use within Unity. These were used for development of the smart glasses, user testing and model validation, and demonstrations at trade shows and investor meetings. I also helped demo the hand tracking at SIGGRAPH 2024.

Ultraleap Physical Hands is a fundamentally new way of interacting with 3D objects within XR, directly using your own hands. Rather than previous interaction solutions that rely on pre-determined poses to snap to, or slow physics simulations, this runs dynamically at runtime, correctly wrapping your hands around objects as expected, all within a single physics update cycle. It enables greater immersion with the simulation, while retaining the high fidelity and low latency that users desire.

I solely led and undertook nearly all the research this encompassed for over a year, with the final version of my work available within the Ultraleap Unity plugin. This encompassed a large amount of physics and heuristic research, while maintaining low compute and latency. Not only was the final solution better than what was previously available within the plugin, it was also lighter weight, both from a compute side, and a project requirement side.

Ultraleap's showcase of VR interactions. The demo encompasses a number of interactions I directly researched and created, such as Physical Hands for interaction, Jump Gems for teleportation, and the zero-gravity toy.

I also helped architect many of the underlying systems and did extensive performance work to improve the experience on stand-alone VR headsets.

Jump Gems are a VR locomotion method I developed, with the emphasis of being able to perform all the required actions with a single continuous interaction. A user would pinch the gem (located on their wrist), aim to where they want to teleport to, and then release to teleport.

They are available within the Ultraleap Unity and Unreal plugins, and have been used as the teleportation method throughout Aurora.

I rebuilt a large amount of the underlying application functionality of the Ultraleap Control Panel to work directly in Unity. It involved a lot of work that covered the full stack of the Ultraleap tracking service, all the way from USB information to image streaming and camera calibration.

The Ultraleap XR Launcher is an internal showcase of 2D interactions with direct hand interactions. I worked on several key interactions such as hover states, sliders, and text fields, as well as many parts of the underlying technical functionality.

My PhD summarises the research projects undertaken while at Ultraleap. It's three projects that look into the differences between controllers and hand tracking, while attempting to replicate the missing haptic feedback that comes with the use of camera-vision based hand tracking. I developed two studies, the first about varying UI visual reactions and their association to haptic responses, and the second about fine motor skill tasks when comparing hand tracking to controllers.

The haptic rendering algorithm I developed during the second study was patented.

The PhD also covers the work undertaken to create the user study tool PlayRecorder, featured below.

PlayRecoder is an open-source logic recording tool for use within Unity. It's designed for recording user study trials, and works within both the editor and built applications. A suite of extra tools are included to help the researcher during their analysis.

The tool was built during my PhD, and was subsequently used during my main user study. It allowed for me to easily record every user within my simulations, and then quickly analyse their actions by using the messages, timeline, and statistics features.

By utilising graphics principles of displacement maps, and then mapping the associated values to the intensity frequency of haptics, we were able to produce varying types of textures. This was all achieved using ultrasound transducers and I demoed the application at WorldHaptics 2019. This work also resulted in a patent, which can be viewed in the link above.