cancel
Showing results for 
Search instead for 
Did you mean: 

New Unity UI + OVR Look-Based Input HOWTO

Anonymous
Not applicable
LookInputModule.jpg

Edit6: version 5 of example project/code:
https://www.dropbox.com/s/g8ptl7w9xdewp ... 5.zip?dl=0
Version removed old Oculus integration and uses Native VR support + 0.1.2 utils so it works with latest runtimes (0.7 or 0.8).

Edit5: version 4 of example project/code (fixed for and tested with Unity 4.6.4f1):
https://www.dropbox.com/s/3bpz5rgimbxdk ... 4.zip?dl=0
This version also adds support for a cursor that scales size with distance so it will always appear the same size but be at the right depth.

Edit4: version 3 of example project/code:
[removed - see updated example above]

Edit3: New version of example project and code posted:
[removed - see updated example above]

Edit2: If you don't have time to read all of this thread and just want the code and a sample project, I uploaded one here:
[removed - see updated example above]

Edit: fixed some issues with the code to handle InputField and also was selecting wrong game object in some cases.

I have been messing around with the new Unity GUI stuff in 4.6 release, and I figured I would share a basic solution to get up and running quickly with a look-based input system. Here is how to get a basic look UI working:

1. Add some kind of UI element to your scene. This will also add a Canvas object and an EventSystem object if you don't already have them in your scene. All UI elements must be parented by a Canvas.

2. On the UI Canvas object, set the Render Mode to World Space

3. Arrange your Canvas in world space and add more UI elements as you would like. To get started, I would just add some buttons.

4. Under the OVRCameraRig->CenterEyeAnchor object, add a regular Unity Camera at position and rotation 0,0,0, and set this camera's Culling Mask to "Nothing" so it won't actually bother rendering anything and waste CPU/GPU cycles. I named it "Look Camera". This camera is just for use by the UI event system. By putting it under the CenterEyeAnchor object, it tracks perfectly with the head. The OVR cameras don't seem to work with the UI system, probably because they utilize render textures and not screen space rendering. This dummy look camera solves that problem.

5. For every UI Canvas object you have in the scene, drag this "Look Camera" object into the "Event Camera" field.

6. create a new script called BasicLookInputModule.cs. Copy in this code:

using UnityEngine;
using UnityEngine.EventSystems;
using UnityEngine.UI;
using System.Collections;

public class BasicLookInputModule : BaseInputModule {

public const int kLookId = -3;
public string submitButtonName = "Fire1";
public string controlAxisName = "Horizontal";
private PointerEventData lookData;

// use screen midpoint as locked pointer location, enabling look location to be the "mouse"
private PointerEventData GetLookPointerEventData() {
Vector2 lookPosition;
lookPosition.x = Screen.width/2;
lookPosition.y = Screen.height/2;
if (lookData == null) {
lookData = new PointerEventData(eventSystem);
}
lookData.Reset();
lookData.delta = Vector2.zero;
lookData.position = lookPosition;
lookData.scrollDelta = Vector2.zero;
eventSystem.RaycastAll(lookData, m_RaycastResultCache);
lookData.pointerCurrentRaycast = FindFirstRaycast(m_RaycastResultCache);
m_RaycastResultCache.Clear();
return lookData;
}

private bool SendUpdateEventToSelectedObject() {
if (eventSystem.currentSelectedGameObject == null)
return false;
BaseEventData data = GetBaseEventData ();
ExecuteEvents.Execute (eventSystem.currentSelectedGameObject, data, ExecuteEvents.updateSelectedHandler);
return data.used;
}

public override void Process() {
// send update events if there is a selected object - this is important for InputField to receive keyboard events
SendUpdateEventToSelectedObject();
PointerEventData lookData = GetLookPointerEventData();
// use built-in enter/exit highlight handler
HandlePointerExitAndEnter(lookData,lookData.pointerCurrentRaycast.gameObject);
if (Input.GetButtonDown (submitButtonName)) {
eventSystem.SetSelectedGameObject(null);
if (lookData.pointerCurrentRaycast.gameObject != null) {
GameObject go = lookData.pointerCurrentRaycast.gameObject;
GameObject newPressed = ExecuteEvents.ExecuteHierarchy (go, lookData, ExecuteEvents.submitHandler);
if (newPressed == null) {
// submit handler not found, try select handler instead
newPressed = ExecuteEvents.ExecuteHierarchy (go, lookData, ExecuteEvents.selectHandler);
}
if (newPressed != null) {
eventSystem.SetSelectedGameObject(newPressed);
}
}
}
if (eventSystem.currentSelectedGameObject && controlAxisName != null && controlAxisName != "") {
float newVal = Input.GetAxis (controlAxisName);
if (newVal > 0.01f || newVal < -0.01f) {
AxisEventData axisData = GetAxisEventData(newVal,0.0f,0.0f);
ExecuteEvents.Execute(eventSystem.currentSelectedGameObject, axisData, ExecuteEvents.moveHandler);
}
}
}
}



7. Drag this script onto the EventSystem object in the scene.

8. fix Submit Button Name and Control Axis Name to match what you use in the Input Settings if you don't use Unity defaults.

9. disable or remove Standalone Input Module and Touch Input Module on the Event System object.

That's it. Push play. Look around with headset and you should see it highlight the buttons/sliders/etc. as you are looking around. When you hit the submit button it will trigger that UI element (buttons On Value Changed operations will be called if you set any of those up). Sliders get selected when you click them and can then be manipulated with the axis that was chosen on the BasicLookInputModule component.

I hope someone finds this useful since there isn't much out there on how to do this yet.

ccs
105 REPLIES 105

Anonymous
Not applicable
I'll have to keep playing with it to see if I can work out the kinks. I haven't encountered the double selection or frozen selection issue. I'm not perfectly replicating how the pointer stuff works in StandaloneInputModule, so I am probably missing some minor detail. Unity has some pretty complex handling in there and I was trying to simplify here for clarity and for my own sanity as well. 🙂

I also have world interaction I am doing outside of the UI stuff where I want to use the same button for selecting these other game objects and I have a separate raycast routine for them. I also want to reuse the controller axis for other purposes when not interacting with the UI. For that I made LookInputModule function ClearSelection public, and I call that in my other input stuff when button is clicked while raycasting over one of those other objects outside of the UI. In the code I posted, ClearSelction was private, but it is a useful public function. In your other code you could call:

if (!LookInputModule.singleton.guiRaycastHit) {
LookInputModule.singleton.ClearSelection();
}

You could actually put this into LookInputModule Process() if you want the selection to clear when you are looking away from UI stuff, or another option is to not accept the input button and axis control when guiRaycastHit is true. That way UI only has controller focus if you are looking at some UI element. This has given me a good idea to add another option to my example to control that: focusOnlyWhenUILook: when not looking at any UI element, ignore the controller/keyboard/etc.

Another option is to simply remove the Select() calls. The Select() is only there so UI elements like scrollbars/sliders get the controller axis focus and can be used while not looking at the slider/scrollbar, but if you don't need that you can remove it. InputField won't work without selection though, so if you need that you might keep the Select() in there and use the ClearSelection() method.

Anonymous
Not applicable
New version of project/code:

https://www.dropbox.com/s/ncs8d9mwy192f99/LookInputSampleV3.zip?dl=0

LookInputModule.jpg

Changes:
* more robust button handling
* added ignoreInputsWhenLookAway option
* added deselectWhenLookAway option

New code:


using UnityEngine;
using UnityEngine.EventSystems;
using UnityEngine.UI;
using System.Collections;

public class LookInputModule : BaseInputModule {

// singleton makes it easy to access the instanced fields from other code without needing a pointer
// e.g. if (LookInputModule.singleton != null && LookInputModule.singleton.controlAxisUsed) ...
private static LookInputModule _singleton;
public static LookInputModule singleton {
get {
return _singleton;
}
}

// name of button to use for click/submit
public string submitButtonName = "Fire1";

// name of axis to use for scrolling/sliders
public string controlAxisName = "Horizontal";

// smooth axis - default UI move handlers do things in steps, meaning you can smooth scroll a slider or scrollbar
// with axis control. This option allows setting value of scrollbar/slider directly as opposed to using move handler
// to avoid this
public bool useSmoothAxis = true;
// multiplier controls how fast slider/scrollbar moves with respect to input axis value
public float smoothAxisMultiplier = 0.01f;
// if useSmoothAxis is off, this next field controls how many steps per second are done when axis is on
public float steppedAxisStepsPerSecond = 10f;

// guiRaycastHit is helpful if you have other places you want to use look input outside of UI system
// you can use this to tell if the UI raycaster hit a UI element
private bool _guiRaycastHit;
public bool guiRaycastHit {
get {
return _guiRaycastHit;
}
}

// controlAxisUsed is helpful if you use same axis elsewhere
// you can use this boolean to see if the UI used the axis control or not
// if something is selected and takes move event, then this will be set
private bool _controlAxisUsed;
public bool controlAxisUsed {
get {
return _controlAxisUsed;
}
}

// buttonUsed is helpful if you use same button elsewhere
// you can use this boolean to see if the UI used the button press or not
private bool _buttonUsed;
public bool buttonUsed {
get {
return _buttonUsed;
}
}

public enum Mode {Pointer,Submit};
// LookInputModule supports 2 modes:
// 1 - Pointer
// Module acts a lot like a mouse with pointer locked where you look. Where you look is where
// pointerDown/pointerUp/pointerClick events are used
// useCursor is recommended for correct precision
// axis control of sliders/scrollbars/etc. is optional
// 2 - Submit
// controls are selected and manipulated with axis control only
// submit/select events are used
// in this mode you can't click along a slider/scrollbar to set the slider/scroll value
// useLookDrag option is ignored
public Mode mode = Mode.Pointer;

// useLookDrag allows you to use look-based drag and drop (see example)
// and also drag sliders/scrollbars based on where you are looking
// only works if usePointerMethod is true
public bool useLookDrag = true;
public bool useLookDragSlider = true;
public bool useLookDragScrollbar = false;

// useCursor only applies when usePointerMethod is true
// the cursor works like a mouse pointer so you can see exactly where you are clicking
// not recommended to turn off
public bool useCursor = true;

// the UI element to use for the cursor
// the cursor will appear on the plane of the current UI element being looked at - so it adjusts to depth correctly
// recommended to use a simple Image component (typical mouse cursor works pretty well) and you MUST add the
// Unity created IgnoreRaycast component (script included in example) so that the cursor will not be see by the UI
// event system
public RectTransform cursor;

// when UI element is selected this is the color it gets
// useful for when want to use axis input to control sliders/scrollbars so you can see what is being
// manipulated
public bool useSelectColor = true;
public bool useSelectColorOnButton = false;
public bool useSelectColorOnToggle = false;
public Color selectColor = Color.blue;

// ignore input when looking away from all UI elements
// useful if you want to use buttons/axis for other controls
public bool ignoreInputsWhenLookAway = true;

// deselect when looking away from all UI elements
// useful if you want to use axis for other controls
public bool deselectWhenLookAway = false;

// interal vars
private PointerEventData lookData;
private Color currentSelectedNormalColor;
private bool currentSelectedNormalColorValid;
private Color currentSelectedHighlightedColor;
private GameObject currentLook;
private GameObject currentPressed;
private GameObject currentDragging;
private float nextAxisActionTime;

// use screen midpoint as locked pointer location, enabling look location to be the "mouse"
private PointerEventData GetLookPointerEventData() {
Vector2 lookPosition;
lookPosition.x = Screen.width/2;
lookPosition.y = Screen.height/2;
if (lookData == null) {
lookData = new PointerEventData(eventSystem);
}
lookData.Reset();
lookData.delta = Vector2.zero;
lookData.position = lookPosition;
lookData.scrollDelta = Vector2.zero;
eventSystem.RaycastAll(lookData, m_RaycastResultCache);
lookData.pointerCurrentRaycast = FindFirstRaycast(m_RaycastResultCache);
if (lookData.pointerCurrentRaycast.gameObject != null) {
_guiRaycastHit = true;
} else {
_guiRaycastHit = false;
}
m_RaycastResultCache.Clear();
return lookData;
}

// update the cursor location and whether it is enabled
// this code is based on Unity's DragMe.cs code provided in the UI drag and drop example
private void UpdateCursor(PointerEventData lookData) {
if (cursor != null) {
if (useCursor) {
if (lookData.pointerEnter != null) {
RectTransform draggingPlane = lookData.pointerEnter.GetComponent<RectTransform>();
Vector3 globalLookPos;
if (RectTransformUtility.ScreenPointToWorldPointInRectangle(draggingPlane, lookData.position, lookData.enterEventCamera, out globalLookPos)) {
cursor.gameObject.SetActive(true);
cursor.position = globalLookPos;
cursor.rotation = draggingPlane.rotation;
} else {
cursor.gameObject.SetActive(false);
}
} else {
cursor.gameObject.SetActive(false);
}
} else {
cursor.gameObject.SetActive(false);
}
}
}

// sets color of selected UI element and saves current color so it can be restored on deselect
private void SetSelectedColor(GameObject go) {
if (useSelectColor) {
if (!useSelectColorOnButton && go.GetComponent<Button>()) {
currentSelectedNormalColorValid = false;
return;
}
if (!useSelectColorOnToggle && go.GetComponent<Toggle>()) {
currentSelectedNormalColorValid = false;
return;
}
Selectable s = go.GetComponent<Selectable>();
if (s != null) {
ColorBlock cb = s.colors;
currentSelectedNormalColor = cb.normalColor;
currentSelectedNormalColorValid = true;
currentSelectedHighlightedColor = cb.highlightedColor;
cb.normalColor = selectColor;
cb.highlightedColor = selectColor;
s.colors = cb;
}
}
}

// restore color of previously selected UI element
private void RestoreColor(GameObject go) {
if (useSelectColor && currentSelectedNormalColorValid) {
Selectable s = go.GetComponent<Selectable>();
if (s != null) {
ColorBlock cb = s.colors;
cb.normalColor = currentSelectedNormalColor;
cb.highlightedColor = currentSelectedHighlightedColor;
s.colors = cb;
}
}
}

// clear the current selection
private void ClearSelection() {
if (eventSystem.currentSelectedGameObject) {
RestoreColor(eventSystem.currentSelectedGameObject);
eventSystem.SetSelectedGameObject(null);
}
}

// select a game object
private void Select(GameObject go) {
ClearSelection();
if (ExecuteEvents.GetEventHandler<ISelectHandler> (go)) {
SetSelectedColor(go);
eventSystem.SetSelectedGameObject(go);
}
}

// send update event to selected object
// needed for InputField to receive keyboard input
private bool SendUpdateEventToSelectedObject() {
if (eventSystem.currentSelectedGameObject == null)
return false;
BaseEventData data = GetBaseEventData ();
ExecuteEvents.Execute (eventSystem.currentSelectedGameObject, data, ExecuteEvents.updateSelectedHandler);
return data.used;
}

// Process is called by UI system to process events
public override void Process() {
_singleton = this;

// send update events if there is a selected object - this is important for InputField to receive keyboard events
SendUpdateEventToSelectedObject();

// see if there is a UI element that is currently being looked at
PointerEventData lookData = GetLookPointerEventData();
currentLook = lookData.pointerCurrentRaycast.gameObject;

// deselect when look away
if (deselectWhenLookAway && currentLook == null) {
ClearSelection();
}

// handle enter and exit events (highlight)
// using the function that is already defined in BaseInputModule
HandlePointerExitAndEnter(lookData,currentLook);

// update cursor
UpdateCursor(lookData);

if (!ignoreInputsWhenLookAway || ignoreInputsWhenLookAway && currentLook != null) {
// button down handling
_buttonUsed = false;
if (Input.GetButtonDown (submitButtonName)) {
ClearSelection();
lookData.pressPosition = lookData.position;
lookData.pointerPressRaycast = lookData.pointerCurrentRaycast;
lookData.pointerPress = null;
if (currentLook != null) {
currentPressed = currentLook;
GameObject newPressed = null;
if (mode == Mode.Pointer) {
newPressed = ExecuteEvents.ExecuteHierarchy (currentPressed, lookData, ExecuteEvents.pointerDownHandler);
if (newPressed == null) {
// some UI elements might only have click handler and not pointer down handler
newPressed = ExecuteEvents.ExecuteHierarchy (currentPressed, lookData, ExecuteEvents.pointerClickHandler);
if (newPressed != null) {
currentPressed = newPressed;
}
} else {
currentPressed = newPressed;
// we want to do click on button down at same time, unlike regular mouse processing
// which does click when mouse goes up over same object it went down on
// reason to do this is head tracking might be jittery and this makes it easier to click buttons
ExecuteEvents.Execute (newPressed, lookData, ExecuteEvents.pointerClickHandler);
}
} else if (mode == Mode.Submit) {
newPressed = ExecuteEvents.ExecuteHierarchy (currentPressed, lookData, ExecuteEvents.submitHandler);
if (newPressed == null) {
// try select handler instead
newPressed = ExecuteEvents.ExecuteHierarchy (currentPressed, lookData, ExecuteEvents.selectHandler);
}
}
if (newPressed != null) {
lookData.pointerPress = newPressed;
currentPressed = newPressed;
Select(currentPressed);
_buttonUsed = true;
}
if (mode == Mode.Pointer) {
if (useLookDrag) {
bool useLookTest = true;
if (!useLookDragSlider && currentPressed.GetComponent<Slider>()) {
useLookTest = false;
} else if (!useLookDragScrollbar && currentPressed.GetComponent<Scrollbar>()) {
useLookTest = false;
// the following is for scrollbars to work right
// apparently they go into an odd drag mode when pointerDownHandler is called
// a begin/end drag fixes that
if (ExecuteEvents.Execute(currentPressed,lookData, ExecuteEvents.beginDragHandler)) {
ExecuteEvents.Execute(currentPressed,lookData,ExecuteEvents.endDragHandler);
}
}
if (useLookTest) {
if (ExecuteEvents.Execute(currentPressed,lookData, ExecuteEvents.beginDragHandler)) {
lookData.pointerDrag = currentPressed;
currentDragging = currentPressed;
}
}
} else {
// the following is for scrollbars to work right
// apparently they go into an odd drag mode when pointerDownHandler is called
// a begin/end drag fixes that
if (ExecuteEvents.Execute(currentPressed,lookData, ExecuteEvents.beginDragHandler)) {
ExecuteEvents.Execute(currentPressed,lookData,ExecuteEvents.endDragHandler);
}
}
}
}
}
}

// have to handle button up even if looking away
if (Input.GetButtonUp(submitButtonName)) {
if (currentDragging) {
ExecuteEvents.Execute(currentDragging,lookData,ExecuteEvents.endDragHandler);
if (currentLook != null) {
ExecuteEvents.ExecuteHierarchy(currentLook,lookData,ExecuteEvents.dropHandler);
}
lookData.pointerDrag = null;
currentDragging = null;
}
if (currentPressed) {
ExecuteEvents.Execute(currentPressed,lookData,ExecuteEvents.pointerUpHandler);
lookData.rawPointerPress = null;
lookData.pointerPress = null;
currentPressed = null;
}
}

// drag handling
if (currentDragging != null) {
ExecuteEvents.Execute (currentDragging,lookData,ExecuteEvents.dragHandler);
}

if (!ignoreInputsWhenLookAway || ignoreInputsWhenLookAway && currentLook != null) {
// control axis handling
_controlAxisUsed = false;
if (eventSystem.currentSelectedGameObject && controlAxisName != null && controlAxisName != "") {
float newVal = Input.GetAxis (controlAxisName);
if (newVal > 0.01f || newVal < -0.01f) {
if (useSmoothAxis) {
Slider sl = eventSystem.currentSelectedGameObject.GetComponent<Slider>();
if (sl != null) {
float mult = sl.maxValue - sl.minValue;
sl.value += newVal*smoothAxisMultiplier*mult;
_controlAxisUsed = true;
} else {
Scrollbar sb = eventSystem.currentSelectedGameObject.GetComponent<Scrollbar>();
if (sb != null) {
sb.value += newVal*smoothAxisMultiplier;
_controlAxisUsed = true;
}
}
} else {
_controlAxisUsed = true;
float time = Time.unscaledTime;
if (time > nextAxisActionTime) {
nextAxisActionTime = time + 1f/steppedAxisStepsPerSecond;
AxisEventData axisData = GetAxisEventData(newVal,0.0f,0.0f);
if (!ExecuteEvents.Execute(eventSystem.currentSelectedGameObject, axisData, ExecuteEvents.moveHandler)) {
_controlAxisUsed = false;
}
}
}
}
}
}
}
}

kashen
Honored Guest
Just curious, is there a way to use this OVR Look-Based Input to say, for example, pick up and drop a 3D object in the scene instead of just UI? Would greatly appreciate it if someone could point me in the right direction!

virror
Explorer
This way of distributing the code is a bit messy, dont get me wrong, really appreciate what you have done here!
But could you please package this as a unity package instead so it can be imported with ease into existing projects?

Agua
Honored Guest
Thanks again for sharing this.
How would go about selecting GUI objects with just the raycast?
For example, if you look at the button for more than 2 seconds it clicks the button. If you look at it for less than 2 seconds it is not clicked.

Anonymous
Not applicable
virror - I hear you on being messy. When I started this thread the code I shared was quite simple so it fit nicely within the forum post in a code section, and I hadn't intended to go further with it, but it kind of snowballed into a bigger project. 🙂 Next time I share, I'll share as a unitypackage or I might even add it to the Unity Store as a free asset. If I go that route, I can't package it with the OVR stuff included, so that is a downside to going that way. Sharing the way I did ensured a complete working project without requiring any hookup with the OVR prefab.

Since my last post, I have added a 2d slider component which allows setting x/y by dragging a button around within the 2d space. This is useful for a color picker, which I also made as a UI component. I also made a dropdown/popup list component/prefab which you use a set string list and connect it to another component using a delegate callback, or you can directly hook it to an enum field on another component. This has nothing to do with the Rift, but these are useful UI components, and you can control them easily with the same look methods as the other existing components. If I can find some time, I'll clean it all up and package it up to share.

Agua-

It wouldn't be too hard to add a time-based select/click. Basically in the GetLookPointerEventData() function in my LookInputModule, you should save the last looked at object into a private variable. When the last looked at object doesn't match the new looked at object, you start/reset a countdown timer for the length of time you want the delay to be. Then you add a check to see if that timer has gone below 0. If it has, and the looked at object is valid, you simply call the pointerDown or pointerClick handler on that object (look at the GetButtonDown(submitButtonName) code to see what it does and mimic that). This is an interesting concept, so if I get some time, I will add this to my LookInputModule as a feature. I can see using this in the GearVR apps I have in mind to develop. If I add this, I'll also add a little timer wheel or slider icon next to the red-dot pointer to show the timer progress. Thanks for sparking an idea! I don't really see how this will work with sliders/scrollbars though. For that I think you really need a button.

ccs

CubicleNinjas
Protege
Simply wanted to thank you for the helpful sample files. Appreciate it and looking forward to updates!

Anonymous
Not applicable
"kashen" wrote:
Just curious, is there a way to use this OVR Look-Based Input to say, for example, pick up and drop a 3D object in the scene instead of just UI? Would greatly appreciate it if someone could point me in the right direction!


Hey kashen, I recently integrated this system into my project. While the method I use is not a 1:1 "pick up and drop a 3D object in the scene..." I do manipulate a GameObject once look based input has established a target-of-interest. So I imagine if you wanted to; you could essentially track the duration of time passed since target-established, and then force the target GameObject transform to 'follow' the LookCamera (used on the CenterEyeAnchor according to original creators instructions). I'm not sure I'm performing this in the best way possible, but solving the problem for my project required adding in a basic public function to access the private class variable that contains the raycasting look input taget obj.

Let me know if I can help further.

CaptainOculus

M_i_c_a
Protege
Thank you so much, you have no idea how valuable this is to some people 😄 I finally have my first working GUI based off of this project. Keep up the great work :!:

panuszew
Honored Guest
Hey,

I am pretty new to the unity + oculus topic so I am looking for some help. I would like to change the script so that looking at the button for some time (like, 1sec) will trigger some script (e.g. camera change). Fo camer changing via keyboard I was just using:

#pragma strict

public var camera1 : GameObject;
public var camera2 : GameObject;
public var camera3 : GameObject;
var activeCamera : int;

function Start () {}

function Update () {

if (Input.GetKeyDown(KeyCode.V))
{
switch(activeCamera){

case 1:
camera1.SetActive(false);
camera2.SetActive(true);
activeCamera = 2;
break;
case 2:
camera3.SetActive(true);
camera2.SetActive(false);
activeCamera = 3;
break;
case 3:
camera1.SetActive(true);
camera3.SetActive(false);
activeCamera = 1;
break;
}



}

}


Could You please give me some tips on how to do that?