Tuesday, September 9, 2014

RowJam! VR Rowing

Last weekend my friend Edwon and I made a VR rowing game! Here's the video Edwon put together to announce our strange creation.

A few years ago I stopped going rowing in a real boat because of back problems. The rowing part was actually no problem, but because of all the heavy lifting associated with getting a boat into the river, and all the expected work cleaning the rowing club every few weeks, I decided it was best to stop for a bit. I wanted to get a Concept 2 Rowing Erg at home so that I could still do some rowing exercise, but when I eventually did, it just sat idle for long periods of time because I prefer to go hiking or cycling instead of sitting on a stationary erg. I used to listen to TED talks while rowing on the erg to reduce the boredom of moving back and forth and not going anywhere, but after getting too lazy even for that, I decided I needed to do something to liven up using the erg.

Originally I started off by using a small python library written by a Concept 2 forum member to connect my Linux laptop to the machine. I used it to record data, but it wasn't doing anything more than the PM3 monitor attached to the erg, so it seemed quite useless. Since I am a game developer, I thought about gamifying the erg, and making some kind of 3d world through which I could row - I especially wanted to do it after seeing the Oculus Rift become reality and experiencing it for myself. However, a combination of laziness and difficulty in setting up my computer, an Oculus (also, remember that the original OR had giant cables and no positional tracking) and the erg prevented me from doing this. Fortunately, my more proactive friend Edwon got sick of hearing me talk about my idea for a virtual reality rowing game and forced a weekend game jam we named RowJam.

Although I had already managed to connect the erg to my Linux laptop, the setup was quite a pain. Although mac and Linux include Python libraries by default, Windows does not, and not only does Python need to be installed, a couple of usb access libraries need installing, and getting it all working together is not always straightforward. However, I recently changed the backend to be a Python script that uses websockets to send the data to any networked computer that supports the websocket protocol - that means just about ANY computer is capable of connecting to the erg as long as it has a modern web browser, such as Chrome or Firefox, and that also means that simple apps can be developed that are OS agnostic.

So over the two day RowJam, we managed to connect Unity to the erg computer and get data about the amount of power exerted by the user and simulated speed of the boat. This enabled us to use the erg to make a boat in the game move forward, and by adding some simple graphics we were able to fake rowing on water and through space. The head movement was all handled by the Oculus Rift's head tracking and gyros, and although we didn't make everything correctly to scale and had didn't have super amazing graphics in the game, the feeling of being in the boat and looking over the side into the water was really cool.

Unfortunately, the actual rowing part needs a lot of work in order to better sync up the movement of the user in the boat and the movement of the boat. In real rowing your movement in the boat does not affect the boat's movement to a great degree (i.e. if you slide up and down the sliders without pulling the oars), but I'd like to get the boat movement to feel a bit more natural nonetheless. I'd also like to work on a new version that tries to push the feeling of being inside a boat moving through an environment a bit more, try to fix problems with the backend so that many users can connect to the erg at once (even using different applications to visualise data) and try to make it feel a little more interactive than just adding power to the boat through exercise, such as using the Oculus head orientation to steer the boat.

I also have some other ideas for games using the erg that don't require an Oculus or powerful computer. One such idea was to use the erg to level up your character in a game through real exercise. After talking to Edwon about this idea, we thought it would be amazing to use Smart watches with heart rate monitors to detect your real-world activity and level up your game-world character's stats. I haven't looked too far into this, but I think there are already some apps that try to gamify exercise to some extent (achievements and King of the Mountain awards in Strava, for instance), but I'm not yet aware of any game that uses physical exercise as a way to increase your stats, or even generate a (game-world) currency. Anyway, since the backend for the erg now uses websockets, and the amount of things easily achievable in modern web browsers is growing all the time, it should be fun and interesting to try doing more erg based experiments in the future.

Although the source code for the backend is available on GitHub, it's a mess at the moment, so I won't link it :P

I'll make another post as soon as I clean it up and make it work properly. This should just be a case of making the erg thread communicate with the main thread that the clients are connected in, but since I haven't really done any multi-threaded programming where threads need to communicate, it might not be so straightforward... We'll see.

13 comments:

  1. Hi Andy,

    I came to this same conclusion about a year ago, when I asked myself, what would make me go to the gym? The only way was if they could turn a gym equipment into a game. The only machine I could see it working with was the Rowing machine.

    Here is what I came up with. Hope this adds fuel to the fire, because this is a brilliant idea and has a definite future Andy!

    ...

    Actually I will post my notes in another comment :)

    ReplyDelete
  2. HARDWARE

    ROWING MACHINE/ FIXED HEAD ERGOMETER (INPUT)

    INFRA RED (IR) SENSORS (FOR TRACKING) ATTACHED TO
    FOOT STRETCHER x 2 SENSORS (INPUT)
    SEAT x 1 SENSOR (INPUT)
    HANDLE x 2 SENSORS (INPUT)

    IR CAMERA (INPUT) ATTACHED TO HEAD OF ROWING MACHINE, TRACKING SENSORS ON
    FOOT STRETCHER (INPUT)
    SEAT (INPUT)
    HANDLE/S (INPUT)
    VIRTUAL REALITY (VR) DISPLAY HEAD UNIT (INPUT)

    PERFORMACE MONITOR (INPUT)
    EG. CONCEPT 2 PERFORMANCE MONITOR 4 (PM4)
    PLUGS INTO PC UNIT
    INPUTS DISTANCE SPEED AND RESISTANCE DAMPER (INPUT)


    VR HEADSET DISPLAY UNIT WITH INTERGRATED HEAD TRACKING (OUTPUT/INPUT)

    INBUILT IR SENSORS ATTACHED (INPUT)
    TRACKS HEAD POSITIONING AND MOVEMENT
    THIS WILL DETERMINE CAMERA POSITION AND ANGLING

    3D RENDERING (OUTPUT)


    PC UNIT (OUTPUT)

    WILL INCLUDE
    THE GAME SOFTWARE

    MONITOR PORT (EG. CONCEPT 2 - PERFORMANCE MONITOR4)
    INTEGRATED WITH PERFORMANCE MONITOR SOFTWARE
    PC/GAME WILL BE ABLE TO ALTER MONITOR SETTINGS IN CONJUNTION WITH IN GAME PHYSICS OR TRAINING MODE TO ADJUST MACHINES RESISTANCE DAMPER.

    VR HEADSET DISPLAY UNIT PORT
    INTEGRATED WITH VR HEADSET DISPLAY UNIT SOFTWARE

    NETWORK ACCESSABILITY
    WIFI CONNECTIVITY
    ONLINE GAMING CONNECTIVITY [SEE ROW PRO (ONLINE ROWING AND PERFORMANCE MONITORING)]
    LAN GAMING CONNECTIVITY
    ADD ON'S
    HEADSET CAMERA (OUTPUT/INPUT)
    ABILITY TO SWITCH ON/OFF, IN GAME, ON SCREEN CAMERA VEIW, OF REAL WORLD ENVIRONMENT, IN REAL TIME
    SMART WATCH INTERGRATION (OUTPUT/INPUT)
    TAKES IN FITNESS INFO
    MONITORS, UPDATES AND UPLOADS, DAILY ACTIVITY, PROGRESS, PROGRAMS, FITNESS GOALS OR FITNESS STATUS
    ENABLES SYSTEM TO MATCH USER WITH APPROPRIATE FITNESS CATEGORY

    SENSES EFFECTED
    EYES
    EARS
    BODY

    ReplyDelete
    Replies
    1. GAME
      Game will require a User Login Profile.
      Game include the following types of Game Modes. They could almost be considered individual gaming packages.

      Delete
    2. TRAINING
      Pick a body of water, anywhere in the world. Pick a time of day. The worlds lakes and rivers are now available to you to enjoy your rowing experience.
      Tutorial (Solo)
      Game will get user to perform movement order. That is the movement order for the best technique.

      Legs first, Core, Arms, Arms, Core, Legs

      Game will teach proper technique for using machine.

      Eg. Order, Power Timing as demonstrated in the following Youtube video https://www.youtube.com/watch?v=sfnAhEAa_T8

      Using input data received by IR camera, while performing instructed motions
      ie, tracking positioning of head sensors, in relation to seat sensor, handle sensors and foot sensors,
      Game will use information to determine correct/incorrect posture and will influence proper technique for using machine.

      Time trials (Solo)
      Race the clock to beat your own time. See you’re the ghost lap of your best time and try to stay ahead of it.
      Time trial different distances
      500m, 1000m, 2000m, 6000m etc.
      Choose a time limit and see the distance you can make in set time
      Your times will be recorded under your profile. Times recorded can be uploaded to a World Stage data base.
      Monitors performance and graphs results so user can analyse performance.

      Coxswain Challenge
      Choose a level of training, from easy to difficult
      Follow the commands of the Coxswain
      Great for interval training
      The Coxswain dictates speed, and for how long and over what distance. The Coxswain provides in game motivation and encouragement as well to push you to your limits.
      The Coxswain is your own personal trainer

      Delete
    3. ARCADE
      Arcade is where the 2 sensors on the Handle bar come in to play. The handle can now act as a turning/steering mechanism in game. By angling the Handles left sensor down, the user can steer left in game. By angling the Handles right sensor down, the user can steer right in game. This feature also strengthens the users control on the Handle bar, to keep a sturdy steer in game.

      The Chase Games
      Fun games – Choose a level of Difficulty

      Get chased by all sorts of different dangers in the water. This involves sprint sessions, with rest periods. This helps the user to push their limits while having fun. The exercise will be Interval training focused, ranging from Low to High Difficulty depending on the level selected.
      Game Ideas
      Piranhas in the Amazon
      Sharks in Hawaii
      Anacondas in Africa
      Crocodiles in Australia
      Tidal waves
      Dinosaurs and sea monsters
      Drug lords at night, under cover of night, while trying to avoid the spot lights

      Space Race Games
      In a land far, far away, in a not to distant galaxy, where gravity is kind. they have racing machines that are powered by the driver pulling a chain to turn the hyper wheel that charges up the hyper drive.

      Get that hyper drive fully charged to reach hyper speed. (Imagine Mario Cart with hover crafts)

      This Game Mode will be very unique in that the user will experience the opposite sensations of other Modes in which the sensation is travelling backwards in game.

      Again the exercise will be Interval training centred, ranging from low intensity to high intensity depending on level of Difficulty.

      Game Ideas
      Race the different tracks with a sci-fi twist
      Blast through portals
      Collect packages for extra boost
      Collect lightning package to boost at light speed
      Race on the Sun
      Race through space

      Variations in tracks

      Glass Tube races (picture water park slides). Racers speed through protective clear tube tunnels where they can see the environment outside the tubes. Racers can also see their opponents. At certain junctions in the race, there may be
      Through water
      Underground
      In Space
      Out run a collapsing tube

      Desert canyon races (picture Star Wars – Phantom Menace, Pod racer scene)
      Rest periods coincide with drops down cliff face etc.
      Push hard to make it up cliff faces as quickly as possible

      Under Water races
      Race while avoiding becoming lunch

      Delete
    4. WORLD STAGE
      (See CRASH-B SPRINTS and Row Pro)
      Challenge users across the globe. Celebrities, friends, family.
      Represent your country in the Olympics and Commonwealth Games, race against other countries. Single and team events.
      Train with your squad. For example 8 man sculls can simulate being out on water while in game, taking their normal positions in game view, eg at bow end or stern.
      Challenge users in your gym
      Challenge other squads.

      Delete
  3. THE GAME SOFTWARE & TECHNOLOGY
    What the Software will do.
    The Software will receive input from IR Camera, monitoring head sensors, seat sensors, foot and handle sensors. Software will interpret sensors in game as follows
    IR sensor tracking input
    Head sensor tracking input will
    Determine the position movement and angle of users head
    Determine the position and angle of in game camera view while inside row boat or space race machine.
    Determine where the users head is positioned in relation to seat, handle and foot sensors.
    Determine users torso height by measuring distance between head and seat sensor tracking sensor
    Determine correct operating technique adjustments required if need be by using all tracking sensors, to track body movement and positions in tutorial mode.
    Foot stretcher sensor tracking input will
    Determine the users feet position while in game
    Determine leg length by measuring distance between foot sensor and seat sensor at full length extension
    Determine correct technique by measuring length of full leg extension in relation to seat sensor
    Determine correct operating technique adjustments required if need be by using all tracking sensors, to track body movement and positions in tutorial mode.
    Handle sensor tracking input will
    Determine users hand and oar position and movement while in game
    Determine arms length by measuring distance between torso and handle at both full arm extension and full pull completion
    Determine correct technique through above mentioned triangulation
    Determine steering left and right when in Space Race mode
    Allow user to enter in login details or adjust setting using right and left gesture

    Seat sensor tracking input will
    Determine users posterior/groin movement
    Determine length of legs in relation to foot sensors
    Determine leg movement in game
    Full extension
    Full contraction
    Determine position of camera view in game when triangulated with all sensors
    Determine correct technique
    Rowing Performance Monitor Unit
    The Software will receive input from the Rowing Performance Monitor Unit, translating speed, resistance, and distance data in game. I will input and effect physics of game.
    Software will also be able to adjust resistance in game to meet each challenge/mode setting.
    Software will also have the ability to show varying graphs for statistical analysis.

    ReplyDelete
    Replies
    1. The In Game Camera View
      The Camera view in game will be influenced by
      the movement of the boat or vehicle through an environment
      boat on water
      space racer on track
      the movement of users seat
      the movement of the users head and torso

      PC Unit Recommended Specs
      Intel CPU - Core i3-2120T 2.6GHz
      AMD CPU – Phenom II X4 810
      GPU – Geforce GTZ 650 1GB
      GPU – Radeon HD 7750 1GB GDDR5
      RAM – 4GB
      OS – Win 7 64
      DIRECT X – DX11
      HDD SPACE – 20 GB

      Delete
  4. Did this ever get any traction? I had this same idea this morning (and I'm a little bummed someone already thought of it) and I have some ideas which could potentially make this blueprint a reality. Would love to know where this stands and if everyone is serious in pursuing if you haven't already.

    ReplyDelete
  5. and my name is Brian, apologies for the unknown tag.

    ReplyDelete
  6. Any news on Rowjam VR? My Waterrower is collecting dust.... Would love to use it with my Vive on...

    ReplyDelete
  7. Would love to see this thing come to life. Have a Waterrower and a Vive and don't use either nearly enough. I'm also 25 lbs overweight. Would pay $100 or more for a piece of software that gets me to use both of those things while making me skinny.
    Put a bracket on the rower handle to hold the Vive controller and track the controller and headset in the playspace and that should be all the input you need to accomplish the goals that were initially laid out. With the advent of the Vive and it's tracking capabilities that should eliminate the need for all the complexity of the inputs from the rower. Sure you won't know the resistance being used to accurately measure the calories/force being expended but if you know the Waterrower is being used at the recommended water level you can estimate it from the speed and length of the strokes that the Vive controller is providing.

    ReplyDelete
  8. Hello, I am curious how you connected the Unity world to the Concept 2 and Oculas? What software was used? Thank you!

    ReplyDelete