New feature: Fluid Lean using Cell Phone Sensor

[No Trickster]

UPDATE:  LIGHTING FIXED IN SECOND VIDEO: I've uploaded test 2 , which was made on the same night I made test 1.  This is all the footage from that day, so this saturday I'll film more footage.  I hope to make a good one.

Hi, I'm leaving this video, I modified TDM a little so it can use the data from a cell phone accelerometer.  I don't have to explain in this post how "the user experience is enormously expanded", just look at the video and see for yourself.  Before testing this myself, I didn't think it would be so gratifying using it.  It just feels so natural...

Right now only works with Side Lean (left / right).  But it WORKS VERY WELL and the movement is really smooth, no glitches or bad data.

(This is just the first test video, SORRY FOR THE LACK OF LIGHTING, I have so many more things to do to improve the videos, I have more thoughts to make better videos ... this is just the "proof of concept" video that I so eagerly needed to post ... and forum users needed to see  )

I'm new to this forum, THANKS TO TDM DEVELOPERS, WITHOUT THE MOD AND THE CODE THIS WOULD NOT HAVE BEEN POSSIBLE.  The engine is really a good Thief adaptation and the Fan Missions (lots of them) are very good, I must have tested 20 missions and 3 or 4 FMs amazed me by the architecture, the design, the textures, the 2D intro videos, the voice acting ... It really displays A LOT of work and many committed people to this project.

-------------

 

Edited February 22, 2020 by No Trickster

[Anderson]

It's decent. In various simulators people use TrackIR for games like that where you can move your head without moving your body in-game.

The fluid lean itself is useful. However the phone accelerometer seems a little of a gimmick in this case, no offense intended. Most people would probably tie the same command to a gamepad.

[No Trickster]

 

56 minutes ago, Anderson said:

It's decent. In various simulators people use TrackIR for games like that where you can move your head without moving your body in-game.

Yes , I was aware of that, I googled it and saw youtube videos.  Many systems like that for Flight Sims, etc.  But I have to tell that those systems need:

1) Buying (or)

2) Preparing the hardware (if maybe is unexpensive)

3) Configure 1 or 2 or 3 different softwares (for setting the right image "calibration")

Doing with a cell phone required me to:

1) Buy nothing (I already had it)

2) Download 1 App and configure it (1 minute)

3) IT JUST WORKS, AND IT IS EASY

(More things:  It doesn't need a secondary device to track the object, like camera trackers, AND: it is completely wireless!  This one I  )

56 minutes ago, Anderson said:

The fluid lean itself is useful. However the phone accelerometer seems a little of a gimmick in this case, no offense intended. Most people would probably tie the same command to a gamepad.

Have you tried this?  Right now, I know you can play with keyboard and mouse, and you can't do that with those.  And gamepad, does the gamepad allow for "fluid lean" or just "LEAN LEFT ENABLED"  / "LEAN LEFT DISABLED" kind of movement?

Hint:  I think my thought maybe came from remembering Rainbow Six 3: Raven Shield ... I took the word "Fluid" from them, and it had exactly this kind of Lean, in which you could do it with a key pressed while moving the mouse axis.  I remembered that one to be a very cool feature that I never saw again, maybe some other game implemented it, but those are a few ones I think.

Edited February 21, 2020 by No Trickster

[No Trickster]

Will post more videos later today or tomorrow!

[Anderson]

55 minutes ago, No Trickster said:

 

 

Have you tried this?  Right now, I know you can play with keyboard and mouse, and you can't do that with those.  And gamepad, does the gamepad allow for "fluid lean" or just "LEAN LEFT ENABLED"  / "LEAN LEFT DISABLED" kind of movement?

The analogue sticks will work for fluid leaning.

[No Trickster]

Just to clarify one matter:  what I implemented is not a trick.  It doesn't use the gyroscope nor "the linear accelerometer data" , it uses gravity data.  With that, NO DRIFTING PROBLEMS EXIST, because it is data from an absolute measurement.  The angle read from the cell phone is the right angle.  The input data is fast enough now, but I think I can make it a little better.

I'll clarify all matters in subsequent videos, I'll try to tackle all the benefits of how this is implemented, and try to explain them clearly, without the benefit of the doubt.

Edited February 21, 2020 by No Trickster

[No Trickster]

Any ideas for new video creation would be appreciated !!!

[peter_spy]

Games like Dishonored used the right stick along with another button held to indicate you want to lean and not strafe.

[No Trickster]

8 hours ago, peter_spy said:

Games like Dishonored used the right stick along with another button held to indicate you want to lean and not strafe.

Yes, that's exactly what Raven Shield implemented back in 2003.  If we try to make a list of games that implemented fluid lean, of all the games that allow leaning, I think we will find that maybe 3% of all "leaners" are "fluid leaners".

[peter_spy]

It's probably safe to say though that nobody will want to use it in TDM the way shown above. It would have to be either full VR or one controller implementation.

[No Trickster]

23 minutes ago, peter_spy said:

It's probably safe to say though that nobody will want to use it in TDM the way shown above. It would have to be either full VR or one controller implementation.

Can you further explain why?  Both of them:  why do you say full VR only or one controller only?

One of the reasons I made this was (I give 2 reasons) that VR is expensive to buy (in most cases) and because many years ago (no VR existed back then, at least not the VR as it is today) existed the idea that when you are (in game) in front of a corner of a wall, and you naturally have the tendency to lean, to look past that corner.  I've also seen people playing with keyboard and mouse like 20 years ago and they had a tendency to lean to a side to look further ... that move is made instinctively.  So that idea "follow the player side movement with the monitor" always was in my mind.

I know full VR is way better than this, it gives you an amazing experience, and this kind of leaning can't compete with that.  Anyway, this can improve gameplay experience if you don't own a VR headset.  Does TDM support VR headsets?  I'm asking out of ignorance, I just made a search in the forum and looks like it does not.

One more thing:  if I play with keyboard + mouse, I have to use Q and E for leaning.  If I use my head, there is another controller: keyboard + mouse + phone but I don't have to use Q and E anymore, then the player is freed from having to use fingers for leaning.  And you have 2 keys that can be bound to another actions ... Q and E are direct neighbors of W, and your middle finger is always over the W if you play with keyboard.  This could be extended to gamepads.  Ok, so if you play with the best gamepad around, why you would want to lean with the analog stick and not with your head?  (I'm asking this thinking of all possibilities, so ... if you were playing, why would you choose one kind of lean over the other? )

Edited February 22, 2020 by No Trickster

[cabalistic]

It does not, currently. I'm working on it, and there's an older alpha of VR support. Hopefully, after the upcoming 2.08 release, I'll be able to update the alpha version again

[No Trickster]

One further question:  Is there a poll in this forum that asks which types of controllers do players use?

Edited February 22, 2020 by No Trickster

[peter_spy]

10 minutes ago, No Trickster said:

Can you further explain why?  Both of them:  why do you say full VR only or one controller only?

IMO additional control gear breaks focus / immersion easily. Besides, noone will want to tape a phone to their heads. Also, it probably feels a bit weird to tilt your head and see the image inside the monitor tilt as well, while the monitor frame doesn't move. Assuming you're not sitting with eyes glued to your monitor, but in a safe distance, so you see the whole screen without moving your head or eyes. As you said, VR with full head tracking sounds more like the way to go. Or, maybe one of the controllers, although I'm not sure if using e.g. PS4 tilt controls for leaning would not be distracting. Definitely needs prototyping and testing.

[No Trickster]

I'll reply further about these issues, with no intention to be aggressive.

1 hour ago, peter_spy said:

IMO additional control gear breaks focus / immersion easily.

Why?  Are keyboard + mouse "immersive"?

1 hour ago, peter_spy said:

Besides, noone will want to tape a phone to their heads.

You are right, but they would have no trouble at all wearing a VR headset?  If you're saying it because "cell phones emit EM waves", then ok, you can put your phone in airplane mode, shutting of 4G.  But it needs WiFi ... so there would be the discussion "does WiFi" affect your cells?  And so I'll say "if there is any wireless VR headset, do you think it would not use WiFi , or the 2.4 GHz frequency band?"

1 hour ago, peter_spy said:

Also, it probably feels a bit weird to tilt your head and see the image inside the monitor tilt as well, while the monitor frame doesn't move. 

No, that's not right, you're assuming, you haven't tested it.  I tell you that it doesn't feel weird.

1 hour ago, peter_spy said:

As you said, VR with full head tracking sounds more like the way to go. 

Of course, I won't argue that.  But what if the game supports ANOTHER (experimental) kind of controlling it?  Let's say that 5 users out of 100 will use this fluid lean, and that allowing this would NOT BE THE DEFAULT, but with a CVAR you could turn it on or off, so normal users won't be affected if they download and install "vanilla TDM" ... would it not be great to have it and the possibility to test it anyways?  I won't harm TDM in that way, I can assure you that.

1 hour ago, peter_spy said:

Or, maybe one of the controllers, although I'm not sure if using e.g. PS4 tilt controls for leaning would not be distracting. Definitely needs prototyping and testing.

Yes I think the PS4 tilt would annoy you because you would have to have the controller ALWAYS in horizontal.  It depends on how it is implemented.  But I think it is not worth it.

"Definitely needs prototyping and testing":  If you refer to this fluid lean, I can ask who is on linux?  Anyone wants to test this?  You need an Android phone (and the app needed on the phone is on google play, I think it has many years on Goog Play, I didn't develop any Phone App for this).

Edited February 22, 2020 by No Trickster

[peter_spy]

36 minutes ago, No Trickster said:

You are right, but they would have no trouble at all wearing a VR headset?

Oh no, I'm not saying that. VR is still an expensive toy for early adopters, the way it makes so many people sick is still a problem. In comparison, taping a phone to your head just looks silly (although you could say the same about people wearing VR helmets).

 

36 minutes ago, No Trickster said:

Why?  Are keyboard + mouse "immersive"?

Actually, I prefer gamepad as the unobtrusive and compact device. But that required me to learn how to play FPP games with it first, so now it's a breeze. And the analog controls feel great, you're more like a cameraman gliding through the game world. That also prevents eye fatigue (you get there faster with rapid mouse movement).

Edited February 22, 2020 by peter_spy

[freyk]

Personally, I dont like the this leaning method (or a vr helmet) to use instead of my mouse & keyboard.
But maybe you can combine this with our VR project. (as people said it already)
Or use this as a cheap vr alternative:
TDM and steam on pc -> steam link & head movement app on your phone -> phone in a cheap view helmet.
And add connect some accessories to the phone: headphones, mini usb to usb hub,  gamepad (or mouse & keyboard)
More info aboutn TDM on steam link:

-edit-

Edited February 23, 2020 by freyk

[No Trickster]

9 hours ago, freyk said:

Personally, I dont like the this leaning method (or a vr helmet) to use instead of my mouse & keyboard.
But maybe you can combine this with our VR project. (as people said it already)
Or use this as a cheap vr alternative:
TDM and steam on pc -> steam link & head movement app on your phone -> phone in a cheap view helmet.
And add connect some accessories to the phone: headphones, mini usb to usb hub,  gamepad (or mouse & keyboard)
More info aboutn TDM on steam link:

-edit-

OK, nice to know that Steam has that kind of feature and that TDM works with Steam Link.

And no, right now I'm not interested in those "cheap VR" solutions.  I've tested one years ago, it was cool, smooth, on a Goog Pixel 3 that a friend had, tested with Goog Cardboard, but my cell phone doesn't have lots of hardware , and my WiFi is just 802.11 N, so I'm not sure if I have anything for it to run smoothly.

Anyways, I coded did this thing for using it with a monitor.  I like to think of it as an "extension" to "traditional PC gaming" with mouse + keyboard.  I don't dislike using my monitor for gaming.

Edited February 23, 2020 by No Trickster

[freyk]

Just give it a try.

i think if you use the head movement app and the monitor this way,
its not going to be good for you neck & other bodyparts. 

Edited February 24, 2020 by freyk

[stgatilov]

Wow! Looks cool

The latency is huge though. If VR sets processed input with this latency, everyone would get motion sickness very soon So I doubt this could really substitute a dedicated VR hardware. Also, I guess you cannot modify yaw/pitch the same way, since gravity data is one-dimensional, while orientation is three-dimensional. Anyway, with static monitor it makes no sense to rotate head around.

[No Trickster]

Thank you for your reply!!!

2 hours ago, stgatilov said:

The latency is huge though

EDIT:  I forgot something very important:  I think the input lag you saw was on the first seconds after starting the map?  Please, look again.  Follow the entire sequence.  It takes 2 or 3 seconds I think "to build up", and after that "warming up" period it works great and is more responsive.  Maybe look at the video that says test 1, the "darker video".  There I play through a FM until I die.  It is very important that you say which input lag have you spotted, which video and at what time.  (The "warming up" I think it is related to:  (1) intrinsic WiFi operation (when it changes from a "send nothing" state to "send data at X rate" state). (2) OS buffering.  (3) My algorithm for processing data )

So ... you could see it!!!  No one else said anything about latency.  Let me explain:  yes, you're right, it has lag, but I think it can be diminished.  Let's say this:  to make this video I configured it for stability, I mean minimize shaking.  I could try making it go faster and loose some stability.  I really did not test its parameters widely, I only tested 2 times, the first was horribly sluggish, the second was convincing enough to me for making these 2 videos (I didn't think that someone would notice it so early  ).  Some more testing (maybe 15 or 30 minutes) and maybe I will find a better parameter to minimize latency while not losing much stability.  (I must say: all this was forethought, I didn't leave it to chance   )

2 hours ago, stgatilov said:

So I doubt this could really substitute a dedicated VR hardware

No, it won't.  Think of what I made like this:  "an extension to monitor + controller" (where controller could be keyboard + mouse, or your favorite gamepad).  So it doesn't compete with VR, nor did I think it is on the same realm.  This would be for a better experience inside the game while you play with your monitor (and believe me:  it is really fun leaning yourself to lean in the game! ).

2 hours ago, stgatilov said:

I guess you cannot modify yaw/pitch the same way,

You are totally right, but just because of "real physical limitations", not limitations on the phone sensor nor the data.  Example:  when you move the mouse you're moving 2 axis.  I think they would be called yaw/pitch like you said.  Let's call them X and Y axis.  The player rotation in the X or -X direction won't be very useful (although it is "doable" if I code it), because of limitations of your own head and the visual field of your eyes.  The same would happen with rotation in the +Y or -Y,  though it would be worse, because human sight (or field of view) is more limited upside/down than left to right.  So I would say, there's human limitations, not hardware limitations.  Moreover, I'm shortsighted and I use lenses (don't ask me to wear contact  lenses because I hate them) and the lenses (depending on the design of the frame) puts more constraints over your side-to-side and to your upside/down view when rotating your head. 

2 hours ago, stgatilov said:

gravity data is one-dimensional, while orientation is three-dimensional

Not in this case.  The App sends 3 coordinates.  From there I could interpret the "whole orientation" of my cell phone (or any cell phone).  What I cannot achieve with this method is global positioning of the cell phone in a room.  That would require lots of other stuff, and maybe cannot be done (but I don't say it's impossible .... nowhere near impossible).  The 3 coordinates: think of them like this:  if you leave your phone over a table the Z axis is going through it (gravity vector is parallel to  Z axis in this case), so the phone App reads X=0, Y=0, Z = 9.81 (the last value could be gravity vector or could be anything else, depending on the data the sensor sends or that the App sends). 

2 hours ago, stgatilov said:

Anyway, with static monitor it makes no sense to rotate head around

Exactly.  EDIT:  If you're referring to rotating yaw/pitch, then you're right, and I gave the explanation above.  If you're referring to what @peter_spy said in one of this posts, then no, you're wrong.  I mean, this kind of leaning does work, I find  it useful inside the game, and it doesn't feel weird, or disorient you in any way.  Please, test what I'm saying in your house, in a room, wherever ... what happens when you lean to a side and look at the horizon?  Does the image your eyes receive or your brain processes rotate or looks weird?  No, it doesn't.  I tested leaning 90 degrees to a side, it looks like the brain compensates it or corrects it, and makes it horizontal all the time, like if your head never rotated.  I think that's a brain ability, compensation, or whatever, I think it is built-in into humans.

EDIT:  FUNNY DEVELOPMENT NOTE:  before testing it inside TDM, this required various tests for the coordinates received and its processing, and I needed to test which would be the maximum and minimum values "achievable" by a human neck/head going at maximum speed for correctly setting a scaling factor.  The human neck/head was mine,  and the tests were moving my head "in the most violent way possible" from side to side for capturing those minimum and maximum values.  (This could sound weird but it was needed!).  NO HEAD/NECK WAS BROKEN WHILE MAKING THOSE TESTS.   

Edited February 24, 2020 by No Trickster

[No Trickster]

More videos are coming!!!  I need to cover many topics in the videos, including some already discussed here.  Those videos would need more editing.  The 2 videos I posted are of "low production quality" and offers zero explanation of pros/cons, or maybe hints, it doesn't show implications of the development / design decisions, says nothing about performance.

EDIT:  the angle can be tweaked.  It is not like you always have to rotate your head 90 degrees to achieve 90 degrees in TDM,  right now the angle is upscaled a little, so maybe my head at 70 degrees makes 90 in TDM.   BUT EVERYTHING CAN BE EASILY TWEAKED!!!  DON'T THINK THE ANGLES OR THE HEAD ROTATION IS BOUND TO BE MADE ONLY IN THE WAY I SHOWED IN THE VIDEOS!  IT CAN BE EASILY ADAPTED FOR HIGHER SENSITIVITY, FOR EXAMPLE IF YOU ROTATE YOUR HEAD 15 DEGREES THE GAME ROTATES 60 DEGREES!!!   IT IS ADJUSTABLE!  ALSO, A MAXIMUM ANGLE CAN BE SET SIDE-TO-SIDE IN TDM!!!   Remember:  side-to-side lean in TDM was bound with a maximum angle of 15 degrees only!!

Edited February 24, 2020 by No Trickster

[No Trickster]

One other very important source for inspiration:  https://en.wikipedia.org/wiki/Police_911

Look this video, it gives the answer to the question "Can a human lean safely while playing a videogame?" :

 

 

[freyk]

is there software we can try/test? 

[No Trickster]

5 hours ago, freyk said:

is there software we can try/test? 

No, I'm afraid not yet.  I can tell you that I've been working on it to fix some issues, and I already fixed many of them.  Right now it is much better than before, when comparing to  those first versions on the videos.

I was on vacation, and all this "project" was born on my free time.

Anyways, it is not dead, I just didn't make more posts in the forum, because I wanted to fix it and make more videos like I promised.

Windows port:  I just didn't start  it yet.  I know I will have to do it if I want more users to try it, so I'll do it, but I can't say when it will be available.

EDIT:  My vacation ends this Sunday.  

Edited March 8, 2020 by No Trickster