The Dark Mod on an open source engine

[javiert]

Hi,

 

For research purposes, I'd be interested in porting the Dark Mod to an open source engine (mainly because one can then plug in high-level language environments like Java or erlang on which to quickly test new AI). The game engine itself should be no problem, I guess, but I'd like to know if there's any reason (lack of dynamic lighting, for example) for avoiding one. My initial choices are ioquake3 and DarkPlaces. Does this sound remotely feasible?

 

Thanks a lot!!!

 

Javier

[eigenface]

Supposedly, the Doom 3 engine will become open source before the end of the year. My impression is there's more than enough work left to be done on dark mod in the meantime that does not require an open source engine.

[javiert]

I'd rather not wait until the end of the year My impression is that changes in the engine will be done mainly in the graphics end. I wouldn't expect a comms API for bot clients on separate processes, or a high level language AI module, or a distributed game state engine (to allow multiple threads and servers) in the Doom3 engine, so I'll have to build them myself. I don't mind if the final look is not as polished as one would have on the Doom3 engine, but I need to be able to tweak these things right now

[Serpentine]

Personally, I think you'd require an immense amount of time to even get anywhere with doing it. That said, it's always a possibility. ioq3 would require loads of work but gains in portability, darkplaces would be far more work. That said I think that if you were really serious perhaps Xreal would be a better choice as it's far more focused on a modern approach to things, something that will come in handy if you need to add additional model formats or adapt lighting to better fit. Blur might also be an extremely good choice, however I dont know if they have their repo public (they do have assets and such on sf.net iirc)

 

There's also a lot of compromise made in TDM, for example the light gem is a Rube Goldbergs machine and could be vastly simplified, porting things like this directly will get real old, real quick. Content is the other area - you'll need to have md5 support for characters as well as ASE/LWO2 for static models (I guess you could use assimp, tho it's not 100% for our content yet, tho I'm working on a few patches atm).

 

 

[New Horizon]

Xreal would be the ideal choice.

 

http://xreal.svn.sourceforge.net/

 

http://sourceforge.net/projects/xreal/develop

 

http://www.phoronix.com/scan.php?page=article&item=xreal_engine&num=1

 

The website is down, but the SVN repo is still being updated. Contact the author.

 

XREAL would be a far better choice for many reasons.

 

1. It is the Quake 3 engine brought up to Doom 3 standards. Actually, it's graphically more advanced than D3 right now....and probably better optimized.

 

2. It supports all the important Doom 3 formats, like md5 and md5 anim and it also supports most of the doom 3 material shaders.

 

3. The developer of XREAL uses our Dark Radiant level editor, so there is already some part of TDM in there.

 

I think the author shifted focus form Xreal to making an ET-Xreal instead. In fact, here is a test video of it in action.

 

 

Xreal is simply the best chance of porting TDM to an open source engine with as little pain as possible.

[javiert]

There's also a lot of compromise made in TDM, for example the light gem is a Rube Goldbergs machine and could be vastly simplified, porting things like this directly will get real old, real quick. Content is the other area - you'll need to have md5 support for characters as well as ASE/LWO2 for static models (I guess you could use assimp, tho it's not 100% for our content yet, tho I'm working on a few patches atm).

 

@Serpentine: I know the mod is a direct port, and I'd have to redo a lot of things as they are changed. But at least it's a complete FP stealth game to which AI can be hooked The model part is definitely a point. I don't mind having robots chasing Chinese girls using the OpenArena models (although this strays a bit further from the Thief game), but having animations for physical moves would be a great plus. More about Xreal below...

 

Xreal would be the ideal choice.

 

1. It is the Quake 3 engine brought up to Doom 3 standards. Actually, it's graphically more advanced than D3 right now....and probably better optimized.

 

2. It supports all the important Doom 3 formats, like md5 and md5 anim and it also supports most of the doom 3 material shaders.

 

3. The developer of XREAL uses our Dark Radiant level editor, so there is already some part of TDM in there.

 

@New Horizon: I know about XReal. Some "extreme" comments about its prformance from around the net and the fact that the project is no longer available did move this choice a couple of steps backward in my mind. And the lack of a player base. As I said, the graphical part is not so important, but I agree that any modification for a quake3 derived engine will be easier to fit into the Doom3 engine when it is "freed"

 

I think I'll try then ioq3 and try to set up a XReal test env at the same time. I'll evaluate Nexuiz just in case; sincerely, I don't think the network and server concepts have changed so much from the original quake, but I'll heed your advice and keep DarkPlaces only as a backup plan.

 

Thanks a lot to everyone!

[New Horizon]

@New Horizon: I know about XReal. Some "extreme" comments about its prformance from around the net and the fact that the project is no longer available...

 

The Xreal engine is still available and being actively developed, the repository was only updated 6 days ago. The repository is still available to the public, and I just downloaded the code a few minutes ago.

 

The author has simply stopped paying for the website since he is no longer making a 'game' but he is still developing the 'engine'.

 

All the doom 3 functionality is already there. Our animations and AI can be used in Xreal. I've played it before and the performance was fine on my old system, considering it's a next gen engine I expected worse.

 

If you're going to do this, please reconsider the Xreal engine...if it had been around when we started Dark Mod it would have been the obvious choice.

[javiert]

Now that's a couple of reasons Ok, ok, I'll get the XReal code and give it a try! Thanks!

[javiert]

And the Lua support is a plus! Isn't there also an exprimental development for replacing the QVM with a Java VM?

[ungoliant]

this is pretty exciting stuff! have you worked out any timeframe of how long this might take to implement, javier? 3 months? 1 year?

[STiFU]

Keep in mind that TDM still uses a couple of D3 assets, so you won't be able to use those, even if D3 should eventually be open source. But still, good look with your project. Looking forward to some progressupdates in the future.

[javiert]

this is pretty exciting stuff! have you worked out any timeframe of how long this might take to implement, javier? 3 months? 1 year?

 

Pay me 3 months and it will surely take 3 months XD

 

Now seriously, my first task is to isolate a library for the erlang and java VMs that can connect to an OpenArena/XReal server and at least use some functions from the AAS module to create the zones from the BSP tree. That should only be an issue of cutting code out her and there from the client. But it took me some time to find my way around the code...

[Sotha]

Pardon me for intruding the conversation. I think it is cool if someone want's to start porting TDM to an open source engine, but just one question.

 

Why?

 

You mentioned 'research purposes,' some kind of student project? Or you want to do it only because you can?

I admit, I am the curious kind.

[Tels]

Xreal would be the ideal choice.

...

I think the author shifted focus form Xreal to making an ET-Xreal instead. In fact, here is a test video of it in action.

 

 

Xreal is simply the best chance of porting TDM to an open source engine with as little pain as possible.

 

When I look at the video above, I really wonder why I even bother with SEED...

[STiFU]

A friend just told me that xreal doesn't do the bumpmapping for dynamic lightsources, but rather for a static lightsetup by utilizing a deluxe map. According to him, dynamic bumpmapping can be activated but only at very high cost, as the engine misses out on the portals of Doom 3. Kinda makes me think that the engine is not very well suited for the purpose after all.

[demagogue]

Keep in mind if you start a port you're branching off Darkmod. In a year or so TDM will keep updating and we could already be on like 1.09, and now you've got a port for 1.04 that can't even play the 30 FMs already out (without the Doom3 assets), but is even more useless for any version afterwards. And the idea of keeping it up to date with Darkmod's update while you're still developing sounds ... hard, to say the least. (And why would anyone want to make an FM on an incomplete v1.04 when they can build on a complete v1.09?)

 

Now if you branched off Dark Mod 1.04 to make your entirely own game, that might make more sense. Make a modern or ninja version. But then you have a TON of work to make a standalone game out of it. The entire Dark Mod team over the last 6 years still haven't had enough time or manpower to make Dark Mod itself standalone yet and have practically taken it off the table for the foreseeable future because the workload is so insane and unnecessary (in remaking a 1000 things already there). Now imagine an entirely new game with one person. That's just if you want to actually do something practical with this, as opposed to it just being an academic exercise, though.

 

On the other hand, we could just wait for Doom3 to go open source and get all the advantages painlessly for free, and actually do something practical with it.

 

Not to dissuade you from your project, though. I'm playing with branch-exercises myself just for the sheer interest of it; not as ambitious as a full port, but wouldn't it be fun to see some Mirror's Edge moves or a new weapon. If you're going to do your own stuff for fun, it makes sense to build off something that's open & actually good.

[javiert]

Why?

 

I'll try to explain the whole chain of events

 

Some months ago I co-mentored a master's thesis on AI in games for a CS master student and friend of mine, together with a university professor. The issue was trying to create an AI that could play something similar to Thief, not as a guard, but as the taffer himself One of the problems was what game to use. The AI for RTS community have ORTS, which I don't know how good it is, but at least you can tinker with it. We couldn't find something usable out of the box, so half of the time was spent by the poor girl creating a makshift game engine in erlang (a very nice game for a game engine, I have to say). It would have to include the usual navigation (in old D&D-style square tiles), some rudimentary sensory system (sight and sound, with cool but simple algorithms for an object hierarchy following visual resolution [e.g. a 'rock' and a 'person' both look like 'big lumps' when looked from afar] including lighting and a simple model for sound propagation, taking doors into account) and the AI system. The "clients", whether human or artificial, connected via their own connections and played their "turns"; it was not too difficult to convert the turn gameplay into "realtime" gameplay later on. The AI system, in turn, has to make plans according to the initial goal ("go to room X, steal Y and go out without getting caught"), and includes a memory system to re-check features (like objects) seen during the trip in case further planning requires those features.

 

All in all, quite cool But, it does *not* sem so cool on a sprite-based square tile Java window. So I thought that if I could manage the Dark Mod to run on an open source engine, one could eventually include the cool AI system in a standalone client that does not depend on the server. The open source engine, as I've mentioned before, allows further experimentation like, say, a distributed server or automatically deriving the game code from some formal specification. But I'd be glad just to connect an AI client to an open source engine running the Dark Mod for a start.

[javiert]

That's just if you want to actually do something practical with this, as opposed to it just being an academic exercise, though.

 

That's the point. It's mainly an academic exercise. Maybe my previous post makes things clearer.

 

On the other hand, we could just wait for Doom3 to go open source and get all the advantages painlessly for free, and actually do something practical with it.

 

If Doom3 were open source in, say, May, it would be perfectly fine for me, since I don't expect the server and network code to change that much. But I'm the pessimistic kind of guy that thinks that Open Source Doom3 will celebrate the New Year...

 

Not to dissuade you from your project, though. I'm playing with branch-exercises myself just for the sheer interest of it; not as ambitious as a full port, but wouldn't it be fun to see some Mirror's Edge moves or a new weapon. If you're going to do your own stuff for fun, it makes sense to build off something that's open & actually good.

 

I drool for Mirror's Edge moves in Thief. I weep like a child when the occasional small fence in Oblivion prevents me from jumping over it when it touches my *ankles*.......

 

Anyway, XReal does seem like a fine middle ground until Doom3 finally goes out.

[Sotha]

I'll try to explain the whole chain of events

 

That sounds extremely interesting! Thanks for the explanation and good luck!

[demagogue]

Yes that does sound interesting.

 

I studied some AI in college too, though as part of my broader study in philosophy of mind -- so I studied mostly just the philosophical side, not so much the technical and not game AI. But I've also been toying with AI in Dark Mod.

 

Actually I was thinking about just getting a bot to identify objects on a table, pick them up, and move them around on command, a Shrdlu type system, maybe answer some basic questions about them if I get ambitious (I dream of getting them to rationally thieve, but that's pushing it for me)... Darkmod is cool in having an object manipulation system that I can tap into. Anyway, my experimenting is more about the theoretical model than what it can actually do. I'll be happy if my model can do *anything* rational and meaningful!

 

I'm trying to adopt some of the techniques I've been reading in the neuroeconomics and philosophical literature on semantics, and get as far away as I can from an extensional paradigm for predication -- where it's all just naked predicates, properties, & relations, but the system doesn't really "know" what any of it "means". I just read a book (Bogdan's Predicative Minds) that proposed a model for how predication develops in toddlers learning language -- how they learn that grass is actually green in color, that is, that the word "green" in the context of "grass" is like a normative declaration or command to the goal-system directing its vis-sense (& vis-imagination) systems to the color reflecting off grass blades somewhere out there, which is different than merely GRASS(green), where these two things, grass and greenness, co-occur a la the Frege line of predicate calculus, which is how most AI and formal semantics deal with its semantics, but isn't real predication as a contemporary philosopher would like. And I thought there was enough detail in Bogdan's model to operationalize something out of it and I thought hot damn, I want to try to implement this in an AI architecture and see how far I can go with it. Of course the classic predicate calculus is very efficient and probably overwhelmingly more practical, but again I'm more interested in the philosophical issues of semantics, though I'm not blind to efficiency either.

 

And then I thought Dark Mod is as good a platform as any. I already know its guts pretty well at this point. And actually that's one of the fork-projects I get most excited about and the only reason I didn't mention it above is because it takes paragraphs to even explain it, whereas "Mirror's Edge moves" gets the point across faster, but I think they would be cool too! I'm not doing my project for any formal reason though, just my own innate interest since I studied this stuff in college. But if something cool comes out of it, maybe I'll write an article about it. I think I can get started before Doom3 goes open, but I don't mind waiting so much since I'm thinking about this as a very long term thing.

 

Well now I definitely wish you luck with your project. Maybe you can help me if I have any questions, since I'm not that great with programming really, better with logic and concepts. Not sure I could help you, unless you ever want to chat about the esoteric ins and outs of semantics and meaning for fun.

[javiert]

Not sure I could help you, unless you ever want to chat about the esoteric ins and outs of semantics and meaning for fun.

 

I'm afraid getting a real semantics module (not just the 2.0 crap) would be a real deal of work. Specially if you expect the system to learn new semantics My personal Holy Grail is making an agent talk and think about its own intentions.... and lie about them, theory-of-mind-y style.

 

I'd rather get Mirror's Edge (or Tomb Raider!!!) moves now, and much later on think about making the taffer talk

[demagogue]

Yeah I have no illusions about that!

I don't have a real time scale. It's not even a real project; basically just open-ended piddling.

 

You've got a lot of systems to get working and coordinate. (Just getting a visual system to recognize an arbitrary object in the world, much less what it is or anything about it, is a herculean task. And if it or the bot moves holy shit... I was thinking of porting stuff from the ACT-R system so I have something to build off of.) But actually the interesting part for me isn't actually doing it, but just thinking about what kinds of things you'd have to do to do it, making a design doc, and then maybe trying some stuff out just as proof of concept of little parts of it, or maybe (even likely) none of it will ever even get to the point of code. It's about the path, not the destination. Philosophy thought experiments have a different dev cycle than programming. But the main thing is you want to go straight for the jugular.

 

I agree with you on that holy grail.

[stgatilov]

Why do you need open source engine?

 

Why can't you use the game information from the TDM game part which is open source? Also you need to capture player control, but I think it is also doable without Doom3 source. The game source is stuff like rendering and the like. What do you need there?

 

By the way: some simple hacking of the closed source code can be done. For example, I managed to create a library for bot writing in crimsonland:

 

 

And here is the bot from Wide Siberian Open programming olympiad connected to this library playing crimsonland:

 

 

 

 

 

 

[nbohr1more]

Here is a Doom3world thread about creating bots for Doom 3. So added to stgatilov's comment... yeah you could probably still do what you want without GPL (the current SDK is open source...):

 

http://www.doom3world.org/phpbb2/viewtopic.php?f=56&t=9940&st=0&sk=t&sd=a

[javiert]

Why do you need open source engine?

 

I know I could somehow tap in the middle and do some hacks here and there. Actually, that's what I think the guys developing Pogamut (https://artemis.ms.mff.cuni.cz/pogamut/tiki-index.php) are doing. But I'd rather make my bots connect directly to the server instead of passing through several TCP pipes and processes if it's possible. As I said, it's mor than enough for me to be able to connect to a server and move through the map.

 

The issue I see with the SDK is that the control here lies in the SDK itself. That is, if the SDK were a client library I could use to connect to the game, load the map, etc, that would be a very different story. But right now the SDK runs your code.

 

I'll look into the client-side SDK anyway, it will surely save me some time when Doom3 goes open source.

 

BTW, just for flattering puposes, The Dark Mod was my main reason for buying Doom3