Jump to content
The Dark Mod Forums

DarkRadiant meets Python


Recommended Posts

Last weekend I followed an idea I had months ago, about adding scripting support to DarkRadiant. I experimented a bit with boost::python and I must say I'm pretty impressed with how fast this went. It's rather easy to expose existing classes to Python, so that the script can call any member method of that instance. It was a bit more complicated to expose DarkRadiant's Global* interfaces (GlobalSelectionSystem, GlobalSceneGraph, etc.) to scripting, but this is working now too. Even the math objects like Vector3, Vector4 and AABB are already available in the script file including operator (+, -, +=, -=) and constructor overloads. C++ std::map can be traversed with for element in map loops in Python, thanks to boost's map_indexing_suite.

 

I also managed to let Python implement an abstract scene::NodeVisitor class and let the script traverse the scenegraph (performance is quite ok too). This is rather cool, as changes to the script are taking effect instantly (without restarting or recompiling DarkRadiant), just call the script again and Python parses it on the fly.

 

What's needed for this?

The script.dll module (I designed this as optional plugin) and the python25.dll for the Windows version. For compiling, it's necessary to have the boost::python headers and static libraries available (which is not too hard) and the Python headers, which I plan to push into the w32deps folder.

 

What's the plan?

I plan to push some of the more high-level algorithms to Python (an example would be the "Bulge Patch" algorithm, which is not performance critical, or other texture alignment algorithms).

 

Ideally, it would be possible to just copy a new script file into DarkRadiant's scripts/ folder and the functionality is immediately available. The editor can be extended rather quickly, and also by people who don't have C++ experience. Of course, basic Python knowledge is required. I hardly know Python myself so far, but it's rather easy to learn and much less strict.

 

I still have to expose a few more Global* interfaces, but this is a rather quick process. Once this is done, I want to expose the Patch and Brush classes and then think about linking Commands and shortcuts to Python scripts.

Link to comment
Share on other sites

Cool dude,

 

DR keeps coming along really nicely.

|=-=------=-=|

happycheeze.deviantart.com

 

Moddb

 

Gamers Outreach, a nonprofit that uses videogames to raise money for chairty.

|=-=------=-=|

Link to comment
Share on other sites

Cool beans!

 

You make it sound like it was a walk in the park to implement, but I'm sure you're not giving yourself enough credit there. :)

My games | Public Service Announcement: TDM is not set in the Thief universe. The city in which it takes place is not the City from Thief. The player character is not called Garrett. Any person who contradicts these facts will be subjected to disapproving stares.
Link to comment
Share on other sites

Oh, it was definitely not that easy, but I was expecting far more problems. The boost::python guys have my deepest respect, they really had their "customers" (e.g. me as a coder) in mind when creating their interface.

 

Part of the problems I faced was my newbieness to Python. I had to learn about the most primitive Python syntax and other stuff before I could even test my script, so there was some googling and trial and error involved. I assume somebody more knowledgable to Python would have been even faster.

 

I already merged my current state of code to the trunk, so I do hope everything compiles on Windows systems without any Python installation (should work). I haven't even looked into the Linux side of things - I assume creating a automake file with dependencies to boost::python plus adding the libboost-python1.34.1 and python-dev packages to the dependencies should do the trick.

 

OrbWeaver, would you be motivated to do another makefile? If not, that's fine as well, I'll try to do that sometime myself then.

Link to comment
Share on other sites

OrbWeaver, would you be motivated to do another makefile? If not, that's fine as well, I'll try to do that sometime myself then.

 

I'm sure I can manage that; Automake just has to check for the header and libraries and then the DEB/RPM packaging stage actually figures out the package dependencies from the compiled executable.

Link to comment
Share on other sites

@OrbWeaver: thanks for the build script, I saw your commits. :)

 

I had to think again about the scripting plugin being optional: what if I move algorithms from C++ to Python and let Events/Commands point to scripts instead? This would make the scripting plugin a mandatory one, wouldn't it? How does Blender handle this?

Link to comment
Share on other sites

If there is a valid architectural reason why the Python plugin should be mandatory, then there is no problem with making it so.

 

However, I would recommend giving consideration to how features could be kept optional if possible, to allow flexibility in building and packaging[1] the application. For example, could commands that depend on Python detect the availability of the script module before registering themselves in the GUI?

 

(I don't know how Blender handles this; I have only compiled it from source once and didn't pay much attention to the build options. AFAIK Blender contains its own Python interpreter, which may or may not be a compile-time build option).

 

[1] If modules are optional they can be placed into separate DEB/RPM packages, which will prevent the main DarkRadiant package having too many package dependencies. This would help if the user wanted to install DR on a system but could not install the required Python package for some reason.

Link to comment
Share on other sites

If there is a valid architectural reason why the Python plugin should be mandatory, then there is no problem with making it so.

At the moment, I'd tend to go into that direction, but I'll give it more thought before moving on.

 

However, I would recommend giving consideration to how features could be kept optional if possible, to allow flexibility in building and packaging[1] the application.

In fact, I figured that the Python scripting ability would help moving towards features being optional, as adding a feature would be a synonym of copying a new .py file into the scripts/ folder. Drag-and-drop the .py file there and rescan the scripts and the feature is there - at least that's how I envisioned it to work.

 

For example, could commands that depend on Python detect the availability of the script module before registering themselves in the GUI?

It surely is possible to detect whether the MODULE_SCRIPT module is available, no problem. I could also move the "python feature-plugin" code to the script module, so that the .py files only get considered if the plugin is activated in the first place. If the script module is not there, the python files are ignored and all the functionality would be missing.

 

For user convenience, I'd rather have the script plugin packaged by default, so that DarkRadiant is ready for new scripts right from the start, without the user having to recompile it or to download additional binaries and mess around in the program folder (which is where things go wrong, usually). Unless this is a problem for the Linux packaging process, but it sounds like this can be resolved by adding a Python dependency to the .deb?

 

In Windows builds, I was able to simply add the Python interpreter by including the "python26.dll" in the install folder, the script.dll loads this file if it's there (and throws an error if it's not there, but DarkRadiant can still start up). The user doesn't need to have Python installed on the actual system, as far as I could see, it works with just this DLL binary.

 

I don't know - would be it be considered obnoxious if DarkRadiant was only able to run with Python available, from a Linux user's point of view? Is the Python interpreter installed on all, say, Ubuntu distributions by default?

 

I assume Python is available for all kinds of platforms, isn't it?

Link to comment
Share on other sites

I don't know - would be it be considered obnoxious if DarkRadiant was only able to run with Python available, from a Linux user's point of view? Is the Python interpreter installed on all, say, Ubuntu distributions by default?

 

I assume Python is available for all kinds of platforms, isn't it?

 

Perl is almost always installed, but Python is often optional. I don't know about Ubuntu, but I know that Suse had python not in it by default in the past.

 

(Btw, can we get a Perl module, too? I don't know Python and I don't have any desire to learn yet-another-syntax...)

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

In fact, I figured that the Python scripting ability would help moving towards features being optional, as adding a feature would be a synonym of copying a new .py file into the scripts/ folder. Drag-and-drop the .py file there and rescan the scripts and the feature is there - at least that's how I envisioned it to work.

 

If a sufficient number of features are to be moved to Python such that DR's functionality becomes limited without it (which I certainly have no problem with, this is an increase in flexibility not a reduction), then this probably counts as a valid architectural reason to depend on Python.

 

It surely is possible to detect whether the MODULE_SCRIPT module is available, no problem. I could also move the "python feature-plugin" code to the script module, so that the .py files only get considered if the plugin is activated in the first place. If the script module is not there, the python files are ignored and all the functionality would be missing.

 

That sounds like a better design even if the plugin is always packaged by default.

 

For user convenience, I'd rather have the script plugin packaged by default, so that DarkRadiant is ready for new scripts right from the start, without the user having to recompile it or to download additional binaries and mess around in the program folder (which is where things go wrong, usually). Unless this is a problem for the Linux packaging process, but it sounds like this can be resolved by adding a Python dependency to the .deb?

 

The DEB dependencies are calculated automatically by DPKG, so this is not a problem for the build process. The only difference is whether the main darkradiant package has dependencies on Python, or whether it is split into (for example), darkradiant, darkradiant-plugins-darkmod and darkradiant-plugins-python, with only the last one having a dependency on Python.

 

If, for example, a user had compiled their own version of Python or Boost.Python for some other purpose, they may not want to change the installed package version in order to install DarkRadiant, so having a separate optional plugin would help here. It is largely a theoretical risk though, and has to be balanced against the user expectation of having core functionality available by default without needing to install extra plugins (although dpkg's "Recommended packages" functionality is intended to address this sort of issue to some extent).

Link to comment
Share on other sites

(Btw, can we get a Perl module, too? I don't know Python and I don't have any desire to learn yet-another-syntax...)

 

I'm not sure if you're being serious, but Python is a good choice here because the Boost.Python library makes it easy to add scripting support to a C++ application.

 

While there may well be equivalent libraries for Perl, I don't think adding even more build dependencies to a product just to avoid people having to learn a (very common) scripting language is a very rational approach.

Link to comment
Share on other sites

Yes, boost::python for teh win - I don't know if there is a similar integration for Perl available. I surely must admit that I don't know enough about Perl to say whether there are ways to reflect the proper C++ classes plus inheritance plus abstract-ness in Perl, but I'm sceptical. As Python is object-oriented, it makes for a better candidate overall, I suppose.

 

But either way, given the work it takes to expose all the proper C++ classes to Python, I'd rather do that only once, if you understand. Sorry, but this is too much to ask. ;)

 

Also, from a non-Python and non-Perl programmer's view I can say that Python scripts have a far easier-to-read syntax than Perl has, looking at the few scripts I saw so far. Generally, I'm always willing to learn new languages, and it appears that Python will be that next new language for me.

Link to comment
Share on other sites

Ok, I think the framework is mostly complete now. It's possible to add new commands to DarkRadiant by coyping the .py files to the install/scripts/commands/ folder and hit "Reload Scripts".

 

An example file can look like this example.py:

# Set the command name so that DarkRadiant recognises this file
__commandName__ = 'Example'

# The actual algorithm called by DarkRadiant 
# is contained in the execute() function
def execute():
shader = GlobalShaderSystem.getShaderForName('bc_rat')
print(shader.getName())

# The variable __executeCommand__ evaluates to true 
# when DarkRadiant executes this command
if __executeCommand__:
execute()

Link to comment
Share on other sites

I'm not sure if you're being serious, but Python is a good choice here because the Boost.Python library makes it easy to add scripting support to a C++ application.

 

While there may well be equivalent libraries for Perl, I don't think adding even more build dependencies to a product just to avoid people having to learn a (very common) scripting language is a very rational approach.

 

As I stated already a year back or so, my time is so limited that I rather do something productive instead of spending all the available time (and some) to learn yet-another-scripting language. Already learned a new one this year and I seriously try to limit myself to actually getting work done instead of just learning new things for the sake of "learning something new".

 

Things like "oh it it is easy and you pick it up in no time" are actually not true, it should be clear to anyone that learning a new programming language takes time, effort and even then you will not get beyond the "beginners" level for quite some time. So the decision becomes then contributing quality code in a language you already know, or contributing very little code in a beginners quality - just so you learned a new language.

 

For myself, I decided to do the former. :)

 

Anyway, Python is certainly a welcome addition to DR - for those who are able to work in it. :)

 

(We would have the same discussion in case you had added Perl because there are bound to be people who know Python but not Perl).

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Yes, boost::python for teh win - I don't know if there is a similar integration for Perl available. I surely must admit that I don't know enough about Perl to say whether there are ways to reflect the proper C++ classes plus inheritance plus abstract-ness in Perl, but I'm sceptical. As Python is object-oriented, it makes for a better candidate overall, I suppose.

 

But either way, given the work it takes to expose all the proper C++ classes to Python, I'd rather do that only once, if you understand. Sorry, but this is too much to ask. ;)

 

Also, from a non-Python and non-Perl programmer's view I can say that Python scripts have a far easier-to-read syntax than Perl has, looking at the few scripts I saw so far. Generally, I'm always willing to learn new languages, and it appears that Python will be that next new language for me.

 

That's entirely understandable and I didn't want my post to be read as a "whining" or "command", just a humble request to consider the possibility. Certainly I won't get in your way.

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

Things like "oh it it is easy and you pick it up in no time" are actually not true, it should be clear to anyone that learning a new programming language takes time, effort and even then you will not get beyond the "beginners" level for quite some time.

No, I disagree. Learning a new programming language, at least to the base level of proficiency necessary to use it for scripting-type tasks, is not difficult. Assuming it's a half-decent programming language.

 

With Python, you just have to get over the lack of braces and then you're laughing. It's a very natural-feeling language. You just need to learn the basic syntax and have a library reference handy and you'll be productive in no time.

 

Anyway, it's not like you were contributing much code to DarkRadiant last time I looked, and you already know C++, so is there really any guarantee that adding a Perl scripting interface would help you be more productive on DR? :) (Not a criticism, just an observation.)

My games | Public Service Announcement: TDM is not set in the Thief universe. The city in which it takes place is not the City from Thief. The player character is not called Garrett. Any person who contradicts these facts will be subjected to disapproving stares.
Link to comment
Share on other sites

No, I disagree. Learning a new programming language, at least to the base level of proficiency necessary to use it for scripting-type tasks, is not difficult. Assuming it's a half-decent programming language.

 

With Python, you just have to get over the lack of braces and then you're laughing. It's a very natural-feeling language. You just need to learn the basic syntax and have a library reference handy and you'll be productive in no time.

 

"no time" was about one year real-time for me learning Javascript - I doubt python is much faster :) Can you wait a year?

 

Anyway, it's not like you were contributing much code to DarkRadiant last time I looked, and you already know C++, so is there really any guarantee that adding a Perl scripting interface would help you be more productive on DR? :) (Not a criticism, just an observation.)

 

I don't have an account on the SVN that is used for DR which might explain my lack of commits :)

 

Also, I know almost no C++, which is a real hindrance working on DR. I started a few attemtps but got horrible lost for even the smallest changes (like adding a new dialog or even just changing a dialog into a splitpane). That might explain it even more :)

 

However, working with scripts and producing level geometry via them is something I know a thing about, or two. ;)

 

There was this little project named "Cowgen" which generated level geometry for Thief 1/2 by writing out level files... one can still find it refernenced in a few places like here http://www.thiefmissions.com/telliamed/ or here http://www.ttlg.com/FORUMS/showthread.php?t=26524).

 

I can dig out a copy if anyone is interested (but be warned, it is horrible outdated, because it is from somewhere around 1998 or so....)

 

One could say that I was quite early creating level geometry programatically, and that is what the new Python (or a never-existing Perl) interface would also allow...

 

however, currently I am struggling to find time to even read all forum posts so you can prolly ignore me completely. wish I had more time :-(

"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

 

"Remember: If the game lets you do it, it's not cheating." -- Xarax

Link to comment
Share on other sites

@OrbWeaver: what would Linux users need to compile DarkRadiant including the script plugin? I assume they need a python-dev package and the libboost-python-dev one? Anything else?

 

Otherwise I'm going to add these to the compilation guide on the wiki. Does the automake script automatically detect whether Python headers are installed on the system and compile the plugin if positive?

Link to comment
Share on other sites

@OrbWeaver: what would Linux users need to compile DarkRadiant including the script plugin? I assume they need a python-dev package and the libboost-python-dev one? Anything else?

 

As far as I can remember, that is all. In Ubuntu (and probably other distributions) libboost-python-dev requires python-dev anyway.

 

Otherwise I'm going to add these to the compilation guide on the wiki. Does the automake script automatically detect whether Python headers are installed on the system and compile the plugin if positive?

 

Yes, it checks for libboost_python headers and disables the plugin if they cannot be found; the configure script prints out the optional compile results at the end of its checks.

Link to comment
Share on other sites

  • 2 months later...

I extended the scripting system a bit, so that the plug-in adds a new tab to the group dialog when the module is loaded:

 

scripting_tab.jpg

 

Of course, one needs to know the syntax, but I guess this could be resolved by providing a small script snippet library on our wiki or something. Just copy & paste the scripts in there and hit the run button.

 

Now I only need a tiny tad more time to add all the script API methods I have on my mind...

 

edit: This is a possible model replacement script:

count = 0

class SceneWalker(SceneNodeVisitor) :
def pre(self, node):
global count
ent = node.getEntity()
if not ent.isNull():
	if ent.getKeyValue('model') == 'oldmodel.ase':
		ent.setKeyValue('model', 'newmodel.ase')
		count += 1
	return 1

walker = SceneWalker()
GlobalSceneGraph.root().traverse(walker)

print(str(count) + ' Models replaced')

Link to comment
Share on other sites

I've fixed up the Linux build script so that the latest changes compile, but I'm not seeing a Script tab in the group dialog. Is there something I need to do on Linux to enable it?

 

EDIT: Never mind, there was another missing file. Fixed now.

Link to comment
Share on other sites

No problem, finding out which .cpp files need to be added to the Makefile.am is pretty easy to do. The slight gotcha is that a missing symbol in a module doesn't become apparent until run-time, hence the almost-question.

Link to comment
Share on other sites

@OrbWeaver: I was experimenting with the GtkSourceView library, and I could get it to work after some hassles:

 

sourceview.jpg

 

The library seems to be well maintained (is used by gedit) and is quite flexible. It would, for instance, be possible to add a simple syntax file for .mtr and D3 .script files and it would be easy to add a "material viewer" to DarkRadiant with syntax highlighting included (RMB on a shader and select "Show Shader Definition" or something like).

 

If we want to keep or expand on this, I could further look into this.

 

The questions are now:

 

- Do we want such functionality at all?

- Is this GtkSourceView stuff available in Ubuntu as dependency?

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recent Status Updates

    • The Black Arrow

      Hope everyone has the blessing of undying motivation for "The Dark Mod 15th Anniversary Contest". Can't wait to see the many magnificent missions you all may have planned. Good luck, with an Ace!
      · 0 replies
    • Ansome

      Finally got my PC back from the shop after my SSD got corrupted a week ago and damaged my motherboard. Scary stuff, but thank goodness it happened right after two months of FM development instead of wiping all my work before I could release it. New SSD, repaired Motherboard and BIOS, and we're ready to start working on my second FM with some added version control in the cloud just to be safe!
      · 2 replies
    • Petike the Taffer  »  DeTeEff

      I've updated the articles for your FMs and your author category at the wiki. Your newer nickname (DeTeEff) now comes first, and the one in parentheses is your older nickname (Fieldmedic). Just to avoid confusing people who played your FMs years ago and remember your older nickname. I've added a wiki article for your latest FM, Who Watches the Watcher?, as part of my current updating efforts. Unless I overlooked something, you have five different FMs so far.
      · 0 replies
    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 4 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
×
×
  • Create New...