Searching existing maps

[SteveL]

The second tracker issue I want to ask about:

0002867: "Search in files" functionality

Sometimes I want to know whether a certain entity/model/texture is already used in any of the released fms. It would be nice if DR could search map files in a certain folder (preferrably in pk4s) for them.

 

This seems an ideal job for a python script in DR. The python interface already comes with methods to let a script explore a darkmod installation, including inside pk4s.

 

I imagine this one popping up a filter menu where you can choose texture / entity / model (anything else?). The specific texture or entity type to search for could default to whatever the mapper has selected.

 

Perhaps it should let users add additional filters if they want: e.g. entity selection could have an "advanced options" button that'd let you match spawnargs too. This could sometimes answer the question "which existing maps would such-and-such a change break?". Only sometimes of course. There are plenty of things you can't work out from just parsing the map file.

 

One example where it would work: On the other tracker issue I just posted (a target entity to change objective amounts) it's tempting to wonder whether the "Amount" setting on objective components could be extended to work for items too (currently it doesn't. The objective will complete the first time you get an entity of the right spec without counting how many you've got). In the game code, there's a comment that indicates whoever wrote the code was considering adding that functionality later. The problem with adding it now would be that there could be maps out there that would break. A mapper could have tried setting up "Get 3 apple cores" using amount "3", found it didn't work, and made a workaround but leaving that amount=3 setting in place. That's a situation that could be tested by searching map and def and possibly script files.

 

Questions:

• Does this tool already exist in some form? I'm sure I've seen people say things like "this model is only used in 4 existing maps"
  
• Anything else useful that it could do?
  

[grayman]

This is a good idea.

 

I've been handling this with this setup:

 

I have a folder where I keep copies of every *.map that's been released.

 

I use Windows Grep to look through the map files for whatever.

 

I don't use the darkmod/fms/<mission>/<mission>.pk4 files because my darkmod folder is too fluid, and there's no guarantee that all released maps are present there at any given moment.

 

(This was the reason I griped about mission *.map filenames not matching or representing their mission names. For example, if I find what I'm looking for in "mc.map" it's not obvious what mission that's from.)

 

I do the same thing with *.script files, to see if someone's used a particular function in the past.

 

This is an essential process to help with the goal of not breaking existing missions with new code.

[SteveL]

Cool. So it needs to let you specify a search folder. Fortunately that selection can be saved between script calls and DR sessions. Re using grep, Python will make it easy to support regular expression searches.

 

Yeah, I see the problem with unhelpfully named maps. Probably best to use this with a folder of pk4s, although it would work with both. I might make a separate python script that can purge a folder of pk4s of all binary files and then recompress the zips, for test purposes and for anyone else who wants to use it. That would only be a loop and a couple of calls to library functions.

[RJFerret]

...choose texture / entity / model (anything else?).

 

Sound, please.

[SteveL]

I've started researching this one. An immediate problem came up: it turns out the DR python interface is set up to let you explore your main darkmod installation, not mission pk4s. Basically you get access to the same set of files that DR uses to populate the entity and media browsers and so forth. It'll tell you the names of any other pk4s it discovers, but you don't get to search them.

 

Not only that, but the python implementation doesn't include the modules you'd normally use in python for searching file paths or reading zips. I know absolutely nothing about implementing python in another app, so I don't know whether there's an easy option of supporting the zipfile module in a later release.

 

There *is* a workaround possible. Python has an internal read-only zip handler that it uses to load python modules stored in archives, in exactly the same way that TDM loads its resources from pk4s. That can be got at by a script, and I've successfully tested misusing it to extract the map, script, and def files from the training mission. It's not a supported use of the package (and you have to read internal private data from the module to get the list of contents in the pk4), but the worst that could happen is that the script stops working one day in the future if the python implementation changes. Not a deal-breaker in my opinion.

 

The script would still have to rely on DR's virtual file system to get the list of pk4s. That means that anyone wanting to maintain a separate archive of released FMs will have to store it somewhere in their mod base or engine path folders. Restricting the search *within* those paths is fine, and the script will be able to remember the setting.

 

Still worth doing?

[grayman]

I'll probably continue with the method I'm using now. I don't want to move my map and script folders under my darkmod folder because I work with different versions of darkmod at various times.

[SteveL]

Yeah I agree that takes the shine off it. I'll take a step back and go do the research on how difficult it would be to support the extra file-handling modules in the embedded python interpreter. I don't think it's out of the question because some file operations are already supported, so it's not like scripts are isolated from the file system. It seems better to implement this through the python interface than through the DR file handling code, given that that's set up to do a different and quite specific job.

[grayman]

Pls make sure that whatever you're doing matches what other folks would use. I'm sure mappers will take whatever help they can get as long as it doesn't mess with their work procedure. New mappers who haven't quite settled into a routine would be particularly grateful, IMHO.

 

You don't need to accommodate my situation, because what I'm doing now works quite well for me, and I'm not looking to have DR take over the job.

[SteveL]

Fair point. I'm conflating two different tools, because when I saw this issue I thought it was an ideal chance to kill 2 birds with one stone. I'd been wanting to make a map-breaker searcher of some kind anyway. But there's no need to shoehorn that one into DR now it turns out that it's an awkward fit. A simple tool for people to use find usage examples would be very useful in its own right. It could have saved me hours myself, like when I couldn't figure out how to use a func_portal, for example.