Jump to content
The Dark Mod Forums

Parsing problem in a DDS file at mission start


grayman

Recommended Posts

Need help

C:\dev\darkmod\devel\auto\scripts>python repeat.py enter_game_exit.seq
Traceback (most recent call last):
  File "console.py", line 7, in <module>
    from automation import automation, tdm_process, commands_runner
  File "..\automation\__init__.py", line 2, in <module>
    from .tdm_process import tdm_process
  File "..\automation\tdm_process.py", line 2, in <module>
    import psutil, os, time
ModuleNotFoundError: No module named 'psutil'
FAILED!

 

Link to comment
Share on other sites

Opening files is not threadsafe. In BFG it is the same.

Basically, there is a set of functions which return a string as "const char*", pointing to internal buffer. I wonder what should we do with them to make them threadsafe but avoid allocation. Maybe make internal buffer threadlocal? It should not be too costly.

UPDATE: 20 matches of static char ???[???] across the code. I think changing signature is not a good option, since it would require fixing a lot of usages...

Link to comment
Share on other sites

3 minutes ago, duzenko said:

It works now

How much time should I give it? First few tries show no crash

Something like 30 attempts should be enough for sure (look at the console files generated --- they are numbered), usually it crashes on 1-5 attempt.

You might also want to use Grayman's config file to have similar conditions. BTW, different VS may be important too.

Link to comment
Share on other sites

14 minutes ago, stgatilov said:

I think it should be wrapped into a critical section it you want to open files form different threads...

I think that function is not called for files in pk4's? And both the .map and loading background .tga are packed.

Can you run

disconnect; purgeimages; loadgame

looped in game console automatically?

Link to comment
Share on other sites

Quote

Can you run ... loopped in game console automatically?

You can create a .seq file, where just write your console commands, one per line. All lines without special character (@  or $ or #) are passed directly to game console. Then you can execute this seq file by running console.py mysequence.seq. It will not repeat by itself,but may be useful. Besides, you can always copy/paste commands in the .seq file ? Or you can use any utility you like to repeatedly run console.py mysequence.seq (batch/shell/python).

Note that you can also "call" existing .seq-files like @file mysequence.seq. Sometimes it is convenient for composability. So you can start console.py without any argument, then call your script from there --- the console won't close automatically after the script is over.

Anyway, the simplest way to repeat same script over and over again is to execute repeat.py mysequence.seq. Basically, it simply executes console.py mysequence.seq 100 times in a row, checking for error code every time (to stop on crash). No need to copy/paste or use batch :D It does not restart TDM by itself --- I had to add "exit" command to make it restart.

UPDATE: It turned out that console.py does not finish automatically when a script is over. So you have to add $sys.exit(0) or @finish at the end of your script to make it work with repeat.py.

P.S. I don't know if builtin game console supports repeat. Maybe it does. Maybe you can add support for it like: repeat "disconnect; purgeimages; loadgame" 100 (easy to do if quoted text is passed as single argument).

Link to comment
Share on other sites

I have tried to add critical section over the whole OpenFileReadFlags and OpenFileWrite methods. I have run 45 repeats without a crash already. So I think it is a good fix to lock a mutex in most of the methods of idFileSystem, and allow only simple reading/writing concurrently.

Meanwhile, I'm not certain which locking mechanism should I use. There is Sys_EnterCriticalSection which works with globally numbered sections, and Sys_MutexLock which can work with locally created lock. The global numbering across locks allows to avoid deadlocks by checking that locks are taken in order of increasing number in all possible call stacks. However, if there are many places to have locks, then this list of critical sections can turn into a mess pretty soon.

Link to comment
Share on other sites

I think I found the 'state' you were looking for

It's the zip file current position, changed on line 2725 in FileSystem.cpp

I leave it to you how to best address this. Maybe the modern version of our Zip library can do this better in another way.

Off to midnight mass now

Link to comment
Share on other sites

I'm afraid newer version of the library won't help. Also, changing to a newer library is not an option.

I think minizip allows to have several unzFile handles into one pk4 file and use them in parallel. In fact, D3 code already does that: since you can open 10+ files in one Zip and store them anywhere, they duplicate unzFile handle using unzReOpen. Therefore right now only opening a file has to be guarded --- reading it is already OK, as long as you don't share the particular file between threads (which is logically wrong anyway).

Link to comment
Share on other sites

I assume only unzSetOffset64/unzGetCurrentFileInfo64 need sync point? Or what is the point of unzSetOffset64 here?

By newer version I meant that D3's original zip interface could be thread-unsafe but since that the lib might have evolved in this regard and now offer some other approach to do this?

Link to comment
Share on other sites

I have created an issue: 5079

Also, hopefully fixed the problem in the latest revision of code SVN. Grayman, I guess it is trivial for you to build an executable yourself?

 

Here is what remains to be done here:

On 12/24/2019 at 11:18 AM, stgatilov said:

I think the main question to Duzenko is: why is background thread doing anything at all?

I recall you said that the background thread is used only when cvar is enabled. I don't see suspicious cvar values in the config file.

UPDATE: I think first of all background thread should be fixed: it must be trivial to validate that it does absolutely nothing or is not spawned at all.

I recall that I tried to validate that background thread does nothing under cvar when I looked through related commits. But I failed to verify it, since the place where requests are put into thread's queue has not check for the cvar. Then I even put a breakpoint into the background thread and noticed that it loads a few images. I regret I did not raise alarm immediately ?

Link to comment
Share on other sites

1 hour ago, stgatilov said:

Here is what remains to be done here:

I recall that I tried to validate that background thread does nothing under cvar when I looked through related commits. But I failed to verify it, since the place where requests are put into thread's queue has not check for the cvar. Then I even put a breakpoint into the background thread and noticed that it loads a few images. I regret I did not raise alarm immediately ?

Svn rev 8467

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recent Status Updates

    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 4 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
    • JackFarmer

      @TheUnbeholden
      You cannot receive PMs. Could you please be so kind and check your mailbox if it is full (or maybe you switched off the function)?
      · 1 reply
    • OrbWeaver

      I like the new frob highlight but it would nice if it was less "flickery" while moving over objects (especially barred metal doors).
      · 4 replies
    • nbohr1more

      Please vote in the 15th Anniversary Contest Theme Poll
       
      · 0 replies
×
×
  • Create New...