Jump to content
The Dark Mod Forums

Parsing problem in a DDS file at mission start


grayman

Recommended Posts

Yes, my WIP.

I tried "The Bakery Job" with a build of the latest SVN sources, and this happened:

bakery.thumb.jpg.5e2ac144de8a70c5c17c3adcf7b44563.jpg

 

bakery.map certainly doesn't contain "unknown punctuation".

 

These issues don't happen all the time.

I'll try with the debugger suggestion.

 

Link to comment
Share on other sites

Any more thoughts on this?

The latest SVN Windows exes are crapping out for me on these parsing problems. Tried The Gatehouse and Business as Usual (again).

Becoming an issue because I can't rely on being able to run SVN for anything: playing the game or working on WIPs. I've shelved work on the Steele WIPs because from here on they rely on 2.08+. Just to keep working, I've started a new non-Steele WIP that relies on 2.07.

 

Thanks.

  • Like 1
Link to comment
Share on other sites

8 minutes ago, duzenko said:

We had a build config called 'memory log' or something. What was it supposed to do?

Well, it enables various defines.

You can see the defines in BuildDefines.h. There also a bit of writeup here, but it is more focused on TypeInfo, which is not necessary.

You can temporarily uncomment ID_DEBUG_MEMORY and probably ID_REDIRECT_NEWDELETE in Debug build, that should be enough to enable whatever debug heap which is provided by ID code. And it won't require typeinfo as the "with memory log" requires right now.

 

However, note that TDM was developed by many people over 15 years, and nobody cared about such tools. So don't be surprised to get all sort of problems with them. I guess people often directly use malloc/free and operator new/operator delete for allocating memory. The define ID_REDIRECT_NEWDELETE is a very hacky attempt to fix that, but it probably does not influence STL usage, etc. Some universal tool may be more precise.

Link to comment
Share on other sites

One of the things which came to my mind is string pools.

All idDict-s store keys and values inside a string pool. If you access same string pool from many threads, that is a problem. The correct approach would be to have several pools indexed by thread ID or TLS index. Another way is wrapping all string pool functions into a critical section, but I think it is inferior. Anyway, unless I can see the crash, I cannot say anything for sure.

Link to comment
Share on other sites

1 hour ago, duzenko said:

There's a stack trace above

That's not much. But stack trace confirms the idStrPool theory: it crashes on access to a string inside idDict. A crashdump will help, because it embeds stacks of all threads, and the state of heap, where it is sometimes possible to notice something.

Config file is the most wanted thing here. I won't be surprised that with right settings, we will reproduce the problem on our machines.

Link to comment
Share on other sites

Ok, I have reproduced the problem. Although I see nothing suspicious in config. I'll commit automation script to assets SVN shortly. You can run it is python repeat.py enter_game_exit.seq from the directory where you see the .seq-file.

This is heap corruption due to multithreading. The newly introduced "background image loading" thread loads a TGA file of menu background. Somewhere during loading, it gets a path "C:\TheDarkMod\darkmod\fms\business3\guis\assets\loading_screen.tga". In the main thread, this string somehow gets into a local idStr of idFileSystemLocal::OpenFileReadFlags. When this string is destroyed, it notices that someone wrote past the end of buffer.

So we have to review the whole idStr for working properly with multithreading.

Link to comment
Share on other sites

I think the main question to Duzenko is: why is background thread doing anything at all?

I recall you said that the background thread is used only when cvar is enabled. I don't see suspicious cvar values in the config file.

UPDATE: I think first of all background thread should be fixed: it must be trivial to validate that it does absolutely nothing or is not spawned at all.

Link to comment
Share on other sites

4 hours ago, stgatilov said:

I think the main question to Duzenko is: why is background thread doing anything at all?

I recall you said that the background thread is used only when cvar is enabled. I don't see suspicious cvar values in the config file.

UPDATE: I think first of all background thread should be fixed: it must be trivial to validate that it does absolutely nothing or is not spawned at all.

If idStr is not thread safe then the we need to fix it. Yeah the imageloading thread must not do anything with the cvar disabled but that's not the problem here.

Link to comment
Share on other sites

Do you think maybe it's idFileSystemLocal  or idLexer that is unsafe?

Are you running TDM with pk4's or uncompressed? It could be the pak linked list in FileSystem

idFileSystemLocal::BuildOSPath >> static char OSPath?

Quote

this string somehow gets into a local idStr of idFileSystemLocal::OpenFileReadFlags

What local string? What is its C++ identifier? The only local idStr I can see - netpath - is not used for files in pk4's.

BTW still not able to repeat it.

Link to comment
Share on other sites

7 hours ago, duzenko said:

If idStr is not thread safe then the we need to fix it. Yeah the imageloading thread must not do anything with the cvar disabled but that's not the problem here.

I hope you will fix it in some nearest future.

Yes, idStr better be thread-safe. I'll look into it, of course.

As for idDict and idStrPool... I'm not certain. Looking at the D3BFG code, they did not make custom allocators thread-safe. I see no atomic operations and mutexes in Heap.h/cpp, and non of that in idStrPool and idDict. Making all that stuff thread-safe will make it significantly slower. I guess they simply fixed stupid non-thread-safety like static and global variables, and ensured that access to custom allocators and string pool is single-threaded (e.g. from game code only).

 

Quote

still not able to repeat it.

Did you try to run the repeat.py script? I ran it directly on SVN with Win64 Debug build. I cannot repeat the debug heap dialog message, since most of the time TDM process simply "disappears" somewhere during loading. I assume it is a crash, but I have disabled something like WER or JIT in my system, so I don't see the message box. I guess I should reenable it somehow.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recent Status Updates

    • Ansome

      Finally got my PC back from the shop after my SSD got corrupted a week ago and damaged my motherboard. Scary stuff, but thank goodness it happened right after two months of FM development instead of wiping all my work before I could release it. New SSD, repaired Motherboard and BIOS, and we're ready to start working on my second FM with some added version control in the cloud just to be safe!
      · 0 replies
    • Petike the Taffer  »  DeTeEff

      I've updated the articles for your FMs and your author category at the wiki. Your newer nickname (DeTeEff) now comes first, and the one in parentheses is your older nickname (Fieldmedic). Just to avoid confusing people who played your FMs years ago and remember your older nickname. I've added a wiki article for your latest FM, Who Watches the Watcher?, as part of my current updating efforts. Unless I overlooked something, you have five different FMs so far.
      · 0 replies
    • Petike the Taffer

      I've finally managed to log in to The Dark Mod Wiki. I'm back in the saddle and before the holidays start in full, I'll be adding a few new FM articles and doing other updates. Written in Stone is already done.
      · 4 replies
    • nbohr1more

      TDM 15th Anniversary Contest is now active! Please declare your participation: https://forums.thedarkmod.com/index.php?/topic/22413-the-dark-mod-15th-anniversary-contest-entry-thread/
       
      · 0 replies
    • JackFarmer

      @TheUnbeholden
      You cannot receive PMs. Could you please be so kind and check your mailbox if it is full (or maybe you switched off the function)?
      · 1 reply
×
×
  • Create New...