Analysis of 2.12 TDM Fonts

[Geep]

I have updated my Refont program, to now have a function that can analyze a font DAT file for missing or problematic characters. As part of a broader inquiry, I've just applied that program, individually, to all current 'english' font DAT files. I'm reporting the overall results here. I'm sure this will not be a surprise to some of you, but may be to others.

Background

As you know, TDM fonts are based a bitmap system, derived from 256-character code pages, of which "English" and "Russian" are currently supported. "English" is actually Latin-1, with additional characters to cover more European languages in a single codepage. This is (in theory) quite good for major European languages, less so for less-prominent ones. For each font, bitmaps are distributed in 3 sizes (12, 24, 48 pt), with the engine doing interpolation scaling.

Current Font Findings

12-pt Size for All Fonts

Only ASCII (i.e., lower range 0-127) characters are provided, no European. For some fonts (stone, mason, mason_glow), the 12-pt DAT is not distributed, so the engine will substitute a larger size, which typically has better Latin-1 coverage.

For Fonts Used in Main Menus, HUD, or Subtitles

The numbers shown approximate the number of "characters needing work"

Fontname             Size-24    Size-48

 carleton               20       16
 carleton_condensed     20       35
 mason                  --       33         Since 24 pt is not distributed, engine uses 48 pt.
 stone                  30       83

For Additional Fonts Used in FM Readables, Etc.

Except for one font (treasure_map), all the remaining fonts are ASCII-only, i.e., no characters in the upper range. In the lower range, routinely the 24 and 48 pt sizes have equivalent coverage. Most of these fonts are fully or nearly complete, while some neglect certain punctuation symbols. The worst is "everette", with 24 "needing work" characters.

Further details are here:
 

Spoiler

Details about the 24 and 48 pt Sizes

The numbers shown approximate the number of "characters needing work". Note that in some cases, only a correction to font metrics in the DAT file would be needed. Other cases would need bitmap surgery.

For Fonts Used in Main Menus, HUD, or Subtitles

 carleton Size-24 (20 need work) [Note 1]
    Ń (0x8c), Ș (0x8d), Ț (0x8e), đ (0x90), ś (0x91), ŝ (0x95), ô (0x98), ŕ (0x99), ǔ (0x9a), ń (0x9c), ș (0x9d), ț (0x9e) + 8 more [Note 4] in upper range

 carleton Size-48 (16 need work) [Note 1]
    Ô (0x88), Ń (0x8c),Ș (0x8d), Ț (0x8e), đ (0x90), ĉ (0x96), ă (0x9b), ń (0x9c),  ș (0x9d), ț (0x9e), § (0xa7), Ñ (0xd1), á (x0e1), ñ (0xf1), ù (0xf9) + 1 more [Note 4] in upper range.

 carleton_condensed Size-24 (20 need work) [Note 1]
    Same set as carleton size-24

 carleton_condensed Size-48 (35 need work)
    All in ranges 0x80-0x8e, 0x90-0x9e, plus § (0xa7), Ñ (0xd1), á (0xe1), ñ (0xf1), ù (0xf9)

 mason Size-48 (33 need works) [Note 2]
    All in ranges 0x80-0x8e, 0x90-0x9e, plus Ą (0xaa), ą (0xba), ř (0xf7)
    Also, treatment of Non-breaking Space (NBSP)(0xa0) and Soft hyphen (SHY) here is inconsistent with other fonts, but unclear what correct treatment should be.

 Stone Size-24 (30 need work) [Note 3]
    Ń (0x8c), Ș (0x8d), Ț (0x8e), đ (0x90), ô (0x98), ŕ (0x99), ș (0x9d),  ț (0x9e), ð (0xf0) + 21 more [Note 4] in upper range.

 Stone Size-48 (83 need work)
   2 in lower range with bad boxes (p & y) + 20 in upper range with bad boxes (could enumerate these here, but too tedious) + 61 more [Note 4].

[Note 1] Same stats for carleton_bold, carleton_glow
[Note 2] Same stats for mason_glow. For mason and mason_glow, 24 pt is not distributed, so engine uses 48 pt.
[Note 3] A more precise and in-depth analysis of 24-pt stone, including bitmap inspection, was presented earlier.
[Note 4] Bounding boxes are shared among code points. Bitmap inspection would be needed to id which glyph "owns" bounding box, and which codepoints are interlopers. In many cases, this represents a deliberate substitution strategy, e.g., to redirect a missing accented character to another similarly-accented or unaccented character. Better than nothing.

For Additional Fonts Used in FM Readables, Etc.

Only the author of one font started to add upper-range (128-256) characters, but didn't get far along:

 Fontname     Size-24  Size-48

 treasure_map    107    96

Remaining fonts are ASCII-only! Characters in the upper range (of which TDM defines 124) were not done. These are indicated as "124+" below, plus any additional problematic characters in the lower range. These latter are generally punctuation symbols.

 Fontname       Size-24 and Size-48

 andrew_script        124+0
 bamberg              124+0
 camberic             124+3
 carolingia           124+3
 chrishand            124+6
 ellianerelle         124+0
 everett              124+24
 gothica2             124+20
 jd_hand              124+0
 lotharus             124+11
 mac_humaine          124+8
 medusa               124+1
 nancy                124+4
 popsies              124+1
 rapscallionpirate    124+10 (for 48 pt, +9 for 24 pt)
 shoppinglist         124+1
 summertime           124+0

 

 

Edited April 7, 2024 by Geep
fix link

[wesp5]

I don't really know what this is about, but I will use the occasion to remind everybody that there is a surplus pixel at the lower left side of the capital W character. People speculated this might come from another character nearby in a font table but I don't see how V could have a pixel at that position....

Edited April 7, 2024 by wesp5

[Geep]

On 4/7/2024 at 5:40 PM, wesp5 said:

I don't really know what this is about, but I will use the occasion to remind everybody that there is a surplus pixel at the lower left side of the capital W character. People speculated this might come from another character nearby in a font table but I don't see how V could have a pixel at that position....

I'm assuming we're talking about Stone 24pt font here, used in subscripts & elsewhere.

Images of letters are arranged into a bitmap by a process that doesn't have to conform to a tidy alphabetic grid. See Figure 2 of my new article https://wiki.thedarkmod.com/index.php?title=Font_Bitmaps_in_DDS_Files

In the figure, you can see there's an accented "a" to the left of the W, whose foot is presumably within W's bounding box. I think I looked at this earlier & concluded that I couldn't shrink W's bounding box enough in the DAT file. So probably the pixels of the W need to be moved to the right... I haven't tried such surgery yet.

[wesp5]

It's good to know that somebody is taking care of this in the long run, thanks :)!

[Geep]

Because DDS is a lossy format, every direct edit of a DDS file may reduce its quality slightly. So that means either:

• hunt down the corresponding TGA file with the same timestamp
• otherwise, edit the DDS as few times as possible.

In the case of Stone, if the TGA can't be found and editing DDS is needed, I'd want to think about batching up all the bitmap edits needed for i18n... but that's a lot of work.

I might add, there a third possibility in the case of the readable fonts that are currently ASCII only. If they were sourced from TTF, and you find the TTF (easy in a few cases), then see if it covers Latin-1 or more. You can (with some complications RE the conversion software) regenerate an improved TDM font from that.

[datiswous]

15 hours ago, Geep said:

otherwise, edit the DDS as few times as possible.

You can confert it to TGA first before editting?

[Geep]

I'm guessing there's a way to do that. Then back to DDS to distribute. That process would still cause an extra generation of quality loss, but at least you'd have a TGA in hand, for iterative editing without cumulative losses. But maybe first ask long-time TDMers to hunt for archived TGAs.

[Amadeus]

9 minutes ago, Geep said:

I'm guessing there's a way to do that. Then back to DDS to distribute. That process would still cause an extra generation of quality loss, but at least you'd have a TGA in hand, for iterative editing without cumulative losses. But maybe first ask long-time TDMers to hunt for archived TGAs.

So, there is actually an SVN repo with these tga font images, they are 256x256. Is something like this what you are looking for:

 

[Geep]

I thought that might be the case. I don't believe I can access the SVN. Could you zip up all the font stuff you can easily find in that archive and send a link to me? TGA/DDS, even TTF for Stone, Mason, Carleton, etc., etc. No hurry. Thanks.

P.S. Try to preserve file timedates

Edited April 10, 2024 by Geep

[Amadeus]

Yep, I'll PM you soon

[Geep]

Thanks, looks useful. Except datetimes look like all Jan 24, 2024, presumably when the archive was formed, not when file edit was done. So it goes.

[Amadeus]

Yeah, sorry about that, that is when the archive was formed, I'm not really sure what I can do about that

[datiswous]

16 hours ago, Geep said:

I don't believe I can access the SVN.

SVN is here? https://svn.thedarkmod.com/publicsvn/darkmod_src/ no idea where these files are

Edited April 11, 2024 by datiswous

[Geep]

On 4/10/2024 at 2:25 PM, Amadeus said:

Yeah, sorry about that, that is when the archive was formed, I'm not really sure what I can do about that

That was just a "nice to have", not "must have". No worries.

On 4/11/2024 at 5:10 AM, datiswous said:

SVN is here? https://svn.thedarkmod.com/publicsvn/darkmod_src/ no idea where these files are

Nor I.

[Geep]

Just for the record to benefit readers here, the archived Stone .tga's turned out to be the original ASCII-only versions, not the versions that got extensive bitmap editing. Too bad.

[Geep]

Also, related to font improvement, I've just released "ExportFontToDoom3All256", a reconstruction of an earlier but now lost tool variant. This is described and available in the wiki article ExportFontToDoom3

I tested that tool using one of the TDM FM fonts, Andrew Script, for which a TTF file is available. I generated a fresh set of bitmaps (newly including any available Latin-1 characters). I also mucked about with FontForge, to reconfigure that TTF to be ordered like the TDM custom codepage.

However, Andrew Script is missing a fair number of Latin-1 glyphs, so it would take some work to make it good (whether by editing in FontForge or post-export as bitmaps). I'm putting that aside for now, since the jury is out on whether Western language support in FMs and their fonts will become viable (see Western language support in 2024?).

Instead, I plan to turn my font-improvement-for-2.13 attention to Stone 24pt, which (because its used in HUD captions) is more clearly worthwhile to work on. Looks like I'll have to convert the Stone DDS to TGA as a prerequisite to bitmap editing.

[Geep]

Some good news! Tels dug around further in his old computers and found some gold. Namely, a whole fonts directory that includes GIMP .xcf files. Tels says:

"The XCF files esp. are what I used to manually draw the new characters (like adding dots to an u to convert it to ü etc.) They contain many layers with different characters, that are layout exactly in the place where they need to be for the patcher script."

That is, for english stone 24 pt, one .xcf file contains two independent RGBA bitmap layers, that can be saved separately as two .dds files. A quick glance of that content appears to match the current distributed stone_0_24.dds and stone_1_24.dds. So I won't need to back-convert from DDS to TGA after all.

@Amadeus, I think a copy of this should be added to the TDM assets repository. Could you do that?
* http://bloodgate.com/mirrors/tdm/pub/scripts/tdm_font_source.7z

[Amadeus]

5 minutes ago, Geep said:

@Amadeus, I think a copy of this should be added to the TDM assets repository. Could you do that?

Yeah, I can do that. Shouldn't be a problem, unless anyone has a particular objection to it being uploaded to the repo

[Geep]

Also, I've built another console app, "datBounds", to visualize the bounding box around each bitmap character. Starting from your choice of .dat, .fnt, or .ref file, it generates a set of .tga files, e.g., stone_0_24_bounds.tga & stone_1_24_bounds.tga. You can optionally include a 2-hex-digit label on each bounding box. More about that in May.

I'm using the results of this to understand the current complex codepoint mapping of the Stone 24pt glyphs, and develop the workplan and grid layouts for upcoming work.

[datiswous]

3 hours ago, Geep said:

Tels

He's alive!!!

[Geep]

Yep, moved on to greener pastures.

BTW, what I quoted above from him

Quote

[The .xcf bitmaps have layouts] exactly in the place where they need to be for the [font] patcher script.

will soon be untrue for Stone 24pt due to my upcoming bitmap surgery. Actually, the values in the current stone .dat file are already misaligned with what any archival font patcher script would produce, due to 2.12 tweaks with q3font & refont.

I'll add a note to the wiki about this (explaining it better than here).

[nbohr1more]

In our SVN under the devel folder we have a font source sub-folder. Do we know for sure that these are all newer?

[Geep]

That may be where @Amadeus got the folder provided to me earlier.

Tels folder has -

- .xcf files, not found in earlier work

- for any given system font (stone, carleton, manon) and size(particularly 24 or 48), often more .tga/.dds files than earlier work (which originally had just ASCII characters). (You can open the files in GIMP to be even more definitive).

[Amadeus]

 

2 minutes ago, Geep said:

That may be where @Amadeus got the folder provided to me earlier.

 

Well, I actually found that initial folder in the HiRes SVN repo. This is different from the TDM assets repo as it has a bunch of source files for textures, fonts, etc, but they aren't really game-ready assets. This is where I found the tga image files for the fonts I thought you were looking for.

I haven't uploaded Tels folder yet, but I was thinking of uploading them in the HiRes SVN repo, just for the sake of backing it up

[Geep]

I'm now considering the .xcf files as being in effect the source texture for the game fonts, at least those that have been bitmap edited in GIMP. So that's an argument for putting them in the TDM assets repo.

But, in any event, they do need to be backed up somewhere official.