Ported font code from Doom3 BFG Edition

[Zbyl]

Hi guys,

 

I haven't seen anyone working on this, so I've ported font code from Doom 3 BFG edition to The Dark Mod. This allows to create fonts that support full unicode range.

I'm porting LangDict now, so all texts will be in unicode and no hacks will be necessary - no code pages, conversions, replacements, separate font files, problems after language change etc.

 

There's one issue: the BFG fonts don't have three sizes (small, medium, large). There is only one font size, and this font is scaled if necessary.

This creates an unfortunate side effect: font scales in GUI that were tuned for old fonts will be wrong now.

In the Doom3 BFG they do a little hack: they load old fonts too, but use them only to emulate old font measuring code. This is not only ugly, but also a maintenance nightmare. For TDM I would propose to drop it and just fix the GUI to use proper font scales.

 

I'd like to know what you guys think about this. (To make it clear: using old and new fonts together is not possible.)

 

Some technical stuff: I've used some new helper files from BFG. They are not strictly necessary, but I didn't see any reason not to use them if it made my work easier. Those are for example: sys_assert.h, sys_types.h, new collections/List.h any maybe something else.

The porting itself was really straightforward - copy/paste mostly.

I've also made a little utility program that converts fonts created by a freeware BMFont (http://www.angelcode.com/products/bmfont/‎) into BFG fonts.

 

Another thing connected to fonts and localization that I'd like to discuss are the toUpper, toLower etc. functions. Those cannot be reliably used for GUI. Think about toLower() for german language: sometimes "SS" have to become "ss", and sometimes "scharf ss". It should be translators job to prepare text in proper case and form. I strongly advise against any conversions like that in GUI output.

The toUpper() and toLower() functions in TDM right now don't work properly for diacritics (like ąęćżźńł, scharf ss) - and it makes GUI look super ugly.

However: providing implementation that will work decently is quite simple and I've written most of the code for it.

The question is: should I finish it, or should TDM just drop toUpper from GUI code?

 

(BTW.: I know that you won't even consider integrating any of this stuff before 2.0 is out.)

Edited September 8, 2013 by Zbyl

[grayman]

What's the problem this port solves? I.e. why do we want it?

 

None of this would be integrated into 2.00. We're waiting on one problem getting fixed, then we're done.

[rich_is_bored]

With all the manual painting Tels put in editing the existing raster fonts to add previously unsupported characters, I imagine it would require a lot of effort to use custom fonts in an FM. If the process can be made easier, code freeze aside, I don't see why anyone would be opposed to it.

[stumpy]

i use custom fonts in some of my maps it sounds like this change would break the fonts i use.

been using ExportFontToDoom3 to convert the true type fonts to doom 3 font dat files.

[motorsep]

@Zbyl: What tool do you use to generate BFG fonts?

 

BFG's fonts improvements were meant to improve Doom 3 fonts - make them of higher resolution and allow for extra characters, that's all. They were not a replacement for original font system. That's why BFG's fonts need old font .dat files to map new characters over them.

[Zbyl]

@Zbyl: What tool do you use to generate BFG fonts?

Like I said: I use BMFont and my custom converter from BMFont to BFG fonts.

BFG's fonts improvements were meant to improve Doom 3 fonts - make them of higher resolution and allow for extra characters, that's all. They were not a replacement for original font system. That's why BFG's fonts need old font .dat files to map new characters over them.

They do replace the original font system (they need original fonts only to know what their maxCharWidth/maxCharHeight is to know how to scale new fonts into old GUI). This is in the code - no guesses here.

 

What's the problem this port solves? I.e. why do we want it?

There are many code pages to support now - language files need to be converted into custom encodings.

Russian language is hacked in quite ugly.

There is no way to display i.e. language list in the GUI properly: i.e. russian in russian, polish in polish, spanish in spanish - no one font supports all the necessary characters.

Separate font files - inconvenient.

Need to reload fonts after language change - inconvenient.

Adding new languages is tricky.

There's no way TDM can support Japaneese, Chineese, Korean etc. right now. It would be a breeze with BFG's fonts.

Basically the I18N support is a big hack right now.

 

With all the manual painting Tels put in editing the existing raster fonts to add previously unsupported characters...

 

I'd like to get a look at this. Is this code somewhere in the public SVN?

The BMFont I used doesn't support this - but I used BMFont just because it was the easiest way I could generate a font for my testing. It would probably take little effort to generate BFG fonts using the content pipeline you already use.

Even better: if breaking backwards compatibility in any way is not an option then BFG code can be modified to work just like the old system, but without the old D3 problems and with unicode support.

But breaking backwards compatibility is ok sometimes. I just wonder what course of action would be best.

 

i use custom fonts in some of my maps it sounds like this change would break the fonts i use.

It would, but there are ways to solve it. It could be done the hacky way (just porting current behaviour).

But since the full D3 code is open now - it could be done the proper way: i.e. support changing font to a custom one mid-text, like the support for changing colors is done.

 

About the broken toUpper()/toLower(): any thoughts on that?

(I've described some of the problems here in http://bugs.thedarkmod.com/view.php?id=3321)

Edited September 8, 2013 by Zbyl

[Springheel]

But breaking backwards compatibility is ok sometimes.

 

Seems like this would break a lot of maps and require a lot of adjustments to the guis and translation system. Having spent the better part of a year fixing compatibility issues caused by going standalone, the benefits would have to be pretty impressive to make it worth the hassle.

[Zbyl]

That sounds like a lot of hassle just to avoid some "inconvenient" loading.

But you ignore all the other issues I mentioned.

[SeriousToni]

I guess the team members are a bit stressed atm because of developing 2.0 as standalone. Maybe it would be better to discuss this some time later for 2.1?

[Zbyl]

I think the benefits of this code are impressive, even if not evident at first, so I'll keep working on this.

It would definitely help me with compatibility to know what tools you use for generating fonts. Is it ExportFontToDoom3 exclusively?

 

I also think I misunderstood the "manual painting Tels put in editing the existing raster fonts". I thought he wrote some tool to support adding new glyphs to existing fonts, but now I re-read it it sounds like you were talking about actual editing of font textures. It definitely would not be a wasted work. The only thing that is needed is: some tool to support adding new glyphs to existing fonts... I'll look into this too.

[New Horizon]

I think you have a far better understanding of the system than most of us, so it might be hard for us to understand the benefits. So this will also improve the look of the fonts as well as under the hood functionality? If you think it's worth it, i say go for it and we can look at the pros / cons after 2.0 has been released.

 

Have you looked at any of the performance side changes in Doom 3 BFG? Things like dual core enhancements and threading? Those things would probably be a nice performance boost for us later on down the road.

[motorsep]

Like I said: I use BMFont and my custom converter from BMFont to BFG fonts.

 

BMFont's page shows 404. As far as I remember, BMFont generated glyph atlas from TTF fonts. How do you generate .dat file for BFG? Can you upload the final BFG font you generated, using BMFont and your secret tool, so I could look at TGA atlas and .dat file, please?

 

They do replace the original font system (they need original fonts only to know what their maxCharWidth/maxCharHeight is to know how to scale new fonts into old GUI). This is in the code - no guesses here.

 

Right, they didn't replace it. You can still use old fonts with BFG, or just new font. However, it's be off. Replacing means you can use new font without screwing GUIs. They added new font system without removing old one.

 

EDIT: http://www.angelcode.com/products/bmfont/ works, which doesn't solve the issue of generating proper .dat files

Edited September 8, 2013 by motorsep

[zergrush]

Have you looked at any of the performance side changes in Doom 3 BFG? Things like dual core enhancements and threading? Those things would probably be a nice performance boost for us later on down the road.

 

This and all of my this. This contribute of yours might seems like a nice addition, but what we really need atm is a bad dude enough to tackle this one with wrestler's arms.

[Zbyl]

Can you upload the final BFG font you generated, using BMFont and your secret tool, so I could look at TGA atlas and .dat file, please?

My test Gothic font is here: https://drive.google...N1U&usp=sharing

BFG .dat is of course in a different format than D3 .dat. The TGA is generated by BMFont - I didn't make any chages to it.

BFG fonts are different for example in that they use only one big texture per font, not many like in D3. This is not a problem these days when big textures are universaly supported.

 

As for the performace improvements: I don't know a lot about this. Only stuff I've read in BFG code review and some articles by Jean Paul van Waveren and Carmack. But I'll take a look if I'll find the time.

[motorsep]

My test Gothic font is here: https://drive.google...N1U&usp=sharing

BFG .dat is of course in a different format than D3 .dat. The TGA is generated by BMFont - I didn't make any chages to it.

BFG fonts are different for example in that they use only one big texture per font, not many like in D3. This is not a problem these days when big textures are universaly supported.

 

Ahh, I see. So font atlas is of the power of two, while original BFG atlases are non-power of two. BFG's glyphs are packed very tightly, rotated/mirrored if needed to fit more characters in one atlas. I suppose it doesn't really matter for regular fonts, but for Japanese/Korean/Chinese fonts just square atlas would be huge :/

 

Have you tested if your font works with BFG (converting ArialNarrow into BFG's font and replacing the original ArialNarrow that comes with BFG) ? If not, and you don't really know how, I can test it if you can convert ArialNarrow.

 

Do you convert glyphs descriptor generated by BMFont into .dat file? If so, can you please share the code of your converter?

[motorsep]

Have you looked at any of the performance side changes in Doom 3 BFG? Things like dual core enhancements and threading? Those things would probably be a nice performance boost for us later on down the road.

 

I have. Here is my performance testing results:

[Zbyl]

Do you convert glyphs descriptor generated by BMFont into .dat file? If so, can you please share the code of your converter?

Yep. I convert the text version of the descriptor into a BFG .dat. Probably converting XML descriptor would be easier, but I just didn't thought of it.

Sure I'll share the code - but at the moment it's rather messy. And in C#. I'll clean it up a tiny bit first.

 

As for testing my font in BFG - sadly I can't do that, since I don't own BFG... I only have the code to play with.

[motorsep]

Yep. I convert the text version of the descriptor into a BFG .dat. Probably converting XML descriptor would be easier, but I just didn't thought of it.

Sure I'll share the code - but at the moment it's rather messy. And in C#. I'll clean it up a tiny bit first.

 

As for testing my font in BFG - sadly I can't do that, since I don't own BFG... I only have the code to play with.

 

Thanks! Ahh, C#.. I hope we can convert it into C++

 

If you can send me ArialNarrow converted with your method, I can test it here (.tga and .dat is what I need).

[Zbyl]

I've created an ArialNarrow font that should work. I'm not sure if the scale will be ok though. https://drive.google.com/folderview?id=0B0pkDNTfAmoeQm9zZG5TeVFqejg&usp=sharing

I've attached BMFont's config and .fnt that I used to create this font too.

[zergrush]

As for the performace improvements: I don't know a lot about this. Only stuff I've read in BFG code review and some articles by Jean Paul van Waveren and Carmack. But I'll take a look if I'll find the time.

 

That would be greatly appreciated, as TDM currently suffers from some serious performance issues on older pcs, and even some choppy frames on modern pcs when playing larger maps.

[motorsep]

I've created an ArialNarrow font that should work. I'm not sure if the scale will be ok though. https://drive.google...ejg&usp=sharing

I've attached BMFont's config and .fnt that I used to create this font too.

 

Ok, tested. Here are the comparison screenshots: http://www.steel-storm.com/files/d3bfg_ArialNarrow_font_comparison.zip

 

Generally speaking, it's a very good start! However, as you can see, there are issues. Font size doesn't match up, somehow there are white boxes around glyphs (_09.tga; terminal GUI on Mars City 1 at the very beginning right off the drop ship) and it's totally screwed up in PDA.

[Zbyl]

Thanks for this test motorsep.

The white boxes appear because I apparently gave too small spacing in BMFont's generation options. I should have bumped the output resolution too.

I need to figure of what settings to use for scaling fonts properly. In D3 standard fonts are of size 48pt, but with what DPI? Maybe 120?

And why the PDA is messed up I have no idea... yet.

[motorsep]

I can send you original TGA atlas and .dat file for ArialNarrow if that will help you to figure out whatever is needed to make identical .dat file (I don't think dpi has anything to do with how font appears in-game)

 

EDIT: Sent you pm

Edited September 8, 2013 by motorsep

[Zbyl]

Thanks motorsep.

I've uploaded source code of my tool here https://github.com/Zbyl/BFGFontTool

[Zbyl]

The reason why my font display incorrectly in PDA is this code in BFG's SWF text renderer:

float linespacing = fontInfo->GetAscender( 1.15f * glyphScale );

My ascenders are 0 now, since they are not used anywhere in text renderer.

(This code is used only in the SWF renderer of the new Doom GUI).