Jump to content


Photo

TDM Benchmark Demo


  • Please log in to reply
90 replies to this topic

#1 greebo

greebo

    Heroic Coder

  • Root
  • 15906 posts

Posted 15 January 2010 - 04:18 AM

For people who want to test their rig using a standardised demo, here's a PK4 for you to download and check out your performance using the TDM training map.

Thanks a lot to rebb for creating and uploading that file. :)

Instructions:
- Download this PK4: http://www.dramtheth...aining_demo.pk4 (~18 MB)
- Save the file into your darkmod/ folder, into where all the other tdm_*.pk4 files are residing.
- Launch TDM
- Install the Training Mission (this will restart TDM)
- Open the console (by typing Ctrl-Alt-~)
- Type timeDemo training.demo and hit enter, the demo will start

-timedemo

(analysis, recording, warping)
doom -timedemo <lump>

The -timedemo parameter plays a demo without limiting graphics to 35 frames per second and upon exiting to the command prompt displays the number of screen frames drawn (gametics) and the time taken to play them (realtics), instead of the ENDOOM screen. The average screen frame rate on a level can be determined with this information (realtics/gametics*35=fps). If a demo recording spans more than one level, the resulting values will be incorrect, as the realtics will correspond only to the last level played.

Like during the built-in demo sequence normally played at the start of a game, this parameter makes most device input bring up the menu while the demo is playing, so pressing the Tab key to select automap mode or pressing Enter to display the last status message can't be used. Some non-playing functions do still work, such as increasing or decreasing the screen size.

Post the results (average FPS, etc.)

Edited by greebo, 02 March 2011 - 04:11 AM.


#2 Ishtvan

Ishtvan

    Programmer

  • Development Role
  • PipPipPipPipPip
  • 14860 posts

Posted 15 January 2010 - 04:36 AM

I tried playdemo and played the demo, but didn't see any results in the console, just "stopped playing.". Are you sure it's not timeDemo that gives results? I'll try that now.

#3 Ishtvan

Ishtvan

    Programmer

  • Development Role
  • PipPipPipPipPip
  • 14860 posts

Posted 15 January 2010 - 04:42 AM

timeDemo ran through the demo at a fast-forwarded speed (15 sec total), not sure if that's what it was supposed to do. But it did report results:
Avg FPS: 197.8

Hardware:
CPU: Intel i7 920
GFX: 2x Radeon HD4870 512 MB, Crossfire
RAM: 6 GB

OS: Win7 64 bit

[EDIT: Graphical settings for this run were:]

Resolution: 1920x1200
Vsync: off
AA: off
Anisotropy: 8x
Ambient: Standard
Interaction: High Quality

(Don't ask me how the settings got this way, they're not what I thought I had set :) )

I had Catalyst AI on, and the skybox in the demo was even crazier than what it usually does (display sky with spinning). It just displayed a bizzare reflection of whatever was on the screen in the sky, no sky to speak of. Might get better results with Catalyst AI off, but I have to go to sleep for now.

#4 greebo

greebo

    Heroic Coder

  • Root
  • 15906 posts

Posted 15 January 2010 - 04:45 AM

Ah, updated my post. Never used this feature myself thus far.

#5 Ishtvan

Ishtvan

    Programmer

  • Development Role
  • PipPipPipPipPip
  • 14860 posts

Posted 15 January 2010 - 04:58 AM

Does it matter what resolution/graphical options you have selected, or does the demo set its own when it runs?

#6 greebo

greebo

    Heroic Coder

  • Root
  • 15906 posts

Posted 15 January 2010 - 05:00 AM

I think that's exactly the purpose of such a timedemo, to test various resolutions and setups without having to actually "play" the game yourself each time, going through the same reproducible scenes.

#7 Sonosuke

Sonosuke

    Member

  • Member
  • PipPip
  • 300 posts

Posted 15 January 2010 - 05:11 AM

Download link does not work for me, only get a site with strange signs :huh:

EDIT: :P

Edited by Sonosuke, 15 January 2010 - 05:15 AM.

Ich konnte mich nicht erinnern Teleportation gezaubert zu haben und doch stand ich da... alleine und nackt.

#8 greebo

greebo

    Heroic Coder

  • Root
  • 15906 posts

Posted 15 January 2010 - 05:14 AM

Download link does not work for me, only get a site with strange signs :huh:

That's Dram's FTP for you, you need to right-click the link and save the target to your hard drive.

#9 7upMan

7upMan

    Advanced Member

  • Member
  • PipPipPip
  • 1081 posts

Posted 15 January 2010 - 05:31 AM

Okay, here is what I can tell: on my Radeon, the sky is displaying in a strange way, much like Ishtvan reported. I have Catalyst AI disabled, so the error must be in the map itself. In all other FMs, the sky is shown properly once Catalyst AI is disabled.

My system:

CPU: AMD Phenom II X3 720 BE @ 3.4 GHz

GPU: ATI Radeon HD4870/1G

RAM: Corsair 4x1GB DDR2-800 CL4

OS: Windows XP 32Bit SP3

Settings: 1920x1200, 2xSS-AA, 16xAF, Bloom off, Vsync off, Shader: High quality, Ambient: Standard



My results: 84,7 fps after the third run (being 69,2 after the first run, so I would recommend at least two runs to have the FM completely loaded into the RAM).



I have a request: is it possible to modify the Training Mission in such a way that two (or more) AIs are spawned that start attacking each other? I don't know if the result of such a fight would be the same every time, but this would certainly stress the CPU, wouldn't it?

Edited by 7upMan, 16 January 2010 - 03:29 AM.

My Eigenvalue is bigger than your Eigenvalue.

#10 _Atti_

_Atti_

    Advanced Member

  • Member
  • PipPipPip
  • 1044 posts

Posted 15 January 2010 - 07:06 AM

84,8 FPs

But the sky was buggy, and had some strange stops compared to the very fast run of the demo.
NOte that the sky isnt buggy when normally playing missions( Bloom is off, catalyst ai is off)

#11 greebo

greebo

    Heroic Coder

  • Root
  • 15906 posts

Posted 15 January 2010 - 07:07 AM

Maybe the portal sky is not working in demos, who knows?

#12 hykao

hykao

    Member

  • Member
  • PipPip
  • 21 posts

Posted 15 January 2010 - 08:30 AM

Posted Image

(a bug: have to set to 1280x720 to get real 1366x768...)

#13 SneaksieDave

SneaksieDave

    QA Lead

  • Development Role
  • PipPipPipPipPip
  • 10124 posts

Posted 15 January 2010 - 10:28 AM

Can numbers like 197 FPS be trusted? Doom3 is capped to 60 FPS for most, so I wonder if that's showing the actual results (wow?) or if the number is just not usable.

Also, maybe the PK4 should contain the map and its parts, as it's possible the training map may change in the future?

#14 New Horizon

New Horizon

    Mod hero

  • Active Developer
  • PipPipPipPipPip
  • 13660 posts

Posted 15 January 2010 - 10:53 AM

On my wife's laptop

Windows Server 2003
Intel Celeron M - 1.6ghz
896mb ram
ATI Radeon xpress 200m -driver 8.573

My settings were fast ambient and standard interaction.

18.6 fps

Seems the portal sky doesn't like the time demo. I had a weird hall of mirrors effect during this where the level ran off into infinity in the sky. Very strange.

#15 Tels

Tels

    Mod hero

  • Member
  • PipPipPipPipPip
  • 15024 posts

Posted 15 January 2010 - 11:02 AM

Seems the portal sky doesn't like the time demo. I had a weird hall of mirrors effect during this where the level ran off into infinity in the sky. Very strange.


There are a lot of things that are simply skipped during the timedemo (for instance sound, I forgot what the other things were, in-game GUI rendering? scripts?) so I am not surprised portalskies don't work. I never understood why they did choose such a bad way of making timings (or rather, why they didn't at least add a "full time demo" version, too). With D3, the demo would reflect the renderspeed, but it would not accurately reflect the in_game FPS. It is even worse with TDM, where lots more CPU cycles are spent elsewhere.
"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

"Remember: If the game lets you do it, it's not cheating." -- Xarax

#16 SneaksieDave

SneaksieDave

    QA Lead

  • Development Role
  • PipPipPipPipPip
  • 10124 posts

Posted 15 January 2010 - 11:15 AM

Well I suppose it's not so much a "how many FPS will I get in TDM!!" measure as it is simply a benchmark for comparison among systems. Guy1 gets 88, Guy2 gets 121, Guy3 gets 197. Guy3 wins. But he shouldn't expect his framerate will be 197, because it won't. Just like that famous benchmark test, whatever it's called -- it's not 12,000 FPS; the score is 12,000.

Anyway it won't mean anything if people attempt to run different versions of the map.

#17 7upMan

7upMan

    Advanced Member

  • Member
  • PipPipPip
  • 1081 posts

Posted 15 January 2010 - 12:33 PM

Well I suppose it's not so much a "how many FPS will I get in TDM!!" measure as it is simply a benchmark for comparison among systems.  Guy1 gets 88, Guy2 gets 121, Guy3 gets 197.  Guy3 wins.  But he shouldn't expect his framerate will be 197, because it won't.  Just like that famous benchmark test, whatever it's called -- it's not 12,000 FPS;  the score is 12,000.


...hence my question about if it's possible to have AI in the timedemo, to make the test as close to the actual performance as possible. But I know zero about the inner workings of the Doom 3 engine (or any other graphics engine, for that matter), so you'd be the ones who could give an informed answer.


My Eigenvalue is bigger than your Eigenvalue.

#18 Tels

Tels

    Mod hero

  • Member
  • PipPipPipPipPip
  • 15024 posts

Posted 15 January 2010 - 12:38 PM

System:

CPU : AMD Athlon(tm) 64 X2 Dual Core Processor 5000+
GPU : Nvidia Geforce 9800GT (512 Mbyte)
Ram : 4 Gbyte
OS  : Kubuntu 9.10 64bit
Swap: 4Gbyte configured, but none enabled.
Driver: 185.18.36

Basic settings:

Resolution:  1920x1200
Anisotropic: 1x
Antialising: Off
VSync:       Off
Blur:        Off
Shader:      High quality
Ambient:     Standard

Three runs:

First:  77.4 seconds, 38.5 FPS (very first run, nothing in diskcache yet)
Second: 61.6 seconds, 48.3 FPS
Third:  64.0 seconds, 46.5 FPS

Observations:

* Opening the door to the outside area (on the first after starting D3 only) causes a full 2..3 seconds pause, even when everything is already in the diskcache

* The sky doesn't work her, either (hall of mirrors). However, a curious effect is that whenever it is in view, things slow down to a crawl. Whenever there is no sky (indoors), everything whizzed by superfast. So we might make a second demo without portalsky, as well as try to figure out if this is a demo-artefact or if portal skys really pose some serious CPU overhead (I know it is not the GPU, see below).

* My GPU is a beast. Nothing slows it down. Modifying the settings and doing two runs for each of them (after D3 restarts) shows that regardless of setting, the timing various more across runs than between different settings. For instances setting anisotropic to 4x and antialiasing to 4x results in 65.6 and 59.1 seconds, which is faster than the baseline settings above (the first run can always be discarded due to the caching issue). Turning bloom on made no difference (but I couldn't see a visual difference, either). 8x anisotropic and 8x antialiasing made no difference either.

* The timedemo seems to be CPU bound, lowering the resolution to 1024x600 did result in 62.8 and 60.1 seconds, which is not really faster than the baseline above.
"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

"Remember: If the game lets you do it, it's not cheating." -- Xarax

#19 Tels

Tels

    Mod hero

  • Member
  • PipPipPipPipPip
  • 15024 posts

Posted 15 January 2010 - 12:42 PM

Well I suppose it's not so much a "how many FPS will I get in TDM!!" measure as it is simply a benchmark for comparison among systems. Guy1 gets 88, Guy2 gets 121, Guy3 gets 197. Guy3 wins. But he shouldn't expect his framerate will be 197, because it won't. Just like that famous benchmark test, whatever it's called -- it's not 12,000 FPS; the score is 12,000.


That is true, however, if the benchmark measures only a subset, the score of "12000" vs. "3000" doesn't mean much,because actual framerates could be "20" vs. "19" on the two systems (or even "19" and "40", depending on what we measure and what is in the real map).

Anyway it won't mean anything if people attempt to run different versions of the map.


That is true, but something completely different :)

As for the question about AI, the original D3 had AI (imp) in them, so we might try this, too.

The scripts seem to run (the trainer messages appear), and HUG and GUI work, too, so that is good news. I think we can ignore the "not playing audio" as that would not slow things down on modern systems by more than a few % points.
"The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore, all progress depends on the unreasonable man." -- George Bernard Shaw (1856 - 1950)

"Remember: If the game lets you do it, it's not cheating." -- Xarax

#20 Melcar

Melcar

    Newbie

  • Member
  • Pip
  • 4 posts

Posted 15 January 2010 - 04:19 PM

Benchmark doesn't seem to want to work for me. Running the provided command simply launches the training mission again. If I try to launch it from the main menu the console complains that it can't open "demo/training.demo". Running Kubuntu Karmic.

Edited by Melcar, 15 January 2010 - 04:28 PM.


#21 Melcar

Melcar

    Newbie

  • Member
  • Pip
  • 4 posts

Posted 15 January 2010 - 05:06 PM

Managed to get it to run. Had to place the .pk4 in the /.doom3/training_mission folder instead of the instructed /.doom3/darkmod/fms folder. Also experienced issues with rotating sky/ceiling, even when the actual game runs fine. System specs. and results are below:

CPU: Phenom X4 9650 @ 3.0GHZ
RAM: 2x2GB DDR2
GPU: HD4850 512MB @ 750/1100
OS: Kubuntu Karmic 64bit
GPU Drivers: 9.12

Game Preferences:
1680 x 1050
4xAA
16xAF
Bloom Off
Vsync. Off
Ambient Rendering Standard
Interactive Shader High Quality

Demo Run 1: 69.8fps
Demo Run 2: 88.1fps
Demo Run 3: 87.6fps

Edit:
Launched the game a second time and started getting much higher results.
Run 1: 73.5fps
Run 2: 101.2
Run 3: 101.1fps

:huh:

Edited by Melcar, 15 January 2010 - 05:20 PM.


#22 Melan

Melan

    Contest Winner, Wordsmith

  • Campaign Dev
  • PipPipPipPip
  • 4280 posts

Posted 15 January 2010 - 05:20 PM

My PC:

CPU: AMD Athlon 3200+ 2.01 GHz
RAM: 2 GB
GPU: ATI Radeon HD 4800
OS: Windows XP SP whatever


My settings:

Resolution: 1280x1024
Antialiasing: 4x
Anisotropic: 8x
Ambient: Standard
Shader: Standard
VSync: OFF
Bloom: OFF


Results:

1st run: 2976 frames in 68.15 sec = 43.7 FPS
2nd run: 2976 frames in 46.6 sec = 63.8 FPS
3rd run: mostly the same as the 2nd


Come the time of peril, did the ground gape, and did the dead rest unquiet 'gainst us. Our bands of iron and hammers of stone prevailed not, and some did doubt the Builder's plan. But the seals held strong, and the few did triumph, and the doubters were lain into the foundations of the new sanctum. -- Collected letters of the Smith-in-Exile, Civitas Approved

#23 Chiron

Chiron

    Member

  • Member
  • PipPip
  • 192 posts

Posted 15 January 2010 - 06:00 PM

# CPU: AMD Athlon 64 X2 Dual Processor 4200+
# RAM: 2GB
# GPU: GeForce 7300 GS 512MB GP
# OS: WinXP Pro SP3
# Video Driver: Nvidia 191.07 WHQL


Resolution: 80O x 600
FSAA: Off
Anisotropic: 4x
Ambient: Standard
Shader: Standard
VSync: OFF
Bloom: OFF


1st run: 40.7 FPS
2nd run: 49.1 FPS


I am still wondering why my FPS sucks that much, even though my settings are already really low...is it the GPU ?

Edited by Chiron, 15 January 2010 - 06:01 PM.


#24 Midnight

Midnight

    Member

  • Member
  • PipPip
  • 446 posts

Posted 15 January 2010 - 09:56 PM

Here are the results of a test I did to compare each different setting and the overall effect they had on the FPS.

CPU : Intel Q6600 Core2Duo 2.4GHz
GPU : Nvidia Geforce 8800GTS (384 MByte)
Ram : 2 GByte (1066 MHz DDR3)
OS  : Windows XP SP3 
Resolution:  1680x1050

I wanted to keep the resolution native at 1680x1050, which I know
is one of the biggest framerate hogs, but I like full res :rolleyes:

Initial test achieved 25.6 FPS with all settings maxed out (except for Bloom, which was off for all tests).
The following represent turning off or altering each setting, whilst leaving the
others at max (to avoid muddying the results):

26.3 FPS - Anisotropic: 1x 
31.0 FPS - VSync:       Off
25.7 FPS - Shader: Standard
25.7 FPS - Ambient: Fast
50.6 FPS - Anti-Aliasing: Off
50.4 FPS - AA x4
49.7 FPS - AA x8
25.6 FPS - AA x16 (original test with all settings maxed out)

I was expecting anti-aliasing to be the largest hit on FPS, but I was surprised to see that
AA x4 and even AA x8 had fairly negligible effect on framerate, whereas AA x16 had a
huge impact on framerate. Luckily, I can't really tell much difference in visual quality
when using 16, so I'll not use this in future.

Since I can't stand not using VSync, the only other option available if a performance
boost is needed is lowering the resolution.

Edited by Midnight, 15 January 2010 - 09:59 PM.


#25 7upMan

7upMan

    Advanced Member

  • Member
  • PipPipPip
  • 1081 posts

Posted 16 January 2010 - 03:26 AM

My PC:


Melan, just out of curiosity, do you know what type of HD4800 your graphics card is? After all, there are no less than 4 types of the 4800 type on the market: 4830, 4850, 4870 and 4890, and that's not including the X2 and overclocked variants. And your CPU is a single-core AMD Athlon 64 3200+?
My Eigenvalue is bigger than your Eigenvalue.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users