My Autopano Pro forum

Sharing knowledge around Autopano

You are not logged in.

Announcement

The documentation of AutoPano Pro can be found online at our Wiki. Before posting a question, please check with the manual!

#1 2008-02-27 09:27:44

[bo]
community overseer
From: Bulgaria
Registered: 2006-05-05
Posts: 1092

Benchmark system for APP

Alexandre wrote:

I was thinking of a 2 parts system : a benchmarking log system that would be available in the software ( computer details, speed, ram, cores,  etc ) and a set of standard panorama that everyone could grab and test on their computer : a benchmarking panorama cases repository. This benchmarking kit with input images will be accessible to everyone to make their own test. This way, we could really compare configuration between them because we use the same images sets.

I can only add that each case has to come with a .REG file that resets APP settings, as well as PROJECT file, so each user benches under the similar conditions. Obviously, only Render is worth benching.

It could be helpful to have at least three different set of images - small (that renders in 5 minutes on some reference system), medium (15-20 minutes render time) and large (1 hour).


Some of my panoramas, posted in the Autopano Pro flickr group.

Offline

 

#2 2008-02-27 11:06:48

fma38
Moderator
From: Grenoble, France
Registered: 2005-12-07
Posts: 2454
Website

Re: Benchmark system for APP

It could be usefull to give the disk space needed, especially for big projects...


Frédéric

Canon 20D + 17-40/f4 L USM + 70-200/f4 L USM + 50/f1.4 USM + Tokina 10-17 3.5-4.5 AF DX Fisheye
Merlin/Orion panohead + Papywizard on Nokia 770

Offline

 

#3 2008-02-27 16:57:20

hankkarl
Member
From: Connecticut, USA
Registered: 2006-02-21
Posts: 1137
Website

Re: Benchmark system for APP

[bo] wrote:

Obviously, only Render is worth benching.

Bo, you must have a much better system than I do.

It takes a while to load images from RAW or HDR files.  It takes a while to detect the pano.  It takes a while to do geometric analysis and to optimize a pano.  It even takes a while to bring up an image into the CP editor, remove an image, etc.  Sometimes just rendering after removing an image takes a while (due to the "need" to recalculate RMS), but this later mostly affects spherical panos.

It also takes a while to do color correction, or play with levels.

What do I mean by "a while"?  A while is anything over 2 seconds, the historic number for a computer's response to a human input.  But many operations, like optimization, can take minutes.

And if you remember the "nightmare pano" I sent to Alexandre, it took my system hours to detect the pano.

Note that there are also other system things that affect speed:
- antivirus, and if you turn off AV runtime for the APP temp directory
- all the other "stuff" on your pc -- firewall, antispyware, etc
- email that checks every xx minutes
- other background tasks
- Ethernet controller and other devices (ie dumb that uses the PC processor or smart that has its own processor).
- ethernet, firewire, USB speed if your disk is on those, and what else shares that link.
- viruses and malware
- temperature (some processors, especially laptops, slow down if they get too hot)
- laptop power--some slow down if on battery
- how full your disk is -- windows slows down at IIRC 20% free space and crawls at 10% free space -- don't ask what happens when you get below 5%

An automatic way to find this stuff is great, but we should expect a range of answers, even on the same system.

Last edited by hankkarl (2008-02-27 17:05:15)

Offline

 

#4 2008-02-28 10:25:28

[bo]
community overseer
From: Bulgaria
Registered: 2006-05-05
Posts: 1092

Re: Benchmark system for APP

Well, that's what I meant in the OP... Rendering is much more consistent than detection, also much easier to measure in time (than, say, color correction or moving an image), IMHO.

Those things you listed will affect performance, of course, but they would be the general, every-day "static" on your system, so you should get consistent results. 10 or 30 sec difference won't matter on a hour render.

Maybe the best thing to do is to have some king of standalone EXE and a set of images (RAW & TIFF & JPEG). When you run the EXE, it processes the images, detects a pano,  renders and saves a log file + sending report to a central server. And maybe run the test several times, so you get an average time...


Some of my panoramas, posted in the Autopano Pro flickr group.

Offline

 

#5 2008-02-28 14:39:18

hankkarl
Member
From: Connecticut, USA
Registered: 2006-02-21
Posts: 1137
Website

Re: Benchmark system for APP

Hi Bo,

I agree that rendering would probably be static on your system...if you don't run any other programs when APP is running.  Eg opening APP and PS takes a lot of memory smile

But even if you and I had the same system hardware, the software running in the background may affect things so one is noticably faster than the other.

Your idea is a good one, but we must all realize that we're getting approximate times, not times that are accurate to the microsecond.

Offline

 

#6 2008-02-28 15:18:08

fma38
Moderator
From: Grenoble, France
Registered: 2005-12-07
Posts: 2454
Website

Re: Benchmark system for APP

The accuracy of the tests will depend on how much results we have. It will need to launch the bench several times, in several conditions (on the same machine). Finally, grouping all resulats from all users, we will have a general line, saying what is more important (hd, memory, proc).


Frédéric

Canon 20D + 17-40/f4 L USM + 70-200/f4 L USM + 50/f1.4 USM + Tokina 10-17 3.5-4.5 AF DX Fisheye
Merlin/Orion panohead + Papywizard on Nokia 770

Offline

 

#7 2008-03-01 20:57:14

foundation
Member
Registered: 2007-01-15
Posts: 122

Re: Benchmark system for APP

Keep in mind guys that the more exhaustive and controlled you make the benchmark, the less likely large numbers of people are going to do it. Also, while I agree we need a baseline for the preferences, part of what I'd like to see is how the different preferences affect the render times. We also don't want to spend the development team's time on coding changes for us to benchmark with, I'd much rather have new features! That being said, hopefully adding a total time elapsed recording for the batch renderer wouldn't be too onerous for the Kolor team in the next beta to save us having to sit with a stop watch.

Offline

 

#8 2008-03-02 08:46:32

DrSlony
Member
From: Poland, Zielona Góra
Registered: 2007-11-03
Posts: 946
Website

Re: Benchmark system for APP

Poor Alexandre :]

Offline

 

Board footer

Powered by PunBB
© Copyright 2002–2005 Rickard Andersson