Sharing knowledge around Autopano
You are not logged in.
Alexandre wrote:
I was thinking of a 2 parts system : a benchmarking log system that would be available in the software ( computer details, speed, ram, cores, etc ) and a set of standard panorama that everyone could grab and test on their computer : a benchmarking panorama cases repository. This benchmarking kit with input images will be accessible to everyone to make their own test. This way, we could really compare configuration between them because we use the same images sets.
I can only add that each case has to come with a .REG file that resets APP settings, as well as PROJECT file, so each user benches under the similar conditions. Obviously, only Render is worth benching.
It could be helpful to have at least three different set of images - small (that renders in 5 minutes on some reference system), medium (15-20 minutes render time) and large (1 hour).
Offline
It could be usefull to give the disk space needed, especially for big projects...
Offline
[bo] wrote:
Obviously, only Render is worth benching.
Bo, you must have a much better system than I do.
It takes a while to load images from RAW or HDR files. It takes a while to detect the pano. It takes a while to do geometric analysis and to optimize a pano. It even takes a while to bring up an image into the CP editor, remove an image, etc. Sometimes just rendering after removing an image takes a while (due to the "need" to recalculate RMS), but this later mostly affects spherical panos.
It also takes a while to do color correction, or play with levels.
What do I mean by "a while"? A while is anything over 2 seconds, the historic number for a computer's response to a human input. But many operations, like optimization, can take minutes.
And if you remember the "nightmare pano" I sent to Alexandre, it took my system hours to detect the pano.
Note that there are also other system things that affect speed:
- antivirus, and if you turn off AV runtime for the APP temp directory
- all the other "stuff" on your pc -- firewall, antispyware, etc
- email that checks every xx minutes
- other background tasks
- Ethernet controller and other devices (ie dumb that uses the PC processor or smart that has its own processor).
- ethernet, firewire, USB speed if your disk is on those, and what else shares that link.
- viruses and malware
- temperature (some processors, especially laptops, slow down if they get too hot)
- laptop power--some slow down if on battery
- how full your disk is -- windows slows down at IIRC 20% free space and crawls at 10% free space -- don't ask what happens when you get below 5%
An automatic way to find this stuff is great, but we should expect a range of answers, even on the same system.
Last edited by hankkarl (2008-02-27 17:05:15)
Offline
Well, that's what I meant in the OP... Rendering is much more consistent than detection, also much easier to measure in time (than, say, color correction or moving an image), IMHO.
Those things you listed will affect performance, of course, but they would be the general, every-day "static" on your system, so you should get consistent results. 10 or 30 sec difference won't matter on a hour render.
Maybe the best thing to do is to have some king of standalone EXE and a set of images (RAW & TIFF & JPEG). When you run the EXE, it processes the images, detects a pano, renders and saves a log file + sending report to a central server. And maybe run the test several times, so you get an average time...
Offline
Hi Bo,
I agree that rendering would probably be static on your system...if you don't run any other programs when APP is running. Eg opening APP and PS takes a lot of memory ![]()
But even if you and I had the same system hardware, the software running in the background may affect things so one is noticably faster than the other.
Your idea is a good one, but we must all realize that we're getting approximate times, not times that are accurate to the microsecond.
Offline
The accuracy of the tests will depend on how much results we have. It will need to launch the bench several times, in several conditions (on the same machine). Finally, grouping all resulats from all users, we will have a general line, saying what is more important (hd, memory, proc).
Offline
Keep in mind guys that the more exhaustive and controlled you make the benchmark, the less likely large numbers of people are going to do it. Also, while I agree we need a baseline for the preferences, part of what I'd like to see is how the different preferences affect the render times. We also don't want to spend the development team's time on coding changes for us to benchmark with, I'd much rather have new features! That being said, hopefully adding a total time elapsed recording for the batch renderer wouldn't be too onerous for the Kolor team in the next beta to save us having to sit with a stop watch.
Offline