Request for Comments - Augmented Reality Benchmark

Request for Comments - Augmented Reality Benchmark

Here is a crazy thought - is the industry mature enough for an AR benchmark? Specifically I thought such a benchmark should cover the following two topics:

• Comparing the performance of AR frameworks (such as Qualcomm's AR SDK, Samsung SARI, etc.) using a given set of tests. 
• Comparing the performance of devices using a given set of AR tests.

Which leads to the following questions, I'll be glad hearing your thoughts about them (in the comments, on twitter, or via email, whatever!):

• Will such a benchmark be beneficial to any one? Interesting?
• What tests should the benchmark include? The main problem here is that test should be easily recreated and yet meaningful. We can test "tracking success" by tracking an on screen marker that is transformed by code (to account for change in lights, motion blur, noise and other effects, which can be hard to accurately recreate using a real physical marker).
• Would any one of you volunteer to run those tests? I don't have access to lots of AR capable devices, so unless you would like to sponsor me (i.e. buy me an iPad2), I count on the volunteers to run the actual tests and report back.
• How about benchmarking AR applications? Does it make sense?