Request for Comments - Augmented Reality Benchmark

Here is a crazy thought - is the industry mature enough for an AR benchmark? Specifically I thought such a benchmark should cover the following two topics:

  • Comparing the performance of AR frameworks (such as Qualcomm's AR SDK, Samsung SARI, etc.) using a given set of tests. 
  • Comparing the performance of devices using a given set of AR tests.
Which leads to the following questions, I'll be glad hearing your thoughts about them (in the comments, on twitter, or via email, whatever!):
  • Will such a benchmark be beneficial to any one? Interesting?
  • What tests should the benchmark include? The main problem here is that test should be easily recreated and yet meaningful. We can test "tracking success" by tracking an on screen marker that is transformed by code (to account for change in lights, motion blur, noise and other effects, which can be hard to accurately recreate using a real physical marker).
  • Would any one of you volunteer to run those tests? I don't have access to lots of AR capable devices, so unless you would like to sponsor me (i.e. buy me an iPad2), I count on the volunteers to run the actual tests and report back.
  • How about benchmarking AR applications? Does it make sense?

4 comments:

Gary Hayes said...

A quick reply:

Yes benchmarking is good and at different levels - consumer experience, developer level performance.

Be worth knowing what is defined as AR here - just recognising images and markers ? Might be worth looking at performance of GPS integration/locking and so on.

But overall hard to do as the tech is changing so fast that any benchmark will be outdated as it is published in such an immature area -

One thing I would like to see is much more on consumer experience across locative, recognition apps and devices...particularly the experiences on dedicated builds Q sdk, string, Sari etc: vs browser based more limited experiences...also which devices particularly android are best for AR and so on -

Sorry just a quick response in middle of other things...hopefully others will chip in with more thought through responses :)

Joey1058 said...

You have benchmarks only as far as early adopters will provide feedback. There are still those of us holding out for something better. Of course what that something better is, remains a separate debate.

Anonymous said...

quick respond. just waste time as they said earlier, for now, focus on implementation how to use in our daily life, until the market accept it then we talk benchmark

Anonymous said...

AR paradigm is clearly overpriced. So you learned to map the points and triangulate. Great. So what?

Post a Comment