Thanks to the work of researchers from the Korean Gwangju Institute of Science and Technology and the Swiss EPFL, this needn't be the case anymore. In a paper titled "Point-and-Shoot for Ubiquitous Tagging on Mobile Phones" accepted to ISMAR 2010, they present a method to scan surfaces and create "recognition-models" by using your phone (no data is sent to a remote server).
You don't even need to take the perfect straight-on picture. As the video below shows, this means you can augment hard to reach surfaces. Best of all, you can share those models with your friends.
A little bit more detail over Wonwoo Lee's blog.