Data size is indeed one of the key factors to consider when building vision systems, and in this respect, the current champ is Image Net: http://image-net.org/

They have images labeled according to the WordNet ontology, and the dataset is still growing. They also have more detailed annotations on subsets of the data for various things (object, attributes, etc.)