Our visual memory is a strong tool, but drawing always from life does not train us to recall what we have seen. On a workshop years ago I heard from the tutor that it is possible to draw almost everything we have seen from memory quite accuratly. It might take some effort to recall visual impressions of the past in a wake state, but why shouldn´t we be able to imagine things as clear as in many of our dreams?
Recently I have been doing many sketches and drawings from memory as for example the little tree landscape in this post.
Compared to Stephen Wiltshire my capabilities are simply embarrassing. I tend to think that Giovanni Piranesi had similar brain capacities to store uncounted details of buildings in his mind as there are hardly any detailed preparatory sketches or drawings by Piranesi.