To know a society, you must be there and not there. You need to experience it, but you need to be detached. I think that it should more be a mental exercise, than literally banishing oneself to a mountaintop. Objectivity is a valuable tool, but we need to know what we're talking about, wouldn't you agree?
I would also like to add that I think anyone can philosophise, anyone can analyse and ascertain certain moral ideas or social truths, be they a spiritual teacher or, like myself, a cleaner.