Many born with silver spoons in their mouths are restless souls and just the same I'm sure, if you've tried to view someone as their whole and not their individual parts, you can find someone who lives beneath an overpass with the airs of nobility. Where you live, how you are born, and how you are raised are only small parts which make up our individuality.
I've met young unwed mothers who live in projects who are thoughtful individuals with hearts of gold and the willingness to give that which is priceless, time and love... and I'm sure we all know of wealthy individuals who are highly regarded for their "philanthropic efforts" and yet all they do is throw money at the problem. Likewise... we've all probably seen, met, or heard of an individual who is the exact opposite.
Some people are born wanderers... they are never satisfied with life. Others wander for a purpose... need it be spiritual, emotional, or financial.
Perhaps, for Sesshomaru, his wanderlust is a combination of all of these. He seems to be seeking something but he also seems to have motive and purpose... a destination, of sorts. (I personally don't think Sesshomaru does anything without a purpose. If the end justifies the means he would act like a snotty noble and hobnob with the best of them... but only if it meant getting what he wanted out of it.)
Also... call me a sinner but even if Rumiko Takahashi says he has no home in an interview doesn't mean he never did or never could.
Even manga-ka are not faultless or farseeing.
So yes, I agree that where he lives is entirely speculative.
P.S. The idea of Sesshomaru living with his mother? Priceless. That would be a hilarious fic. "Mom! We ran out of milk!" "Get a job, you bum!"