@tanguatlay,
Never the latter;" If we had moved the
structure we'd say "...the house." If we had moved only the self we'd say "We've moved," "We've moved to a new house" (somewehat unlikely), "We've shifted houses," or even maybe, "We've shifted house." (
with misgiving)