I think both have the same meaning, but that second one strikes me as having a different connotation. The first one feels like something normal - we need to leave at a certain time and now is that time. The second one has a feeling like something is wrong and we should leave. "Uncle Bob is drunk and is fighting with Aunt Mary, it's time we go home." (I wouldn't use "went"). Maybe others will chime in with a different take.