Some writers suggest it comes from the remoteness of left field, but only in very asymmetrical ballparks is left field more distant than right field.
If "out of left field" meant "coming from a long way away," then I suppose that might make some sense, although, as the writer notes, left field isn't any more distant than right field. But since the phrase means "from a totally unexpected direction," there's no link between the distance and the fact that it comes from left field (indeed, the farthest point from home plate is in center
field, not right or left field).
Others suggest it alludes to the 'wrongness' of left as opposed to the 'rightness' of right.
I think this is closest to the truth. Left fielders don't carry any kind of stigma, but left-handed pitchers have a longstanding reputation for being sorta' flaky.
A correspondent of William Safire's in the "New York Times" said it was an insulting remark made to those who bought left-field seats in New York's Yankee Stadium during the years that Babe Ruth played right field, putting them far away from this outstanding player.
Yeah, that doesn't make a whole lotta' sense.
Perhaps the most likely theory is that it alludes to inmates of the Neuropsychiatric Institute, a mental hospital, which was located behind left field in Chicago's old West Side Park. Hence being told you are 'out in left field' would mean you were accused of being as peculiar as a mental patient.
This is a great story, and it might
even be true. The old West Side Park was located on a rectangular block bounded by Taylor, Polk, Wood, and Wolcott streets
. Home plate was at the northwest corner of the block at Polk and Wolcott, and the left field wall ran along Wood street on the east. Cook County hospital was across Polk street to the north of the park. The Psychiatric Institute
was located at 1601 W. Taylor, so it was over the left field wall, but it wasn't across the street (instead, it was about two blocks away). As this photo from 1905 shows, the buildings over the left field wall appear to be residential structures.
It's possible, then, that "out of left field" might have referred to the mental hospital located over the left field wall at West Side Park, although I think that's a stretch.