I think there has to be some sort of balance between the two -- what you propose would not be able to accomodate those wanting to view the events.
Do you really honestly think they could hold an international swimming competition of the likes of the olympics at a local swim club? It would be difficult to even have the competitors fit it such a venue forget about the coaches, spectators, etc.
But I do agree building huge venues that later become unused does not make sense. Rather I feel it should be well thought out on where/what make sense. For example would it make more sense for say a country to host the event - but have the competitions in various areas of the country where it would make most sense and have the capacity to host that particular sport rather than build to suit?
Hold track for instance, in an area that already has a prominent track capacity - then limited building to suit would be needed. Holding all the sports in one area makes it difficult -- simply for capacity.
As an aside -- I honestly would love to have this event in Boston - simply from a selfish side of view, I would love to have it in my own backyard. Granted I would probably work from home during a good chunk of it as I'd imagine it would be next to impossible to commute to work.