That's a good idea.
We know a family with a kid a bit younger than sozlet who, when we first met, only spoke Polish -- she definitely got exasperated, though she enjoyed spending time with him in general (they waltzed a lot

). Since then he's picked up a lot of English though and now they're mostly talking in English.
So that's kinda both ways -- kids pick up other languages scary-quick, especially in an immersion environment, but there's likely to be exasperation at the beginning.
Some sort of activity that doesn't
require language sounds good, if only digging holes.