I can see why you'd have trouble.
Also, there is a major variation between different kiwis as to how strong the kiwi accent is.
To my ear, some US accents sound the same to my ear when they come from different areas, some are clearly different.
On the whole I'd expect most English speakers to be much better at discerning the difference between various US accents than you guys are at differentiation between say, England, Oz, South Africa and NZ. This is because we get bombarded by US shows and you guys seem rarely to broadcast stuff from other English speaking countries.
In the US most people think I am from England, for instance.