I can see why you'd have trouble.
Also, there is a major variation between different kiwis as to how strong the kiwi accent is.
To my ear, some US accents sound the same to my ear when they come from different areas, some are clearly different.
On the whole I'd expect most English speakers to be much better at discerning the difference between various US accents than you guys are at differentiation between say, England, Oz, South Africa and NZ. This is because we get bombarded by US shows and you guys seem rarely to broadcast stuff from other English speaking countries.
In the US most people think I am from England, for instance.
One of many things that struck me while I was in Australia was not only the number of American television shows aired in prime time but the number of Australian shows that were remakes of US series.
I agree that we don't air as many shows from other English speaking countries as they may air US shows, but a big part of the reason for the imbalance is the size of our entertainment industry and the scope and variety of its offerings. Having said that we do get a good bit of programming from the UK and to a certain extent the UK is to the US what the US is to Australia. A great many of our American made TV shows are reworkings of British series. In fact I would go so far as to suggest that we have copied far more shows of the UK than most Americans realize. A fair number of Americans know that "The Office" is a reworking of a British show because Ricky Gervais has fairly wide exposure here, however I doubt very many people know that the hilarious series in which Redd Foxx starred, "Sandford & Son" was based on the UK series "Steptoe & Son," that "Three's Company" was based on the UK's "Man About The House" (Frankly, I think we could have gotten by just fine without that import, but it was very popular among my countrymen) or that the hugely popular and critically acclaimed "All in the Family" was based on the British "Till Death Do Us Part."
For the most part I think our television industry has done a very good job adapting UK hits for American audiences, but there are some notable exceptions. Our "Top Gear" for example is a very pale imitation of the UK show and neither "Payne" nor "Amanda's" couldn't hold a candle to the brilliant "Fawlty Towers" upon which they were based. I do think that in a few cases our "copies" are superior to the original, but they tend to be when the UK series is a broad comedy. For all the talk of the dry, subtle with of the Brits a lot of their comedy (on TV at least) is pretty broad. It's all a matter of taste of course.
Our Public Broadcasting network PBS might as well be BBC America for all the UK shows it runs, but we already have a BBC America where we can catch "Doctor Who" among others. One UK show which I would like to see our TV people try to copy is one of my favorites, a strange and darkly funny show called "The Misfits." It would be tough to match the quality of casting and writing for what I also considered the two main leads: "Nathan" played by Robert Sheehan and his "successor" character when Sheehan left the show, "Rudy" played by Joseph Gilgun. The succession was one of those extremely rare instances when a brilliant character played brilliantly by an actor is replace by another that is equally brilliant and played with equal mastery. When Sheehan left the show, I thought it spelled doom for the series. If you've not seen it, it's basically an ensemble cast of all good actors playing interesting characters, but Nathan
was clearly the show's sun around which the other characters revolved. Thankfully when he left, the producers didn't try to keep Nathan
as a character and simply recast the role. I've seen this work at times but it's a very tough thing to pull off. Instead they created a new character "Rudy" who replaced Nathan
as the focus of the show's wild and bizarre humor but who never-the-less had a very different personality all his own. Robert Sheehan is/was a very good looking young man with a load of charisma (I'm surprised his career didn't explode after he left) while Joseph Gilgun, to put it bluntly, is quite funny looking but strangely, equally charismatic. I haven't seen Gilgun in anything since then and Sheehan in only one other non-lead role. Perhaps they have thriving careers in the UK. I certainly hope so. The only member of the ensemble to have wider exposure (in the US at least) since his role ended is Iwan Rheon who played the shy nerd "Simon" in the series, but who now plays the sadistic psychotic, Ramsay Bolton
in "The Game of Thrones."
Doing a little searching while writing this post I found that Joseph Gilgun with be starring in AMC adaptaion of the comic book "Preacher" airing this year. Looking forward to that. I also found that he appeared in another excellent UK show that airs on BBC America, "Ripper Street," but I obviously missed that episode. Apparently he played a Fagin like character which I can see him doing perfectly.
Personally I have no problem differentiating between the accents you've cited (I now have kiwi down after my trip and the series " The Flight of the Conchords"), but I've probably had more exposure to them than a lot of Americans. Unless you speak like Crocodile Dundee or Steve Irwin I'm not entirely surprise that you might be mistaken for British by Americans. I met quite a few Aussies who could pass (no offense intended) for Brits here in the US although everyone of them had the Oz up-speak that gives you away.
It is tough sometimes to distinguish between Canadian and American accents, but it doesn't take long for a Canuk to give themselves away with certain words. Obviously "aboot" is one but not every Canadian says the word that way whereas virtually all of them pronounce Organization with a long "i." It a dead give away.