Hitler needed to invade England, he did not need to invade Russia.
Oh yeah, sure . . . he needn't have worried about insignificant things like the RAF and the Royal Navy. The first German fallschirmjäger
(paratroop) division did not reach full strength until 1941. In their first mission in Holland in 1940, their performance was a complete failure. They later did very well in Belgium, but the division consisted of just two under-strength regiments. It is highly unlikely that they would have been able to set up and hold an airhead while troops were ferried across the channel, which is the only thing that would have made a successful invasion possible. Had Hitler not interfered, Hermann Göring's plan to destroy the RAF in southern England might (might, mind you) have succeeded, at which time the Kreigsmarine would have had a very narrow window of opportunity to ferry and escort troops across the channel.
and Prinz Eugen
made their "Channel dash" in 1942, Adolf Galland says that he was able to defend them by keeping 50 aircraft overhead at all times. However, the two battleships, the cruiser and the destroyers and torpedo boats has a twelve hour start before they were detected. They weren't going to get that steaming across the Straits of Dover. Even with Galland's air cover, hundreds of aircraft succeeded in breaking through to attack. They were attacking a moving target--that would not have been the case in an invasion.
Such an invasion would have been "do-able," but certainly not assured, as your remarks seem to suggest. Hitler screwed the pooch when he interfered with Göring's attack on the RAF. The RAF bombed Berlin, and the idiot Hitler had an hysterical fit and ordered the bombing campaign to be switched to London. No invasion would have been possible without first destroying the RAF, and even then the Royal Navy made it a very dicey proposition. Hitler made sure it would never happen with his idiot decision to bomb London.