http://www.answerbag.com/q_view/42727
What mechanisms are responsible for the reproductive isolation of organisms in a population?
(Skipped over the polyploidy in plants part-figured everyone would be snoozing by the time it got interesting)
.......Successful polyploidy is rare in animals (not unknown, just rare), however, there are other mutation events, including those altering chromosome number, which can lead to some individuals no longer breeding with their parent population. Behavior is also a key isolator, as some members of a population may change their behavior in a way which starts to limit their reproductive interaction with the parent population. Some example follow.
Equids (the "horse" family) are very good examples of altering chromosome numbers; in equids in general, the centromere section of each chromosome seems to be a lot more mobile than is usual in mammals, leading to chromosomes sometimes breaking into multiple, shorter chromosomes, and sometimes fusing into longer, single chromosomes. The chromosome number is amongst the most variable from one species to another of any animal genus in the world. *However*, and note this however, we are not talking about wholesale duplication of chromosomes right away, leading to unpaired chromosomes in sex cells and infertility -- we are talking about a change in the number of chromosomes because of breaking or fusing of the existing chromosomes. This means that other mechanisms come into play.
Two "daughter" chromosomes, a result of a chromosome splitting, can successfully line up against a normal, unsplit version of the original chromosome for sex cell formation *as long as the base sequences are sufficiently similar*. Similarly, a single "child" chromosome which is the result of two fused chromosomes can line up against its two progenitors *as long as the base sequences are sufficiently similar*. This means that an animal carrying a weird number of chromosomes can continue to be fertile, and interfertile with its parent population. However, this fertility is going to be slightly skewed -- interfertility with the parent population is likely to be lower than average, but a normal level of fertility can be recovered where two individuals have similar chromosome number and organisation.
What this means in practice is relatively simple. Say that a single individual acquires a mutation during sex cell formation, so that instead of chromosomes a, b, c, d, e, and f the sex cell contains chromosomes a1, a2, b, c, d, e, and f. That sex cell is fertilised by a normal sex cell, leading to an individual who has both patterns of chromosome (a,b,c,d,e,f AND a1,a2,b,c,d,e,f) in each cell. This individual is able to form sex cells through the mechanism described above, and exactly 1/2 of those sex cells will contain the new number of chromosomes. By the laws of chance, perhaps half, or slightly less than half, of this individuals offspring will therefore carry the new chromosome number -- so rather than only 1 individual in the population with this mutation, you now have, say, 3. Through the process just described, in the next generation you may have 6 or 7.
Unless the new chromosome number is actively selected *against* in some fashion, it will continue to spread in the population -- to the point that (voila!) some of the individuals carrying this mutation mate with each other, resulting in an offspring which has the new pattern from both parents. These individuals are a1,a2,b,c,d,e,f only. And here is where preferential fertility can really kick in -- most species are equipped with a number of mechanisms to detect those individuals most "like" each other, or most desirable in terms of fertility -- those individuals who can detect others with the same chromosome number, through whatever subtle phenotypic effects such a change might have, and who preferentially mate with each other rather than their parent population, will have increased fertility and more reproductive success. In this way a small subpopulation can stop breeding with the parent population and become an isolated population of its own.
Going back to the horses, this mechanism is what operated to isolate the domesticated horse from "wild horse" types like the Przewalski's horse -- domestic horses have 32 pairs of chromosomes, and Przewalski's horses have 33 pairs; and the two types of horse are still interfertile (can create fertile hybrids), though at a lower fertility rate between species than within them. Donkeys have 31 pairs of chromosomes and are no longer fully interfertile with domestic horses -- they can only produce sterile hybrids -- because in the time which has passed since the event which altered chromosome number, further differences have accumulated on the chromosomes which prevent the chromosomes from lining up together during meiosis.
Other mutations may have similar effects, though slightly different mechanism. A perfect example is the sympatric speciation currently happening between the two "races" of European corn borer, Ostrinia nubilalis (a moth). These are still arguably the same species, but several differences keep the races apart even when they inhabit precisely the same field: food-plant preference, maturation time, and sex-pheromone type.
The two races are the "hop-mugwort-E race", which feeds on hops and mugwort, but in addition on nearly 100 other seed and weed plants, which emerges up to 10 days earlier than the other race, and which communicates with the E-isomers of pheromones; and the "maize-Z race", which feeds solely on corn, emerges later, and communicates with Z-isomers of pheromones. The E race is the older one, native to Europe; the Z race originally emerged in France after corn (maize) had been brought there from the Americas, although now it has spread to corn crops throughout Europe and the Americas.
The two races are genetically very, very similar; however, it is known that the choice of food plant controls the time of emergence, which may well have been the beginning of assortative mating (those insects which emerge at similar times being far more likely to mate with each other than with insects which emerge either earlier or later), but the time difference is not complete; there is a period of about 5 weeks in which the two races are both present and sexually mature. The mutation which differentiates the sex pheromone may well have appeared some time subsequent to the beginning of assortative mating according to food plant, but would have spread preferentially in the one sub-population due to the fact that there is a preferential mating skew; and as it stands now, the assortative mating is nearly universal, and the two "races" ignore each other almost entirely because of the sex pheromone differences. That opens the way to further mutations accumulating which makes the two lines differ from each other -- we are actually _watching_ speciation happen, in this particular instance.
A recent paper detailing the situation is by Thibaut Malausa, Marie-Th?se Bethenod, Arnaud Bontemps, Denis Bourguet, Jean-Marie Cornuet, and Sergine Ponsard, "Assortative Mating in Sympatric Host Races of the European Corn Borer" (Science, 8 April 2005: Vol. 308. no. 5719, pp. 258 - 260), abstract available at
http://www.sciencemag.org/cgi/content/abstract/308/5719/258.
These do not represent a definitive list, not least because we keep finding new things; however, we can draw from these examples the idea that the isolating mechanisms depend on either genetic or behavioral changes in a small number of individuals out of a parent population, which either makes it impossible to mate with the parent population (as in polyploidy) or more successful simply to mate with each other. I hope this makes sense to you, and I'm sorry if it's too long-winded; to my mind, it takes knowledge of some detail in order to make sense of why & how such isolation works.