About.com give a slightly different explaination:
"Tashlich
From Lisa Katz,
Your Guide to Judaism.
On Rosh Hashanah, observant Jews may follow a custom called Tashlich (casting off) in which they walk to flowing water, say a prayer, and symbolically throw their sins into the water.
Tashlich is done on the first afternoon of Rosh Hashanah. If Rosh Hashanah falls on the Jewish Sabbath, then Tashlich is done on the second day of Rosh Hashanah. If Tashlich was not done on Rosh Hashanah, it may be said anytime during the Ten Days of Repentance.
The practice of Tashlich is not discussed in the Bible, but it is a long-standing custom.
Tashlich Prayer
The Tashlich custom and prayer derived from Micah 7:18-20.
Who is like You, God, who removes iniquity and overlooks transgression of the remainder of His inheritance. He does not remain angry forever because He desires kindness. He will return and He will be merciful to us, and He will conquer our iniquities, and He will cast off our sins into the depths of the seas. Give truth to Jacob, kindness to Abraham, like that you swore to our ancestors from long ago.
It is estimated that in approximately the 14th century, the phrase "
cast off our sins into the depths of the seas" was transformed into a ritual of throwing bread into a body of running water. The act of throwing the bread can be viewed as a metaphor for casting off sins.
http://judaism.about.com/od/roshhashana/a/shana_tashlich.htm