@weiwei,
"I have three five-dollar notes" is correct. "Five-dollar" is a noun phrase used attributively as an adjective, modifying "notes." Thus, it must be hyphenated to avoid possible confusion. "I have three five dollars' notes," on the other hand, is incorrect. The possessive implies that the dollars own the notes, and that these dollars are named "three five."