gardeners and the gardens they make are fragile things
Unlike something like a brick wall which will last for some time, a well kept garden needs constant maintenance otherwise the lawn will become overgrown, and weeds will flourish. That means it's fragile.
creatures of time,
Gardens are affected by time, it's full of living things that if left unchecked will grow, smother some of the weaker plants and become messy. The lilies need to be rescued, because they're not as strong as the weeds which will smother and kill them over time.
hostages to chance and to decay.
Normally we say hostage to fortune. In this case whether or not the garden stays pretty depends on who lives there. If the owner moves or dies, and the house is left abandoned the garden becomes overgrown, and there's always a chance that will happen.