I just replaced my timing belt, which had 4 years and 32k miles. I drive the car every day, just about, and it runs in all seasons, here in Central NJ.
Both belts actually looked very good, and the timing belt had just a bit of glaze, and a small bit of fraying in one spot on the edge. I could definitely have gotten another year and 10K miles out of them both, and probably 20K miles out of the balance belt, but I'd rather replace them early then replace them later, along with valves, guides, head gasket, etc. You get the picture...
My biggest concerns with the timing belt are age in terms of years and oil getting on the belt and degrading it to the point of failure. I think that oil is a real problem, given that right near it there are 3 oil seals, any of which can leak and cause oil to get on the belt(s). The oil filter housing gasket is probably the most usual offender of oil leaks, big ones, but it doesn't affect the belts since it sits on the side and low, and during driving, air should push any oil leaking back from the front.
It sounds strange, but I think that the belts, along with many other things like oil seals, do better when they are run on a fairly regular basis. Sure they wear, but my hunch is that they stay more pliable and are less prone to cracking.