Sand is bad but gravel on pavement is down right scary.
Stationed in Hawaii, riding a SR500:
Once at an intersection, turning left after stopping for the light, I was doing no more ran 10 mph and with no brakes at all, just barely above idle, the front tire went over a tiny patch of sand & graven right in the center of the intersection, no more than the size of a folded napkin, and the front wheel INSTANTLY went out and the bike slammed down on the pavement.  
Minor cosmetic damage and a scraped elbow and palm.
I am extremely careful of and watchful for sand and gravel on the road.
I don't like it on pavement.  I've not ridden dirt much but there is more control slide control on the dirl.

Yep...Gravel can be tricky.  Be careful when you see it and adjust the braking and turning forces if you can't avoid it.

Also don't underestimate how slippery wet leaves can be.

When I was in high school I worked at a Kawasaki dealer that had a dirt road behind that was sprayed with used motor oil to keep the dust down (it was a long time ago.....pre EPA).  The road beside the shop was brick, and traffic on the dirt road would track the oil out onto the brick......Now I want to tell you that oiled brick is really slick....even more so when it rained!