First time I ever took a spill was because of gravel and inexperience. I hate gravel on the road. I used to have to ride down a gravel road to see my girlfriend back in the day. I put up with it because girlfriend, but I grumbled every time.
Yeah gravel is a nasty thing. Last trip I took contained a few miles of gravel road and I hated it. Packed in gravel isn't so bad but when you hit those softer areas that swirly feeling isn't exactly my favorite.