I think milage is important, but only when you reach the 80,000 -100,000 mark. 40,000 will be no problem, as you said, that only amounts to 3,000 per year - if it is a bike you want then go for it, assuming you have fully checked it over of course!
My bike is 11 years old and has 11,000mi on the clock. I saw it as a bonus rather than a requirement when I bought it a few months ago, the majority of 'old style' Fazers have 30,000+ and run perfectly well.
Anyway, these are generally the factors I consider in order importance: general condition, cost in near future (eg. tyres, chains etc), service history, mileage, age.