I have both. Am still restoring my xs650, has taken me a year already.
Bad points:yes handling - swingarm bushings are made of plastic! good quality retrofit ones available. Not a fun job though.
older ones still run points.
clutch is not up to modern standards and it's a pain to get it adjusted right. Finding nuetral at stop light a bit of a dance.
Good points:much easier cafe conversion than the savage!
beautiful engine look and sound.
loads of parts about new and old.
Basically it isn't the best bike (xs650) out of the box but if you are prepared to mess around every weekend for a long time you will get a very cool bike. Not sure I'd do it again but I'm in too deep now.
Got the savage cause I didn't want the hassles of the XS650.
Oh and the XS650 will do two up a lot better than the savage if that's improtant, otherwise I vote Savage.