does sound like a battery problem,  check connections to see if tight.  mine were loose, along with many other connections on my 04.  remember this,  it was assembled at the dealership by a mininum wage stoner dude, that is in most cases.  

A trouble point with more or less any and all vehicles I've owned/driven is the poor connection at the negative (-) post. I don't know why, but the positive post (+) is generally very tight, but the negative is generally looser.  
Get a small chipboard screw, the kind you'd need to hang a picture to the wall.
Loosen the negative (-) post a little, screw in the screw so that the threads bite into the lead/alloy post, then retighten the clamp.
The threads of the screw biting into the lead alloy will now ensure efficient contact no matter what.
Learned this trick some 30 years ago, kept me rolling ever since.

Really works.

Maurizio

Funny I had a very similar problem. I could push the start button 4-5 times and nothing... and then other times it would go on the first push. After a few days of this it really made me mad and my father showed me a trick. He said, "hold the button down BEFORE you push in the clutch, then grab the clutch with the button depressed." Works like a charm EVERY time. I thought it was strange till he told me that on certain bikes the relays are just funny like that. Hmmmph..

In my expirience with this kind of problems, the first thing to go bad on any Suzuki is the clutch switch. Now the first thing I do is to bypass it, even if it works good. The lazy way to do it is to disconnect the plug from the clutch lever and to jump it with a paper clip or just jump it from the back side of the plug while on the lever.

The starter switch is not that hard to take apart, I had the one on my '02 apart when I first got it and found a spider web in it. ??? Just be carefull and don't open it up over a grass lawn, there are small springs and connectors in it that are easy to loose. Clean the contacts with 600 grit wet/dry paper or a pencil eraser. Don't know how handy you are but I remember that a third hand would of been nice when putting it back together.

Well there's good news and bad news.

The good news is that I took the switch apart last night, and the contacts were badly pitted. I dressed them up; they're now nice and shiny and smooth and all seems fine.

The bad news is that based on what I see (size - I'm guessing less than 2 sq mm of contact and construction - no wipe on the contacts) this is going to be an ongoing problem/maintenance item. I'm surprised everyone with a few hundred starts on their bike haven't had the opportunity to either dress the contacts on this switch or replace it.

Dressing the contacts is not too hard to do. (However a brief search of the internet indicates that most manufacturers would recommend replacing pitted contacts rather than dressing them)
This is for a 2004.
Tools - phillips head screw driver, I used a #1 - something to dress the contacts with, I used a jewelers file and emery boads
Parts - A small wire tie to replace the one holding the wires in place, I just reused the original, but it would have been easier to reassemble if I had cut the original off and put a new one back on after the switch was tightened into place.

Two phillips head screws in the housing. Make note of the housing position if you want your throttle to be in the same position when you get done, because it does not positively position on the handle bars.
Two phillips head screws holding the switch in the housing. One of these is attached to a bracket that holds the wire harness in place. Don't drop these they're hard to find. There is a spring between the contacts that is loose at this point also.
The switch is soldered into place so the maintenance will have to be done on the bike unless you want to add a lot of work to the job. (If I was going to unsolder the switch, I'd just replace the switch.)
The movable contact of the switch comes out of the button and is really small. (Don't drop it, it will go under the bike requiring you to get on your hands and knees and use your trouble light to find it.)
Dress the contacts. (I'm not going to go into how to do this here, I'm guessing an internet search will reveal the proper technique.) Or just replace the switch if the part is available from ???
Replace the switch. Here is where things get really fiddly. There is a spring in between the contacts, the screws are really small and they fit down into the housing if you miss the hole. A magnetic screwdriver would have helped here, but I was too lazy to get off of the bike seat to go get one at this point.
Put the housing back in the same location as before. I didn't mark mine, so I had to guess. This I believe could affect the throttle slightly, so I tried to put it right back where it was.
1/2 hour start to finish including the time necessary to look for my jewelers files and emery boards which my son had moved. ("Oh Dad did you check on your bench, that's the last place I used them.") A trouble light to hang on the brake lever is nice, especially when you are on your hands and knees looking for parts. No pictures sorry.

I'm going to try to locate a replacement switch since this one was so badly pitted in under 6,000 miles. I'll probably replace it next time. Of course I don't know how many starts the previous owner put on the bike even though he didn't ride it, he could have been starting it.

One last comment - I've always pulled the clutch lever in before pushing the button. I think that doing this in the reverse order as suggested by QualitySeal, and WD would just move the wear and tear from the thumbswitch to the clutch switch. I don't know how easy the clutch switch is to work on, but it's hard to believe it would be easier than this switch. Changing the order of closing switches would help pick up the solenoid if one of the switches is resistive, so is probably a good emergency procedure, but I'm guessing if you need to do this, the thumbswitch probably needs maintenance already, and I'm guessing the clutch switch is not any larger than the thumb switch and so will also wear out. My .02

Savage_Rob

Yeah, that's a thought.

I'm going to find out what the cost and availability of the switch is and then what I might do is buy a replacement and then build up the contact surface on the old switch.