Interesting
When the petcock is in prime, there's a little finger that pushes on the diaphram to open the valve.
I suppose that in the ON setting, the vacuum fluxuates enough to act as a pump. And the valve is closing fast enough to over pressure the float valve.
Not sure how old your bike is, but, replace the vac line, clamp the ends and see what happens.