I believe this goes back to the time when the Olympic scoring was 0, 3, 5.7 etc. A zero on the score sheet looks like a bullet hole.
In most races, particularly older days, races were started an finished with a rifle.When you crossed the line they fire a bullet in the air. 2nd, 3rd etc get a hoot.
What about handicapping racing? It might not be a "Bullet" even if you cross the line first. You still got to worry that some guy on a slow boat to China will correct over you before you can take the "Bullet" to the bank.