If the bullets leaves the muzzle at slightly different times, which they all do, and you have a lot of vibration, the muzzle can actually be pointing in different directions. Say you have a very consistent load that typically only has a 10fps variation of MV, but every few rounds you get one that is running 25fps different. If you have a bad harmonic vibration, that round is probably going to be a flyer.
When you talk about "working up a load" for a rifle you are looking for a load that causes minimal harmonic vibration so it will be more tolerant of other deviations in the load. If you don't load your own, try several good quality off the shelf loads to try and find one that produces minimal vibration in your particular rifle.