You're probably describing the problem. Barrels "whip" when fired. Since the M4 barrel is supported at the receiver, and by the muzzle, the middle likely is doing all the moving. Unforgiving metal materials, tighter fit all together, throw in some staking tolerance issues and it just doesn't have the ability to move like it needs to. The welds/braze/solder or whatever means they're using doesn't have the shear strength for the added force.
Probably a tolerances issue, because for every one person it breaks of for, there's probably fifty it doesn't.