Assi -
I use my M4 for 3-Gun and occasionally for Trap just to practice manual of arms. I was never happy with the Ghost Ring and housing for the same reasons you mentioned. Even for non-flyer clays on a stand it disrupts your peripheral vision as you swing from target to target. Not sure why anyone would need a ghost ring on a shotgun anyway, but I'm sure someone on this forum will explain it to me! ;-{)
I put a Burris Fast Fire red dot on mine and it helps dramatically. You'll see others have done the same with a Trijicon RMR. The red dot sits well above the ghost ring housing so there is no obstructing your FOV, however, there is no way to co-witness the sights if your optic fails. It will change your cheek weld a little as well but I've rarely noticed it -- just adapted.
If I hadn't fallen in love with pictures of the M4 years ago and chased it down like the hottest blonde in the bar, I would have been better served with a 3-Gun purpose built M2. Lust at first sight I guess... . LIve and Learn (again).