I have yet to install a dot optic on my M4. I have thought about it but I am a huge fan of the ghost ring sights. I have no issues quickly acquiring a target with them.
In fact these type of open sights which are the same type that are on the M1 Garand and M14 rifle are considered to be the best designed open sights.
Maybe a stupid question but, if mounting the Trijicon RMR on the pic rail instead of using a Scalar mount, when looking thru it, do you see the front sight in the FOV?