I just depressed the button, slid the stock all the way up the tube, turned it counterclockwise, and pulled it off (like you suggested).
In other news, don't buy Winchester 12GA bird shot in those bulk 100 round Walmart packs. It doesn't like to cycle (powder charges vary ENORMOUSLY) in my M4, and 2 - 5 rounds out of 50 had NO SHOT IN THEM AT ALL. It was a primer, powder, and wad... I watched the wad fly through the target with zero recoil. Perhaps there was no powder in them either? Regardless, I switched back to Federal bird shot (in the 100 round Walmart pack), and had no problems.
At any rate, my rig consists of the Surefire M80 rail, 7-shot tube extension, Surefire M961XM07 weapon light (large, but I got it dirt cheap), Magpul AFG, and Trijicon TriPower reflex sight (which is absolutely sick on an M4, by the way). She runs crisp.