Millwall had a glorious chance to take the lead inside the first two minutes when Alexander played himself into a good position after a neat one-two with Marc Laird on the edge of the box but he blasted over from six yards.
They were made to pay for the missed chance minutes later when Nicky Bailey put the visitors ahead with a brilliant volley.
Millwall failed to clear their lines effectively and the ball fell to Bailey 25 yards out who hit it first time and flew into the top left hand corner past a helpless Rhys Evans.
Neither side created many openings in what was turning out to be a scrappy game. Just after the half hour mark Southend had the chance to double their advantage.
Alex Revell crossed for Alan McCormack but his header was well saved by Evans.
Millwall pulled themselves back into the game against the run of play when Alexander rose to powerfully head home a Martin corner despite the best efforts of Darryl Flahavan in the Southend goal.
The home side started the second half a lot brighter and took the lead 10 minutes after the break.
The impressive Ali Fuseini played a long ball over the top of the Southend defence for Jay Simpson to run onto, but he was stopped by a handball by defender Peter Clarke on the edge of the box.
From the resulting free kick Martin curled the ball round the wall into the bottom right to give Millwall the lead.
Southend's misery was compounded when substitute Damian Scannell was sent off one minute after coming off the bench for a reckless lunge on Millwall's Zak Whitbread.