With two minutes left on the clock, Kay rose above the Millwall defence to head home Paul McLaren's free-kick and give Rovers their first win in six games.
But it was tough on Millwall, who remain in the relegation zone despite subjecting the visitors to a stern second-half examination.
Clearly in buoyant mood after the midweek FA Cup win over League One promotion pushers Walsall, the Lions enjoyed the better of the early exchanges and were unfortunate not to draw first blood.
Jay Simpson tested visiting goalkeeper Danny Coyne with an early cross-shot before Gary Alexander and Adrian Forbes both went close.
It was a lacklustre start by Ronnie Moore's side, most of whom looked physically and mentally weary after the midweek Cup defeat against League Two side Hereford United.
An honourable exception was Jennison Myrie-Williams, the forward on loan from Bristol City, whose pace and trickery along the right wing looked the only promising outlet for the visitors.
But Millwall could not make their superior possession tell and on 22 minutes they were almost made to pay when a fumble by Lenny Pidgeley, the Millwall goalkeeper, allowed Calvin Zola to turn the ball home only for the effort to be ruled out for offside.
Simpson, a lively presence throughout on Millwall's right flank, led the Lions' response, skipping to the by-line before delivering a low, skidding cross for Alexander, who had the goal at his mercy but could connect only with fresh air.
Such profligacy allowed Ronnie Moore's men to reach the interval with the scoreline intact, but Millwall continued to batter the visitors' beleaguered defence after the break.
Forbes forced a fingertip save from Coyne with a stinging long-range effort and the Rovers keeper was again called into action by a quick turn-and-shot from substitute Neil Harris.
Paul Robinson then had a header cleared off the line from Dave Brammer's far-post corner and with the pressure building on Rovers, it looked like there could only be one winner.
But with two minutes remaining, Kay rose above the Millwall defence to nod home McLaren's free-kick and give Rovers their first win in six games.