The decree stated that Mr Luzhkov, 74, was removed because he had lost “the trust of the president of the Russian Federation”.
In recent weeks Mr Luzhkov – who has been in office since 1992 – had faced harsh criticism from the Kremlin.
The move follows weeks of speculation that Mr Luzhkov would be be forced out after disagreements with the president.
Mr Medvedev signed the decree during a visit to China.
The document added that the mayor’s first deputy, Vladimir Resin, had been appointed the acting head of Moscow.
Russia’s constitution allows the president to fire Moscow’s mayor and regional governors and appoint successors without elections.
After returning from a week’s holiday in Austria on Monday, Mr Luzhkov said he would not stand down as mayor voluntarily. He has not publicly responded to Mr Medvedev’s decree.
Mr Luzhkov has recently been the subject of a constant barrage from state-run TV, which criticised him for gridlock on the capital’s roads and bulldozing of historic buildings. He and his billionaire wife Yelena Baturina have been also accused of corruption.
Mr Luzhkov has denounced all the claims as “total rubbish”, designed to make him “lose his balance”. He has threatened to sue the TV channels concerned.
Mr Luzhkov is one of the most powerful politicians in Russia and is a senior member of the pro-Kremlin United Russia party.
Before the emergence of Vladimir Putin a decade ago, Mr Luzhkov was even tipped by some as a possible future president of Russia, the BBC’s Steve Rosenberg in Moscow says.
And, despite the daily traffic jams in the capital and the allegations of corruption, he has remained a popular mayor, our correspondent adds. – BBC