They asked him to resign, intensifying a power struggle within the ruling party ahead of an election next month.
The ruling Patriotic Front (PF) has been rocked by a bitter leadership dispute since president Michael Sata died in October and was replaced by his deputy Scott, who become Africa's first white head of state in 20 years.
(READ MORE: Guy Scott could be Africa's only white president, just for 90 days)
Scott is ineligible to run for president in a Jan. 20 election because his parents were born abroad.
Fourteen out of seventeen PF ministers supported the vote of no confidence and called for an urgent cabinet meeting to discuss Scott's removal, foreign affairs minister Harry Kalaba said.
"As cabinet we have noted that Dr Scott has persistently worked against the interest of the party," Kalaba said in a statement on behalf of the 14 ministers.
"Clearly the confidence and trust we reposed in Dr Scott was totally misplaced and he has since abused our trust."
Cabinet ministers have the right to remove the president in an official vote. Attorney General Musa Mwenye, who is a Scott supporter, is likely to oppose the move but despite being a powerful and influential figure he can only advise and cabinet could remove Scott without his approval.
Zambia's high court declared defence minister Edgar Lungu the PF's candidate this month but a faction loyal to Scott elected economist Miles Sampa as its leader and said it would take the matter to the supreme court.
The supreme court sent the case back to the high court to give Sampa a hearing that is expected to take place on Thursday.
Questions about Zambia's stability arose when Scott fired Lungu as PF secretary-general on Nov. 3, without explaining why, before reinstating him a day later after street protests.
Divisions in the PF could open the door to an opposition party candidate winning the election, an outcome that would create political uncertainty in one of Africa's most promising frontier markets.