The speaker of parliament said this was decided after a row with the president that has paralysed the state and threatened a shaky recovery from war.
The prime minister, who was appointed last year, fell out with President Hassan Sheikh Mohamud over what legislators said was a dispute over the composition of a new cabinet, prompting Monday's no confidence vote.
The rift has worried Western donors who say it will disrupt efforts to shore up nascent institutions of state and drive back al Qaeda-linked Islamist militants who still control swathes of countryside.
Parliament speaker Mohamed Sheikh Osman Jawari said 184 lawmakers voted against the prime minister, while 65 members supported him. The prime minister's deputies are expected to handle his duties until a replacement is found.
The political turmoil follows a row about corruption related to the central bank. That had already rattled donors who see the probity of that institution as vital to unlocking future debt relief and rebuilding state finances.
Diplomats said the public had been growing increasingly frustrated with the government for not delivering change fast enough. The scale of the task is huge after two decades of war and chaos in which Somalia was carved up into clan fiefdoms and then ruled by al Shabaab militants.