A government warplane opened fire to repel the attackers, local residents and a security source said on Tuesday.
The warplane strafed Boko Haram fighters fleeing in pick-up trucks after raiding Dille, near Lassa in the south of Borno State, for several hours on Monday. The attackers fired on inhabitants and burned homes and churches.
"I counted 26 corpses yesterday evening," one of the residents, Dauda Illiya, told Reuters.
Most of the deaths occurred during the raid but cannon fire from the government jet also killed at least six civilians - four women and two children, residents said.
"The pilot was just spraying bullets anywhere ... People were running here and there. Many people were injured from the bullets," said a local man, Suleiman Haruna.
Nigeria's defence headquarters in Abuja did not respond to a request for comment on the incident, but a security source in Borno State confirmed the deployment of the military plane.
The residents and the security source said 20 militants were killed by local vigilantes who fought back, but this could not be confirmed as witnesses said the raiders carried off their dead in their trucks.
Nigeria's armed forces are facing a fierce offensive in the northeast by the Islamist group Boko Haram, whose marauding bands of fighters have stepped up attacks against towns and villages after kidnapping more than 200 schoolgirls in April.
That abduction triggered an international outcry and increased support from Western governments for President Goodluck Jonathan's fight against Boko Haram, which has killed thousands and abducted hundreds since launching an uprising in 2009.
Boko Haram, whose name means "Western education is sinful", says it wants to set up an Islamic state in Nigeria, whose population is split between Christians and Muslims.
With the abducted schoolgirls still missing three months after their kidnap, Jonathan faces criticism at home and abroad over the deteriorating security situation in Africa's leading oil producer and biggest economy.
Authorities and military experts fear Boko Haram, which has claimed bomb attacks in recent months in the capital Abuja and in the coastal commercial hub Lagos, is seeking to push its insurgency into the more prosperous south.
BOMBING SUSPECT EXTRADITED
Nigeria's state security services said on Tuesday that a Boko Haram suspect and national army deserter, Aminu Sadiq Ogwuche, who was sought for alleged involvement in April 14 and May 1 bomb attacks in the Abuja area that killed 94 people, had been extradited to Nigeria from Sudan.
"Yes, he's with us", State Security spokewoman Marilyn Ogar told Reuters. She said Ogwuche was arrested by the international police agency Interpol in Sudan on a warrant from Nigeria.
The suspected militants who on Monday attacked Dille, which is not far from the Cameroon border, were believed to have come from Boko Haram's Sambisa forest stronghold where at least some of the kidnapped girls are thought to be held.
(READ MORE: Some of Nigeria's kidnapped schoolgirls may never return: Obasanjo)
On Monday, Jonathan promised that the missing 219 girls, snatched on April 14 from Chibok, also in Borno State, would soon return home, teenage Pakistani activist Malala Yousafzai said after meeting him.
Malala, who became a global celebrity after surviving being shot in the head by the Taliban for campaigning for girls' education, was visiting Nigeria to support the international campaign for the release of the abducted Nigerian students.