“There has to come a point at which you say: ‘Enough’,” Mr Obama said.
Mr Trump has been sharply criticised for attacking the parents of a fallen US soldier who spoke out against him.
He has also been condemned for backing the Russian annexation of Crimea.
Mr Obama said he had had policy differences with previous Republican presidents and candidates – but added that he had never thought they could not function as president.
Barack Obama has been steadily upping the pressure on Donald Trump’s Republican Party supporters. Last week, at the Democratic National Convention, the president said he didn’t think the nominee was within the mainstream of modern conservatism.
On Tuesday he called into question Mr Trump’s fitness to hold the presidency – his sharpest words so far – and questioned why the party leaders continue to stand by their man.
Those comments take dead aim at Speaker of the House Paul Ryan, Senate Majority Leader Mitch McConnell and 2008 presidential nominee John McCain, who all have issued statements distancing themselves from Mr Trump’s recent feud with the parents of a Muslim-American soldier killed in Iraq.
The irony, of course, is that Mr Obama’s remarks likely make it more difficult for the party’s top men to renounce their standard-bearer, lest they appear to be caving to their Democratic nemesis.
Instead, the squeeze will continue – and grow more politically intolerable with each new Trump controversy.
At last week’s Democratic National Convention, Khizr Khan – a Muslim whose son was killed serving in the US military in Iraq – criticised Mr Trump’s plan to temporarily ban Muslims from entering the US.
Mr Trump responded by attacking the “Gold Star” family, the term for families that have lost a close relative in war. Democratic and Republican leaders as well as veterans’ groups quickly condemned him.
“The Republican nominee is unfit to serve as president and he keeps on proving it,” Mr Obama said on Tuesday. “The notion that he would attack a Gold Star family that made such extraordinary sacrifices… means that he is woefully unprepared to do this job.”
Mr Trump released a statement on Tuesday afternoon responding to “President Obama’s failed leadership” but did not directly address the president’s criticisms.
“Our nation has been humiliated abroad and compromised by radical Islam brought onto our shores,” Mr Trump said.
Mr Obama also took to task Republican leaders who have frequently clashed with Mr Trump over this statements and policies about women, undocumented immigrants and Muslims.
“What does this say about your party that this is your standard bearer?” Mr Obama asked Republicans. “This isn’t a situation where you have an episodic gaffe, this is daily and weekly where they are distancing themselves from statements he’s making.”
Top Republican leaders such as House Speaker Paul Ryan and Senate Majority Leader Mitch McConnell have been highly critical of Mr Trump, but the billionaire has retained their support.
On Tuesday, New York Representative Richard Hanna became the first Republican member of Congress to publicly say he would vote for Democrat Hillary Clinton. Mr Hanna said Mr Trump’s comments about the Khan family were the deciding factor.
Until recently, many Republicans opposed to Mr Trump had stopped short of supporting Mrs Clinton, saying they would vote for a third party or write-in candidate.