Andrew Puzder said he is withdrawing as President Donald Trump’s nominee for labor secretary.

The fast food executive said in a statement provided to The Associated Press that he was “honored to have been considered by President Donald Trump to lead the Department of Labor and put America’s workers and businesses back on a path to sustainable prosperity.”

Puzder said “while I won’t be serving in the administration, I fully support the President and his highly qualified team.”

Puzder’s confirmation hearing was scheduled for Thursday. But some Republicans had raised concerns about his failure to pay taxes for five years on a former housekeeper who wasn’t authorized to work in the US

Puzder is CEO of CKE Restaurants Inc. He’s been under fire for remarks on women and people who work for his restaurants.

Earlier this week Republicans had been strongly defending Puzder against allegations of not being fit for the position.

Senate Majority Leader Mitch McConnell pivoted to the most muscular defense of Puzder’s nomination, vowing to stick with him despite Puzder’s acknowledgement of having employed a housekeeper not authorized to work in the US.

Puzder fired the worker but did not pay related taxes until after Trump nominated him on December 9, 2016. Several GOP members of the Senate committee that were to have handled his confirmation hearing say they are concerned by the housekeeper issue. Employing people not authorized to work in the United States has sunk Cabinet nominations by previous presidents.

There was no evidence any Senate Republican would have voted against Puzder’s confirmation.

Puzder’s opponents had long laid the groundwork for their case that workers at Puzder’s fast food business, including Hardee’s and Carl’s Jr., were treated shabbily. They argue that the nominee opposes worker-friendly policies — such as a big minimum wage hike and an overtime rule that expands the number of eligible people. And they say that despite Puzder’s plan to divest from his financial holdings if confirmed, he favors business interests over worker protections enforced by the Labor Department. Their Twitter rallying cry against Puzder: #antilaborsecretary.

On Thursday, Senate Democratic Leader Chuck Schumer called on Trump and Puzder to withdraw the nomination, saying the pair’s business records and Puzder’s housekeeper issue fly in the face of the president’s vow to advocate for workers.