Sir Keir Starmer has challenged government agencies to make efficiencies using AI and technology, ahead of the Spring Statement.
The prime minister was speaking after Rachel Reeves confirmed civil service departments will be asked to cut 15% from their budgets.
There is political unease among Labour MPs about the scale of cuts to government departments, which the chancellor will outline in her statement on Wednesday. It comes after the government unveiled sweeping changes to the benefits system, aimed at saving £5bn a year by 2030.
On Monday, Transport Secretary Heidi Alexander denied the government was signalling a return towards an austerity model.
Challenged on BBC Radio 4's Today programme, Alexander highlighted that Labour had increased public spending in its first budget in October.
"We have to take some really very difficult decisions and in terms of welfare we will always make sure that there is a system that provides a safety net for the most vulnerable," she said.
"We're investing over £20bn more in the NHS and we've seen waiting lists coming down in hospitals now for the fifth month in a row.
"The Office for Budget Responsibility last year at the time of the budget said it was one of the most expansionist budgets ever, of all time, so we are not returning to austerity."
Reeves' plan for major money-saving from the welfare budget comes on top of her decision to axe the universal provision of winter fuel payments and only give them to the poorest pensioners.
She told the BBC's Sunday with Laura Kuenssberg programme that government running costs would be cut, with savings made from back office and administrative roles rather than front-line services.
On Monday, speaking to BBC Radio 5 Live, Sir Keir said he was setting the same challenges for government that he was for business.
"One of the areas we will be looking at is can we run the government more efficiently, can we take some money out of government, and I think we can.
"We're essentially asking businesses across the country to be more efficient, to look at AI and tech in the way that they do their business.
"I want the same challenge in government, which is why shouldn't we be more efficient, so we will be looking there and in one or two other places."
Civil Service unions have warned that services could deteriorate with the loss of jobs, with Public and Commercial Services union general secretary Fran Heathcote saying that former prime minister Gordon Brown had attempted to cut backroom staff, and the "consequences of that was chaos".
Sign up for our Politics Essential newsletter to read top political analysis, gain insight from across the UK and stay up to speed with the big moments. It'll be delivered straight to your inbox every weekday.