If there’s one part of the economy that is doing well, it’s the mortgage industry. Record low mortgage rates are driving so many people to refinance that the housing market is on track to see $2 trillion in refinances in 2020, second only to 2003, according to the Mortgage Bankers Association.

But beginning December 1, refinancing will likely cost homeowners more. Fannie Mae and Freddie Mac will charge a new – and hotly contested – “adverse market fee” on refinanced mortgages.

Fannie and Freddie, which guarantee roughly half of the country’s mortgages, do not directly give mortgages to borrowers but instead buy mortgages from lenders and repackage them for investors. The new fee will be levied on the lenders, requiring them to pay an extra 0.5% of the loan amount as a one-time charge to the total loan amount.

That comes out to about $1,400 on an average mortgage, according to the Mortgage Bankers Association.

But the fee will likely be felt by homeowners as lenders are expected to pass the cost on.

“Whether you pay it up front or over the life of a loan, you’ll pay it,” said Mike Fratantoni, chief economist at MBA. “And it may make the difference of whether it is worth it to refinance or not for many people.”

He estimates that the added cost will result in a refinance that is 10 to 15 basis points higher than a purchase loan.

Nearly 20 million homeowners could benefit from refinancing at current rates, according to Black Knight, a mortgage data company. That includes 4.5 million homeowners who could save at least $400 a month and 2.7 million who could save $500 or more each month by refinancing at today’s rates.

Fannie and Freddie argue the fee will not cause the cost of homeowners mortgage payments to go up because a refinance usually lowers payments. They say for an average refinanced mortgage, homeowners can estimate a reduction in savings of about $15 per month due to the fee. Refinancing homeowners who were previously saving $133 on their monthly payments, will now save $118 per month, on average, they said.

Initially, the fee was supposed to go into effect on September 1. But following industry and political backlash, the Federal Housing Finance Agency, which is the regulator that oversees Freddie and Fannie, announced that it would delay the implementation of the fee until December 1.

The agency also added that loans with balances below $125,000 will be exempt from the fee, as well as the affordable refinance products Home Ready and Home Possible.

Jumbo loans, or those too large to be purchased or guaranteed by Fannie and Freddie, are not directly effected, although those loans carry higher interest rates to begin with. Borrowers getting FHA, VA, USDA Rural, or other loans not conforming to Fannie Mae or Freddie Mac standards are not subject to the fee either.

It is possible to avoid the fee by refinancing with a bank or online lender that originates and either holds onto the loan or sells it to private investors, rather than selling to Fannie or Freddie. But those kinds of loans often come with higher rates.

This year, mortgage rates have repeatedly hit record lows. The rate on a 30-year-fixed-rate mortgage fell from 3.65% in mid-March, to 2.27% in mid-November, the thirteenth record low this year, according to Freddie Mac.

The extraordinarily low rates are driving a lot of the volume, said Fratantoni, both for loans to purchase a home and also to refinance a current mortgage.

Fratantoni said this fee will be baked into the pricing from now on.

“Our forecast is that mortgage rates will be trending higher in 2021,” he said. “That was going to be happening anyway leading to lower refinance volumes next year. This fee will add to that.”

The FHFA says that the fee is necessary because it is needed to cover billions in Covid-19-related losses at Freddie and Fannie. Specifically, actions taken by the enterprises during the pandemic to protect renters and borrowers – including offering forbearance programs, buying loans in forbearance and modifying mortgage terms to reduce monthly payments and simplify repayment – which are conservatively projected to cost at least $6 billion, according to the FHFA.

Currently, the FHFA anticipates $4 billion in loan losses due to projected forbearance defaults, $1 billion in foreclosure moratorium losses and $1 billion in servicer compensation and other forbearance expenses.