People buying houses with 3% down payments are generally paying such high interest rates that they don't get to enjoy the home price appreciation because they're paying it all in interest to the bank. Meanwhile they're taking the risk that the house doesn't appreciate faster than inflation, or at all, as was the case for people who bought homes in cities like Detroit. And the bank will want enough interest to cover the risk that the home value declines and they default on the loan, which means many such people are paying more to own than they would to rent, even after including the accumulation of equity.

Homes appreciating faster than inflation is also an unsustainable trend in general. The result has been for housing costs in those areas to increase as a percent of wages, which obviously can't continue indefinitely because the result would be housing costs that don't leave enough for other necessities like food, or that exceed wages outright.

It's true prices probably wouldn't fall to only 3% of what they are now and so the same people couldn't purchase the same house immediately, but rents would fall along with housing costs. The combination of lower housing costs and less paid in interest on huge high-risk loans would allow the same people to own the same house outright in less time, even if it meant renting it for some period of time first. And of course the money they intend to use to buy the house could in the be earning interest before they reach the threshold to buy the house without a loan, which (if the efficient market hypothesis is correct) would give the same risk-adjusted returns in the meantime as investing the same amount in home ownership.

Houses don't have to appreciate at higher than inflation in order for them to create wealth; they just have been in many areas due to the overall economic expansion. They can appreciate more slowly than inflation and the effect of leverage can still give them cash-on-cash returns higher than inflation or alternative investments.

On a conventional mortgage with 20% down, if the house appreciates at 1% per year in a 2% per year inflation environment, a $100K house goes up by $1K each year. Someone who bought that house with $20K down sees a $1K gain on their $20K cash investment, for a 5% cash on cash return. They also have a place to live typically substantially cheaper than they were paying in rent. Obviously, where they increase even faster than inflation, this is wildly beneficial and if they decline much at all, it's terrible.

3% down mortgages seem to cost around 1.25% more than 20% down mortgages. It's about 1/8-1/4% on the base interest rate and 0.5%-1% for PMI. With a base interest rate on a 30-fixed around 4%, paying 5.25% on a 3% down mortgage is still a good deal IMO.

If landlords had to pay cash for rental properties, I'm not convinced that you'd see such a surplus of rental properties such that it would drive rents down significantly. Rents are driven by ability and willingness to pay. Many small landlords would be forced out of the supplying housing to others work. If landlords could borrow money to buy houses but owner occupants couldn't, I think you'd see a massive defection of the housing economy in favor of landlords.

Obviously, anyone could borrow on unsecured terms. It seems likely that medium and large landlords could exploit that (borrowing against the projected cash flows, but without using real estate as collateral for the loans) and that would also result in a large shift of power away from owner-occupants and small landlords.