I saw many contractors that lost their (mortgaged) homes in the oBana economic downturn. Not just a few divorces, as well.
Fees and CG taxes tend to wipe out a lot of what you think you have, especially selling in a down market. Add in mortgage interest you are paying so you can "use" your money to invest, and the returns start looking smaller and smaller.
Most of them lost their homes because the banks were making loans that defied all traditional lending practices. I did a multi-year deep dive on this and I still don't understand how no one went to jail, except for the fact that the "leaders" of both parties are beholden to the big banks.
People who had traditional loans based an standard banking practices came through o.k. People who had interest only loans, 0 down loans, and so on, who were betting on the continued appreciation of real estate got hammered. I have seen loan packages that defy any reasonable belief, and these were packages where the loans were made.
You don't buy a home as an "investment". You buy it because you have to have shelter, and long term, your overall financial situation will be better if you recoup some or most of that cost through a home purchase rather than paying rent.
In fact one of the things I learned is that the banks count on a large percentage of home loans being paid off through sales in 5-7 years --effectively increasing their yield because of all the fees and points that are front loaded.