I understand small business owners do this, but most people would not. Nor should they. Yes the market on average returns much more than 3.5%, but it's not guaranteed. The 3.5% is guaranteed. Most people wouldn't want to put their house on the line for the chance at higher returns in the market.
I'm a bit hypocritical here because I invest while having a mortgage, but I am on a 15 year mortgage. I saw way too many people who took on 30 year mortgages for all the house they could afford, and moved every 5 to 10 years, whose houses never appreciated to the point they had substantial equity, and now they're 60 with a mortgage. Plus, there's the whole risk of spending that extra money rather than investing it, which sinks most people who try this plan.