Plan to aggressively pay off our home, or should we bite the bullet and invest?

My husband and I currently make about $285k from our salaries. We also rent out our first home that we got before Covid so it’s got about $200k in equity in it with a $160k mortgage and a 3.125% interest rate. It nets about $850 per month after mortgage, taxes, and insurance. We consider it our fall back plan if everything blows up. Our car loan gets paid off in a few months, which frees up about $850 per month. No credit card debt.

Our current home has a mortgage that is two years in with a balance of roughly $380k and a 6.875% interest rate. We pay $1k extra per month now and plan to up that to $1850 once we pay off the car loan, since we’re used to that money coming out anyway.

We max out our retirement accounts and have about 6-8 months worth of expenses in a HYSA that is 3.8% I think.

We’re strongly considering just brute forcing down the mortgage of the home live in now, with the goal of having it paid off in less than 5 years. We’re extremely risk adverse because we’ve watched family mishandle money and lose it all, but we’re always going to need somewhere to live and paying so much per month in interest is killer.

Our plan is to pay the $1850 per month extra, then add in a larger payment once or twice a year based on our savings and what we feel comfortable with. I’ll have to double check our terms for recasting, but we’d also like to recast once we make a bigger dent then continue to pay the same amount anyway. I’d like to ideally put at least $60k per year towards the house, which would allow us to also save money while aggressively making payments.. then basically just pay it off in full one day once we get close. Then after that, we can breathe easily and figure out the next step to do with the cash flow/savings. Once the mortgage is paid off, that frees up the $2500 in interest we pay per month plus all the extra payments we’d hypothetically been making.

The idea of having no mortgage on our house is amazing. My husband works in tech and he does worry about job security one day due to AI, so he really likes the idea of us basically being able to sustain our lives on any jobs because we won’t be carrying an expensive mortgage. We’re fortunate that we live in a pretty low cost of living area and have pretty modest tastes (other than the car that we’re about to pay off.. he wanted his one nice car splurge, haha.)

But how dumb is this plan? Does it make sense? Or should we consult a financial advisor and try to figure out the whole investment accounts stuff now instead of going with what we feel is the safer bet? Both of us grew up in households that struggled on and off so we don’t really have anyone close to advise us.. and the fear of the rug being pulled out is always there.