Bad idea to purchase new build and refinance right after closing?

Currently in Escrow for a new build home. Planning on getting a higher rate from the builder’s preferred lender for credits towards closing costs. After closing looking to refinance right away with a brokerage who can offer a really low APR. I shopped around and while the broker I was working with can offer a really great rate it won’t be able to match the builder’s credit. What I did was took their offer and had the preferred lender try to match it and best they did was give more credits towards closing by costs.

Tell me why this would be a bad idea other than hit towards our credit scores (we each are at the 790 range). Chase reps saying there’s no prepayment penalty or seasoning period. The mortgage broker friend of a friend and willing to get the lowest amount for us and will try to minimize his fees and the closing costs for us. I think out of pocket will be about $1,500 for refinancing.

State: California

Purchase price: ~$800k

Solars: 16k (must pay)

Down Payment: 20%

Preferred Lender: Chase

APR: 6.125 (30 year)

Credit from builder: 22k

Credit from Chase: 7.2k

Mortgage broker APR: 4.99 (7/6 Arm)