How hard will my credit score be hit if I close my first card?

I'm thinking of closing my first credit card. I got it in 2013 from a small credit union, then got my second in 2015 from Chase with a much larger line of credit. Those are my only cards. I have an excellent FICO score. Recently the credit union merged with another credit union and the customer service has taken a huge nose dive. My card was stolen and it took nine phonecalls, three emails, a visit in branch and a month and a half for them to take action. They turned off auto pay without telling me and I received a letter in the mail that I was in default. I panicked, called and was told that I wasn't going to face any penalty for being in default, but that I can't set up auto pay because I don't spend the exact same amount of money every month on my card and there's no option to pay off the whole balance. They've been pressuring me to close my high yield savings account for a worse option and answer questions about my IRA with "we don't know how those work, you have to find an accountant". Not to mention they somehow deposited someone else's money in my account and all of the lovely employees I've been working with for years are miserable and quitting. I'd like to close all of my accounts, including the credit card, but I can't get a straight answer from bankers about how this will actually impact my credit score. They just say "it could be bad" but not how bad. I really don't want to deal with this organization anymore, I don't trust them with my money. Is it really going to mess up my credit score that badly? I have no plans to make any major purchases that require a loan anytime soon.