How is the working-age population affected by the number of retirees?

I read a lot on the issue of aging population in some developed countries and how things are worse compared to a few decades ago. At the same time, if you look at most of those countries (Japan, Italy) GDP per capita has actually plateaued over the last decades rather than decreased.
I understand there’s an increased tax burden with aging population, but if GDP/capita stays the same, then GDP/worker increases, as there’s fewer workers. So even with the increased taxation, workers should not be worse off as long as GDP/capita stays the same. Am I missing something?