Sony makes a walkman branded MP3 player. I’m sure there are other similar options on the market, but this segment is likely not that competitive anymore thanks to phones. Phones are kind of the ultimate here - nice screen, third party apps for downloading/streaming content, tons of connectivity, storage, etc. Standalone devices are going to compromise on some/all of those things, unless you pay a premium.

If you’re simply trying to avoid Bluetooth, you could:

• Try your luck at a lightning and/or USB type-C adapter
• Buy a lightning/USB DAC. You might be able to find one that can drive speakers directly, but the easy route here is probably buying two units: DAC and an amplifier if you don’t have on already
• Buy one of the few remaining alternatives
• Track down an older used phone that still has a headphone jack. This should be fairly cheap and if you get a formerly popular phone, like say a Pixel a-series, you can still buy parts. I very recently bought a battery for my pixel 3a

Fiio has a few well regarded contenders but I am unsure if they have batteries that are replaceable.

You could use a Raspberry Pi or similar. I don’t think there’s much of a market for music players any more.