There is no center to the source of the background radiation, as it is was created at a moment the universe was still a fairly homogeneous soup (apparently) with these photons being created (and absorbed) everywhere roughly in equal quantity.
Only after the universe became transparent did these photons become free, and since they were produced roughly homogenously in that soup there are everywhere in the same quanitity: a background radiation.
Since they were not produced in a localized event, they won't have a center. But since they're everywhere, they are a sort of absolute reference for how fast you're going (actually that's a new and interesting idea for me too, pretty neat).
But if they're a reference for how fast you're going, aren't they also a reference for what direction you're going? Can you then not look backwards at the direction they have come from and work out the point that they all appear to originate from? And then haven't you found the centre?
Imagine an omnipresent and homogeneous field, let's say an infinite field of corn. By looking at the corn, you can see how fast you're going, without that observation requiring the field to have a center.
They are coming at you from every direction at once.
What you’re asking is analogous to asking where the center of the Earth is on a map of the surface, and arguing it must exist because of how things on the horizon change when you move.
The reference velocity is the one such that you see an equal amount of cosmic background radiation in every direction. If you're moving relative to it, you'll see extra radiation from in front of you (much like how you see extra cars moving in the opposite direction from you on a highway).
Only after the universe became transparent did these photons become free, and since they were produced roughly homogenously in that soup there are everywhere in the same quanitity: a background radiation.
Since they were not produced in a localized event, they won't have a center. But since they're everywhere, they are a sort of absolute reference for how fast you're going (actually that's a new and interesting idea for me too, pretty neat).