They are a very useful identifier. However they only prove that someone with such a name exists. They are not enough to prove that the given person with that number is the unique person they say they are.
I agree, but what makes it hard shouldn't be having access to that number, it is verifying the number belongs to the person you are trying to impersonate. There is great need for an impersonal unique id for everyone - there are doesn't of people in the world who share the same name as me. It is possible that some of those people share the same birthday as me as well. For most purposes I want to go by my name, but when you need to be sure you need a unique ID.
They were never meant to be a ubiquitous identifier/authentication token.
But they de facto are and have been for longer than they haven’t. At some point, it becomes an abdication of responsibility by the SSA, no matter how much they kvetch about it being “not their problem”.