For a lot of people I think it takes a while to really internalize the ramification of the fact that you can have a pointer to memory you're not allowed to use. (That, plus the way they're used in C as a cheap answer to templates/generics.) I understood that a pointer was an address from the start, but it was still some time before I figured out how to use them correctly.
You've never arrived at a physical address to find a shuttered business or gaping crater? I can understand the syntax of pointers being confusing, but the concept has distinct real world analogies. (That being said, I find the syntax of physical addresses in other countries to sometimes be confusing.)
Certainly I have (the former, not the latter). You won't be surprised when I tell you that it wasn't a horrible tragedy and that I didn't die on the spot because of this, or accidentally purchase a handgun instead of a watermelon. In fact, it was such a minor event that I hadn't taken the time to make plans ahead of time for such an eventuality. Likewise, the first time I used pointers, nobody told me ahead of time that it was even possible to have a pointer that pointed to nothing. Understanding this, and (more crucially) learning exactly how to know when a pointer was and was not valid, was what got me my first merit badge in C. Everything before that was just Pascal with less formality.
C implements a fairly low-level but general model of how a computer works, and a programmer's understanding of pointers is something of a proxy for their understanding of how that model works. Therefore, C has a sort of built-in test for a level of competence in a programmer. Unfortunately, it is by no means perfect, and you can still find people writing in C who have no idea why they should not return a pointer to an automatic variable.
Sure... but I don't really see that that's a problem with the analogy. I'm limited in what I can do to someone else's house, even if I have the address; an address might persist after the house has been bulldozed and replaced; &c...
It's not really a problem with the analogy; I'm just saying that a good analogy is no cure for a lack of experience. My first non-toy C program was a disaster because I knew what a pointer was but I didn't yet understand how to manage them.
Oh, for sure it's no cure for a lack of experience. My contention was solely with the statement that there is no real-world analogy. There is one; it's useful; it doesn't accomplish everything because it's just an analogy.
