Well, if you take M as a title (which Goldeneye seems to indicate it is, since M talks about taking over from her male predecessor), then Robert Brown isn't really that strange; Assume Bernard Lee's M retires. Brown played Admiral Hargreaves in The Spy Who Loved Me, then shows up as M in Octopussy. No reason he couldn't have gotten a promotion/reassignment between those two films. In fact, it's almost continuity.
The oddest continuity issue for me (pre-reboot, that is) was Maud Adams playing both Andrea Anders in The Man With the Golden Gun and then Octopussy. There's no way you can explain it; Scaramanga kills Anders. There's no way around it. She can't be the same person, and if she was, wouldn't Moore have recognized her? Hell, he slept with both of them.
Of course, the biggest continuity problem of all is that you have a field agent that's been on active duty...for 40 years (from Dr. No thru Die Another Day).
Some people have theorized that Dench's M is the same person, and that it gives credence to the idea that Bond is a code name taken up by different agents. But there's so much else, even in recent movies, that prove otherwise; For example, what are the odds that Dench would have both an assistant AND an agent named Moneypenny? Or TWO Chief of Staffs both named Tanner? Of course, there's older examples too, like Moore's For Your Eyes Only pre-credits sequence where he mourns at Tracey's grave, then kills Blofeld, or Diamonds Are Forever, for that matter, where Connery is seeking revenge against Blofeld for killing Lazenby's Tracey.
