He seemed to be directly addressing Mycroft. which would make sense.
But that's a different scene, to this one, where Mycroft mentions the other brother.

So.   A lot of you think that CAM  is a  Holmes Brother.   Mumsie and I  were discussing that very thing last night after we'd seen HLV. (Yes I got her hooked on SHERLOCK!!)   that was an OMG   moment.   at first,  I thot that Mycroft was talking abt  Rudy,  the uncle that he mentioned when Sherlock and John arrived in Baker street after John rescued Issac  Whitney frm the drugs den.   Ok.   No.   Yes,  how DID  Sherlock know about Mind Palaces?   That  is the main connection between him and CAM.   So  it's logical to assume that CAM  is the other Holmes brother.   But Sherlock doesn't know.   I  don't think Lady Smallwood knows either.   Only Mycroft  (and possibly  his mum OR dad) knows.   I  cannot wait to see if we are correct in this theory.   And we bloody well better not wait too long.   

I don't think he talked about a brother at all, he was probably talking about his predecessor or another agent, who couldn't get this job right because of personal feelings. Maybe the agent Anthea (Andrea) mentioned in Empty Hearse, the one who lost his life.

I've been thinking that Mycroft's remark may NOT  have referred to another Holmes brother too.   There's so many possibilities  for mikey's remark.   

I was thinking about this the other day. I don't think Moriarty or Magnussen is the other one. At least, I hope not! XD

If there is another brother, is he dead, which seems likely based on the way Mycroft seems to state it, or maybe the brother was disowned by Mycroft and Sherlock? That seems harsh, but makes sense if the brother wasn't as smart of these two were. And by what Series 3 implied by Mycroft and Sherlock's relaionship, it may seem maybe Mycroft tried to teach this brother, but deemed him too ordinary and stupid? Since Sherlock views Mycroft in some ways as a mentor, followed his example of looking at such ordinary people with contempt. Maybe this other brother, who couldn't get the help of his parents, decided to leave and the brothers just ignore him or don't view him as a brother. 

That is heartbreaking now that I write it out. =(

I thought the same thing, also because they said that it was quite an important line. I'm unconvinced that it was Magnussen; they seem to be implying that something happened to the brother because of Mycroft's doing, not because of Sherlock's doing. 

It's interesting, for the Moriarty thing I came up with a handful of theories, some more plausible than others but for this I really have no clue. I didn't understand the original invention of the third brother either, I always assumed that Mycroft inherited the estate. Who knows, maybe at time of writing, Moftiss didn't have a clue either as to where it would lead and they are just making it up as they go along. 

Thinbeard wrote:

As has been said elsewhere, the third sibling could be a sister. Nothing in Conan-Doyles work excludes it. Listen carefully to Mrs. Holmes speaking to Mary about Mr. Holmes. Very carefully.

Do you mean the scene where Mr. Holmes says to Mary that Mrs. Holmes "gave it all up for children"?
 

I think discovering a Holmes sister would be fantastic.  

Oh, yes. And IMO it would be a typical Moftisson stunt. Taking a character from Canon and turning it into something much bigger. Like they did with Moriarty and Mary on a larger scale. Very exciting!

Maybe this was obvious to everyone, but I didn't really notice it until I saw this gif.

Mycroft's reference to the "other one" is visually reinforced by his reflection appearing in the window. Clever camera work as usual