I agree. I think Moriarty has had too much time in the series already, even though he's a great character.  It was great to have Magnussen as the arch-villain for a change (I wish he hadn't been killed!  I'm sure he had another Sherlock story in him!).  I'd like to see somebody different for S4, whether it's Mary or somebody new. 

There was a remark in the S3 commentary about bringing Yasmine Akram back because she was so good in TSOT.   Now, I never quite understood that, because presumably HLV was already written and they were already going to use the courting the secretary plot, and you would think they already knew who they'd cast and had been planning on using "the bridesmaid".  But maybe not.  Anyway, it seems to suggest that there wasn't an initial plan to use this particular actress (with her Irish accent) so heavily, which also suggests that she wasn't supposed to have a Moriarty connection at that point.   But they could write one in after the fact, particularly if they wanted to bring Yasmine back.