Unfortunately for Richard he was never able to simply concentrate of governing the realm; the hangover of his accession, the presence of Henry Tudor abroad - these things constantly took his attention away.
The death of Queen Anne and rumors of a mesalliance
Richard was plagued by bad luck. In April 1484, Anne and Richard's only son, Edward of Middleham, died. His tomb is at Sheriff Hutton, in the heartland of Neville country.
‘You might have seen the father and mother…almost out of their minds for a long time when faced with such sudden grief’
Anne clearly was not going to have another child. Richard had two bastards for sure, John and Katharine, but there was no talk of any attempt to legitimise them. Nope, the whispering was instead about the inconvenience of Anne's existence. But in fact Anne was ill; although she did her best to play her part at the Epiphany celebrations of 1484, on March 16th she was dead.
And then the tongues really began to wag. Had she been poisoned by Richard? The Tudor historians would have us believe he had; but even the Crowland Chronicle seemed to suggest such a thing:
"In the course of a few days after this, the Queen fell extremely sick, and her illness was supposed to have increased still more and more, because the king entirely shunned her bed, declaring that it was by the advice of his physicians that he did so. Why enlarge?"
And the tongues continued to wag - that Richard would marry his niece, Elizabeth of York, daughter of Edward IV and Queen Elizabeth. After all, it would unite the Yorkist lines, bolster Richard's legitimacy, give the Henry Tudor promise top marry her a real kick. But it's difficult to believe Richard ever genuinely contemplated such a thing - marrying his niece would surely have removed any remaining shred of credibility. But incredibly, the rumors were so strong, Richard was forced to publically deny it:
"In the presence of many of his lords and much other people showed his grief and displeasure and said it never came in his thought or mind to marry in such a manner wise."
Really, it seemed the only thing that could wipe the account clean would be to meet and defeat his rival Henry Tudor in battle.