[There is another pause, this time there may be a sense that he is surprised by that particular retort before he makes a small noise of annoyance.]

And you have other far more pressing matters to attend to, do you? If so then it seems incredible that you had the time to spare in telling people that you don't have time to be bothered by them.