Interestingly, there seems to be way more stories about "the other woman" than "the other man" but Guy de Maupassant's "A New Year's Eve Gift" takes it from the latter's perspective.
The story opens with a man about to sit down, on New Year's Day, to write New Year's greetings to his friends. A charming scene. However, when he begins to write a letter to a woman named Irene, it is clear she weighs more heavily on his mind. Suddenly his concentration is broken by a knock on his door. It is Irene herself and she is clearly quite shaken.
I won't go further, and I especially won't state the ending, though I have to say I found the whole thing quite despicable. It's a well-written story and interesting, I suppose, but it left me feeling icky.
(Did you write a post for Short Story Monday? If so, please leave a link in the comments below.)