I don't want my mother's finacé at our Christmas celebration!

I apologise in advance if this is long, my thoughts are a bit jumbled so I'll try and keep this concise.

My mum's finacé, I'll call him Jim, is a horrible person and I cannot stand him.

A bit of back story:

I don't say that lightly, I don't dislike anyone generally speaking and I do always try and see the best in people - Jim just doesn't seem to have any redeeming qualities at all.

My parents have been separated since I was really young and there's been many men throughout the years, all of whom I've been able to have at least polite interactions with but most of them I got along well with. My dad is re-married and I adore my step mother, so it's not like I'm one of those people who just dislike their parents new partners and create drama for the sake of it.

Jim has always been unpleasant in his demeanor and has had an attitude with me (and my other family members) from the minute we met and i assure you we have been nothing but polite and welcoming to him.

He's highly confrontational, abrupt, he says things that are wildly inappropriate, rude and offensive. He constantly whinges and obsessively rants about things that would not really bother the average person. He's very aggressive in the way he communicates with everyone and just generally makes everyone feel extremely uncomfortable

He controls every aspect of my mum's life.
I know he is financially, emotionally and verbally abusive towards her. I don't think he's crossed the line of physical abuse but I really wouldn't put it past him.

Obviously this is just a snippet of all of this but it's been a long 3 years with him in the picture.

My mum and I aren't close (there's some trauma there which is all a story for another time). We usually only see each other at family events or special occasions where we're cordial with each other.

I usually do a Christmas get together at my house where all my extended relatives get together and exchange gifts etc.

Now, the other day the straw broke the camels back.
I saw Jim at our local shopping centre, we were coming towards each other, maybe 3 or 4 metres apart, made eye contact and everything. I didn't really want to speak to him either but I knew I couldn't get out of it so I smiled at him and intended to stop to say hi.
He gave me the filthiest glare (if looks could kill I'd be dead) and turned around and walked off in the opposite direction he was initially going. I was so stunned I actually laughed out loud, a lady walking past who I don't know from Adam even looked at me like "what was that about?"

Now I'm at the point where I'm like fuck him, I do not want him in my house, I don't want him here at Christmas putting a dampener on everyone's spirits like he has the last few years (my kids don't want him here either and nor would my family be remotely bothered by his absence). Why should I have to accomodate someone who is so blatantly rude to me?

But then I'm conflicted because I don't want to isolate my mum at Christmas because she either won't come without him or he won't allow her to come alone, even though we have our issues that makes me feel guilty.

What would you guys do?