I don’t know much about the US legal system, but I don’t understand how this affair has any bearing on the legitimacy of the case being brought against Trump?

It doesn’t. From Trump’s point of view, this was the equivalent of Trump standing at Mar-a-Lago, trying to score a hole in one on a golf course in New York by hitting the ball by way of Tokyo and just hoping the wind shifts direction, and actually hitting it. Her affair itself has absolutely zero bearing on the Trump case, but the facts that have been unearthed surrounding the affair have brought her general ethics and motives into question. This is typically the death knell of a lawyer’s career.

I presume Fani Willis could be removed, but the case against Trump would still continue under a new DA?

It assumes a number of things:

• GA is traditionally a red state. It is very possible that a new administration may choose (or be told to) to simply let the case die.
• A new team would need time to be brought up to speed and prep for the case. This alone would easily push the case well into 2025 or even 2026, which basically gives Trump the victory he’s been seeking in the first place.
• The ethics questions of why the case was brought in the first place would linger anyway, which could complicate the case going forward, and which Trump and his team would mercilessly exploit in order to sow doubt, create endless delays, and/or even hope for jury nullificaiton, lowering the chances of a successful conviction.