Signal is a centralized app, run by a company. If they are offered enough money or legal threat they will sell out or close.
I am sure people will make an argument that its FOSS and people will just fork it if it goes bad, but a new fork will have 0 users and Signal will still have all of your old contacts. Why not make a switch now? Before it is even more popular and you have more reasons to stay? Why fork it if there are already decentralized apps that use same encryption, like XMPP apps?
Sure you can find flaws in every app, including XMPP implementations, but if we will have to write code for a new Signal fork, why not just fix whatever is that bugs you in XMPP clients?
If you want to use Matrix, that is fine as well, we can always bridge the two open protocols. But you cant bridge Signal if their company doesn’t allow it.
You can pick servers run by groups that have just as good record of privacy or even better or are run by the person you know or yourself.
When you have a decentralized service you can choose who you trust, you are not stuck with one corporation. Picking a completely random server is the worst possible example you could have chosen.