Yes. I've found that maintaining synapse is a nightmare. Overall I like matrix, but I have rage quit running it multiple times over the years.
I have a 256mb vps. It is running inspircd, atheme, znc, bitlbee, ejabberd, mumble, biboumi, and xmpp.js. Plus some non-communication stuff. I ssh in maybe once a year? I have to actively remember to go check on it because it never gives me problems.
I have a 2gb vps that runs synapse. There is always something wrong. Using all of my cpu. Using all of my ram. Using a ton of network. Generally slowing down everything else on the vps. Sometimes synapse crashes or doesn't come back up after an update or reboot. I gave up trying to dig through the gigabytes of log files that are produced. Why am I out of disk space? There is a 25gb table in postgresql for synapse that is... I dunno typing events or something. Don't worry though there is an api endpoint that might fix it. Good luck. I have 5 users.
A year (maybe 2) ago I gave up on federation and I'm just running synapse with sqlite. There are a lot less but not zero problems.
I've actually considered the idea of trying to set up bifrost to log into my xmpp server to use biboumi to connect to bitlbee running purple-matrix so I can join the matrix and synapse rooms on the matrix.org server using my matrix.org account. That fact that I've even thought that is insane. As crazy as it sounds I really do (sorta) actually believe that it might be more stable for me.
My personal server sits at around 1.8GB RSZ with postgres, which is about right for a poweruser personal server. Unfortunately a 2GB VPS is still unlikely to hack it if you're federating. Meanwhile Dendrite uses ~10x less RAM, but is still beta.
I love Matrix and use it every day on a self-hosted server. But it took me at least 3-4 attempts before I got the setup working in Docker (at the time, the Docker documentation was substantially behind).
Once I got it working, some aspects are still pretty janky/I don't understand them. (Generating a homeserver.yaml file was a nightmare, hosting on one domain but addressing at another (chat.foo.bar but with COGlory@foo.bar as my user).
Getting federation working was another nightmare. Hours upon hours of trying to debug why it wasn't working, what ports weren't communicating properly, etc.
Trying to get jitsi working and/or a TURN server has also been impossible to the point where I've given up.
In fairness, I'm not a super tech savvy user - I've taken one programming class in my life, and at the time, was hosting everything off of an MSI Cubi.
And no, the default install is never that hard, not even on FreeBSD. But then, on it's own, Matrix is just yet another chat system, which I really don't need - what I would need are the bridges. And setting up the bridges are horrible.
The need to setup bridges negates the privacy benefits, it breaks the opportunistic E2EE when copies of all messages are sent to e.g. Telegram servers that stores them effectively in plaintext. When the main goal is to invite people to some platform, bridges are a shortcut to allow some people to join, but until everyone joins and the bridge can be removed, there's no improvement.
Interestingly, people don't really consider the privacy aspect. Average users claim they don't want another messaging app, but somehow they find the willpower to jump from facebook to instagram to snapchat to tiktok to wine to telegram to periscope to kik.
My peers use have an inside joke of "stickers > human rights" which describes what motivates people. Matrix or be it any other client, needs an exclusive throw-in feature that appeals to masses, basically something that combines sharing everything like they used to on social media, something that feeds their narcissism (likes etc.), and something that's fun (think tiktok videos). Signal doesn't have it, at least yet. Matrix most certainly doesn't have it. Telegram has some of it.
The usability needs to be intuitive and thought thoroughly, and it needs to make managing (social) life easier. We've already seen this with personal could in e.g. Signal's "note to self", or Telegram's polls -feature (I'm hesitant to consider it a feature since it's inherently non-private method to share opinions to Telegram developers about something: group chats are never E2EE).
My perception of Matrix is that it fails to deliver the stuff listed above, as well as life management, ease of use, moving responsibility of hosting to some trusted peer, and even security agility and default security feel off. The only thing that makes Matrix stand out is the ideological "decentralization means no individual has control over communication" which doesn't resonate with privacy-minded users who care about E2EE-by-default-for-everything, nor to average users who don't hesitate for a second to jump from TikTok to FlobbordSocial that allows them to share videos of their Plumbus X if it gives them social acceptance and likes, i.e. dopamine spikes. These people consider apps fads that come and go, they don't really consider long-term availability because they're always looking for the new shiny thing to fill their internal void. They know they can always call/text/whatsapp their friends if Flobbord suddenly seizes to exist.