As an ex-Xero employee, I just had to blink twice seeing this trending on Hacker News. For context, Xero is primarily huge in New Zealand and Australia with a large UK chunk but not much presence in the US. While I shouldn't say too much, I'm sure the incident history speaks for itself that this isn't unheard of :) https://status.xero.com/history Arguably underreported I might think considering the userbase however
Admittedly, this incident is probably a pretty big one since it was acknowledged on their Twitter account. They'll mention it in response to customers but you only see an actual broadcast to followers when something is really on fire. I hope the responders at least get a bit of sleep (it's currently 11pm here in NZ and the HQ is in Wellington, New Zealand)
Ahh dang, I wrote a whole post and then I accidentally closed the tab :)
As background, Xero is in the same boat as a whole host of different businesses. It's nothing unique to them at all. They started with a single C# monolith and that scaled until it started running into trouble. At this point, you start splitting out your monolith into different services and Xero is quite a ways down that track.
At present, there's still a decent chunk of code that still lives in the original monolithic codebase. The initial home screen (dashboards etc) live in there among other things as an example.
When you navigate between services however (ie; invoicing or payroll), those are actually entirely different sites run by other teams that you're switching between, even though the UI is the same (the navigation bar is itself a separate service too for example which is why sometimes you get incidents where the nav bar disappears, if that service runs into trouble)
There is an in-house CSS library that is standard to keep everything consistent but there may still be older versions of it between services or even different JS libraries depending on how old or new a given service is.
So I guess to answer your question, the answer is yes, there are multiple templating libraries and things like that under the hood.
Beyond that, I probably shouldn't go into too much detail! Most of the above I think you can infer just by navigating around the application and seeing how various cookies, stylesheets and what not change so there's nothing particular unique.
Interesting that their competitor Sage is having almost exactly the same outage, which started at roughly the same time: https://status.sage.com/
Although, I prefer Xero's status update to Sage's at this point: I don't think it's appropriate to claim all your other services are "operational" if you can't log in to any of them!
In Sage's case, they're having an Identity outage and Xero have their own identity setup internally (ie; not dependent on any external services) so it'll be purely coincidental.
3 hours down so far and no notification on the login page, no explanation in the status to say they have diagnosed the issue and no eta for a fix is very very poor for a service of this type. Particularly annoying as I have an invoice to create.
We use Xero for all of our accounting but what has always shocked/surprised me is that their backup/export facilities are so poor. We have all the source data so it's not a disaster if Xero loses data, but we would pay double or triple to have a monthly data dump a la Slack.
One of the first global accounting saas providers. Forced Intuit to rapidly improve their cloud services but have failed to gain significant market share in the US.
Xero doesn’t let you export or backup your data. One Kiwi company does auto Xero backup (Control-C) but their interface is super clunky - I know they are trying to fix their stuff though but the backup service provided is for use in emergency only.
Each time I see an outage at Xero, and they are getting more and more lately, my heart stops.
I fear one day I’ll wake up and my business won’t work.
Still - for SME’s, the alternatives are sparse.
I wish they offered offline and backup.
I wish they had a competitor with similar pricing and features that offered backup.
What’s a backup going to do? It’s not as if a database backup would enable business to continue the next day. For a data model the size of Xero, the logic of the application is inherently tied to the data.
In the event of catastrophic data loss at Xero, you could at least take your debits and credits, transactions, invoices and other basic accounting data with you to another vendor. Sure, Xero adds plenty of application logic, but at the end of the day, it's just double-entry accounting.
I was an early adopter in the US. I've been relying on Xero and it's APIs for my business to operate since 2009! This is the first time I can ever remember an outage like this. There are definitely some things that annoy me about the product, but I understand that when you deal with a companies general ledger, you can't afford to "Move Fast and Break Things." I do not envy the sysadmins getting woken from their sleep right now.
