There's two technical problems that microservices purport to solve: modularization (separation of concerns, hiding implementation, document interface and all that good stuff) and scalability (being able to increase the amount of compute, memory and IO to the specific modules that need it).

The first problem, modules, can be solved at the language level. Modules can do that job, and that's the point of this blog post.

The second problem, scalability, is harder to solve at the language level in most languages outside those designed to be run in a distributed environment. But most people need it a lot less than they think. Normally the database is your bottleneck and if you keep your application server stateless, you can just run lots of them; the database can eventually be a bottleneck, but you can scale up databases a lot.

The real reason that microservices may make sense is because they keep people honest around module boundaries. They make it much harder to retain access to persistent in-memory state, harder to navigate object graphs to take dependencies on things they shouldn't, harder to create PRs with complex changes on either side of a module boundary without a conversation about designing for change and future proofing. Code ownership by teams is something you need as an organization scales, if only to reduce the amount of context switching that developers need to do if treated as fully fungible; owning a service is more defensible than owning a module, since the team will own release schedules and quality gating.

I'm not so positive on every microservice maintaining its own copy of state, potentially with its own separate data store. I think that usually adds more ongoing complexity in synchronization than it saves by isolating schemas. A better rule is for one service to own writes for a table, and other services can only read that table, and maybe even then not all columns or all non-owned tables. Problems with state synchronization are one of the most common failure modes in distributed applications, where queues get backed up, retries of "bad" events cause blockages and so on.

Running code in a single process is MUCH lower overhead because you don't need to transit a network layer and you're generally just passing pointers to data around rather than serializing/deserializing it.

There are definitely some cases where using microservices does make things more CPU/memory efficient, but it's much rarer than people think. An example where you'd actually get efficiency would be something like a geofence service (imagine Uber, Doordash, etc.) where the geofence definitions are probably large and have to be stored in memory. Depending on how often geofence queries happen, it might be more efficient to have a small number of geofence service instances with the geofence definitions loaded in memory rather than having this logic as a module that many workers need to have loaded. But again, cases like this are much less common than the cases where services just lead to massive bloat.

I was working at Uber when they started transitioning from monolith to microservices, and pretty much universally splitting logic into microservices required provisioning many more servers and were disastrous for end-to-end latency times.

All of the catalog data was moved to a service which only served catalog data so its cache was optimized for catalog data and the load balancers in front of it could optimize across that cache with consistent hashing.

This was different from the front end web tier which used consistent hashing to pin customers to individual webservers.

For stuff like order history or customer data, those services sat in front of their respective databases and provided consistent hashing along with availability (providing a consistent write-through cache in front of the SQL databases used at the time).

I wouldn't call those areas where it makes things more efficient rare, but actually common, and it comes from letting your data dictate your microservices rather than letting your organiation dictate your microservices (although if teams own data, then they should line up).

I've seen many systems where there are a few hundred to a few thousand users, not hundred of millions or billions of users. There can also be teams of 5-10 developers who manages +20 microservices. I still don't think those projects have the same needs and could have done something else.

I'll agree that a team of developers with 20 microservices sounds obviously wrong (which is really my point that the site should be divided around data and the orgs should probably reflect the needs of the data--not the other way around). And Amazon back then didn't have that problem.

But I don't have to now.

Cache locality starts making a difference when you have over ten servers, and a significant fraction of the data is cacheable.

For "transactional" Enterprise CRUD apps, often there is very little that can be cached due to consistency concerns.

Servers can only get so big. If your monolith needs more resources than a single server can provide, then you can chop it up into microservices and each microservice can get its own beefy server. Then you can put a load balancer in front of a microservice and run it on N beefy servers.

But this only matters at Facebook scale. I think most devs would be shocked at how much a single beefy server running efficient code can do.

When you have a giant monolith with the "load the world" endpoint, it can be tricky to pinpoint the the "load the world" endpoint (or, as is often the case, endpoint*s*) is what's causing issues. Instead, everyone just tends to think of it as "the x app having problems."

When you bust the monolith into the x/y/z, and x and z got the "load the world" endpoints, that starts the fires of "x is constantly killing things and it's only doing this one thing. How do we do that better?"

That allows you to better prioritize fixing those scaling problems.

There is a point when it all starts to make sense. But that point is when you go into billions worth business, hundreds of devs etc. And going there has large cost, especially for small/medium systems. And that cost is not one off - it's a day-to-day cost of introducing changes. It's orders of magnitude chaper and faster (head cound wise) to do changes in ie. single versioned monorepo where everything is deployed at once, as single working, tested, migrated version than doing progressive releases for each piece keeping it all backward compatible at micro level. Again - it does make sense at scale (hundreds of devs kind of scale), but saying your 5 devs team moves faster because they can work on 120 micoservices independently is complete nonsense.

In other words micoservices make sense when you don't really have other options, you have to do it, it's not good start-with default at all; and frankly Sam Newman says it in "Building Microservices" and so do people who know what they're talking about. For some reason juniors want to start there and look at anything non-microservice as legacy.

It sort of is.

It's not a perfect world. One issue with monoliths is that, organizations like to take a "if it ain't broke, don't fix it" attitude towards things. Unfortunately, that leads to spotty service and sometimes expensive deployments. Those aren't always seen as being "broken" but just temporary problems that if you pull enough all nighters you can get through regularly.

It takes a skilled dev to be able to really sell a business on improving monoliths with rework/rewrites of old apis. Even if it saves money, time, all nighters. It's simply hard for a manager to see those improvements as being worth it over the status quo. Especially if the fact of running the monolith on big iron masks the resource usage/outages caused by those APIs.

That often is, because new solitions tend to come with their own problems.

I can see an advantage regarding resource hogging, but the flip side is the extra point of failure of network calls in microservices.

Not saying which is better, but deployment is orthogonal to logical dependence and correctness.

For instance, in a shopping site, why should a crash in the recommendations engine result in a non-functional webpage (rather than a working purchase page with no recommendations)?

Personally I think microservices start to make sense when you have several hundred developers (an environment I'm currently keen to never enter again - $work has 5 devs and might one day have 6).

I agree that microservices are suitable for large organization, where the organization practically has multiple products (which could be purchased from a vendor or sold to another company).

Networks could have unexpected delays, routing errors and other glitches. At least with a monolith you can often find a stacktrace for debugging. I have seen startups that have limited traceability and logging when using micro services.

When a small startup has to manage "scalable" K8s infrastructure in the cloud, distributed tracing and monitoring is often not prioritized when you are a team of 5 developers trying to find a product market fit.

I am not against microservices (I work with them daily) but you just trade one type of stability problem with another

I love working with microservices at the scale of $WORK, but we're Big Tech. I can't imagine why a 5 person startup would want k8s and microservices. You don't need that scale until you have more than 2 teams, and you're pushing at the very least 15 engineers at that point and usually the sales and marketing staff to make that investment worth it.

You're going to have to explain what you mean by that a bit more... You surely cant mean it as it is written.

If the recommendations team write code that causes the OOM-killer to end their process, making them run it on separate infrastructure insulates your "main store team" from the bugs they write.

I imagine his logic was something like: "How can OOMs happen less often if you run more processes (possibly on the same machine)?", while your comment actually wants to say: "if a specific service is affected by an OOM, with microservices only that specific microservice goes down, since it's probably running on its own hardware".

Split up services where it makes sense and don't over do it. That's how I design my projects.

So if a database service is not available you simply return stale, cached data until the service is back up.

It depends on what you are scaling. I think microservices are fundamentally more scalable for deployment, since changes can be rolled out only to the services that changed, rather than everywhere. Unless your language and runtime support hot-loading individual modules at runtime.

The solution to this is to use the right tool, a build system that supports monorepos like Bazel. Bazel solves this problem wonderfully. It only builds / tests / containerizes / deploys (rules_k8s, rules_docker) what needs to be rebuilt, retested, recontainerized, and redeployed. Builds are much faster, developers have God like visibility to all of an organizations' code, and can easily grep the entire code base and be assured their changes do not break other modules if Bazel test //... passes. It is language agnostic so you can implement your services in whatever language best suits it. It allows you to more easily manage transitive dependencies, manage versions globally across your org's codebase.

Of course Bazel has a steep learning curve so it will be years before it is adopted as widely as Maven, Gradle etc. But in the banks I've worked at it would've saved them tens of millions of dollars.

Also git would need to catch up to large codebases. I think Meta released a new source control tool recently that is similar to git but could handle large monorepos.

> I disagree, in my opinion micro-services hinder scalability of deployment

…and of anything related to testing:

- Want to fire up the application in your pipeline to run E2E tests? Congratulations, you must now spin up the entire landscape of microservices in k8s. First, however, you need to figure out which versions of all those microservices you want to test against in the first place, since every service is living in a separate repository and thus getting versioned separately.

- Want to provide test data to your application before running your tests? Well, you're looking at 100 stateful services – good luck with getting the state right everywhere.

We keep object definitions in a separate repo to not duplicate them and generate a maven dependence from that. Pretty simple actually. We mainly run on Java thought. Would be more complicated when there are multiple languages.

Protobuf or Json schema could come to the rescue if needed.

Admittedly, it isn't a scalability problem you will run into right away.

But when you need to roll out an emergency fix, there is a big difference between deploying to thousands of servers that all have everything, and ten servers running a single service.

The silly thing is that something like the JVM (which we use) really wants to host monoliths. It's really the most efficient way to use resources if you can swing it. The problem is when you give the JVM 512GB of ram, it hides the fact that you have a module needlessly loading up 64gb of ram for a few seconds. Something that only comes to a head when that module is ran concurrently.

200 threads, 12TB of RAM, with a pipe upwards of 200GB/s. This isn't even as big as you can go, this is a reasonable off the shelf item. If your service doesn't need more than this, maybe don't break it up. :)

I believe that this level of service can no longer accurately be described as "micro".

Let alone - AWS Lets you get that machine for less than a junior engineer salary ($7.5 per hour, roughly equivalent to $32 of hourly wage = $67k)

Ruby, Python, PHP and Node.js startups have to figure out how to use the cores while C++, Rust, Erlang and Go have no issues running a single process that maxes out all 64 CPU cores.

So it does not work for FAANG companies but what do I care. I have the rest of the world to play with ;)

Instead, if you build an architecture that can scale the number of corporate users by adding cheap $2k pizza box 1u servers as the company grows, that's much more attractive. Also, you can keep your systems design flexible enough to recompile and take advantage of advancements in hardware tech every 18 months – this gives you better operating margins as your own business starts to grow.

As long as hype chasers in middle management don't get in the way after convincing themselves they too must be like FAANG with a few orders of magnitude less of a consumer base.

They want to be both the architect and the manager and anything you say would be over ruled and since they are the boss its hard to ignore them.

This service is a monolith because it has 10K code and it needs to be broken up.The product is at MVP and its rock solid on Java Spring and it hardly crashes. We are never going to lose data based on the design choices we made. None of that matters.

We need zero down time upgrade, when we had zero customers.

I don't think you actually understand what microservices are. You don't put a load balancer to load balance between different services. A load balancer balances trafic between servers of the same service or monolith.

Microservices mean the servers of different services run different code. A load balancer only works together with servers running the same code.

Uh - what?

>A load balancer balances traffic between servers

Correct.

> of the same service or mononlith

Incorrect.

Load balancers used to work solely at Layer 4, in which case you’d be correct that any 80/443 traffic would be farmed across servers that would necessarily need to run the same code base.

But modern load balancers (NGINX et al) especially in a service mesh / Kubernetes context, balance load at endpoint level. A high traffic endpoint might have dozens of pods running on dozens of K8s hosts having traffic routes to them by their ingress controller. A low traffic endpoint’s pod might only be running on few or one (potentially different) hosts.

The load balancer internally makes these decisions.

I hope this clears up your misunderstanding.

The split is inessential. You can just as easily have homogeneous backends & one big load balancing pool. Instances within that pool can even have affinity for or ownership of particular records! The ability to load balance across nodes is not, as you claimed, a particular advantage of microservices.

Are you replying to the right comment? I made no such claim.

The point is, if a client makes a request to a server, the response should always be the same, no matter where the load balancer sends the request to. Which means it should run the same code.

Nginx doesn't even mention route based or endpoint based load balancing in their docs. Maybe they don't consider it load balancing either.

https://www.nginx.com/resources/glossary/load-balancing/

Describes exactly what I’m talking about.

Just because Nginx is doing it, doesn't mean it's load balancing. It's an extra function tacked onto a load balancer.

Network latency is real, and some systems really do run on a tight latency budget, but most sane architectures will just do a couple of calls to a database (same performance as monolith) and maybe a transitive service call or something.

Not at all. You can run your monolith on multiple nodes just as you can do with microservices. If anything, it scales much better as you reduce network interactions.

Little big concept or mono-micro people like the call it is where it is at. Spending too much time making a complex environment benefits noone. Spending too much time making a complex application benefits noone.

Breaking down monoliths into purposed tasks and then creating pod groups small containers is where it is at. Managing a fleet on kubernetes is easier than managing one in some configuration management stack. Be it puppet or salt. You have too many dependencies. Only your core infrastructure kubernetes should be a product of the configuration management.

Those machines are made very redundant.

All components in a Parallel Sysplex configuration can operate in redundant mode.

Redundant power. Redundant central processor complex. Redundant multichip module (what would be a chipset on microcomputer platforms). Redundant RAM Memory. Redundant Storage. Redundant internal interconnects. Redundant network adapters.

That's why corporations that require high availability to not get sued for millions/billions use them.

Geographically Dispersed Parallel Sysplex.

I'm not sure about that: you could still put something together within that budget, say, a few different VPSes across Hetzner and Contabo (or different regions within the same provider's offerings), with some circuit breaking and load balancing between those. Probably either a managed database offering, or a cluster of DBs running on similar VPSes.

Of course, this might mean that you have 1-3 instances of a service instead of 10-30, but as long as availability is the goal and not necessarily throughput, that can go pretty far.

Otherwise what you have is a budget that is lower than the possible implications of temporary downtime. That doesn't make sense in the real world.

  sed "s/amateur/comparatively poor, from a third world country, without VC money, or have cheap labor/g"

500 USD a month would get you approximately the following resources (taxes vary) on the aforementioned platforms:

  Contabo
  Nodes: 15 to 83 (depending on configuration)
  CPU: 150 to 332 cores
  RAM: 664 to 900 GB
  SSD: 4150 to 6000 GB
  
  Hetzner
  Nodes: 7 to 110 (depending on configuration)
  CPU: 86 to 192 cores
  RAM: 192 to 384 GB
  SSD: 2200 to 4800 GB
  
  (this includes regular VPS packages, not storage optimized ones, or dedicated hosting etc.)

> Otherwise what you have is a budget that is lower than the possible implications of temporary downtime. That doesn't make sense in the real world.

This (depending on the circumstances) does sound like a good point! Maybe "the real world" isn't the best wording, though, and choosing "enterprise settings" or anything along those lines would be more suitable.

But it leads me back to my original statement - extreme requirements for uptime don't come out of nothing.

If you're in a location where IT related labor is extremely cheap - you're just going to have people keep one server up.

I know I used to do exactly that, because the server was more than my annual income. But that didn't last long. After the first 20 minute downtime, the budget for HA solution was allocated. But before a certain point downtime wasn't expensive.

Non-profits would probably be the only reasonable exception, where HA and low budgets could coincide. Otherwise - nah...

> Maybe "the real world" isn't the best wording, though, and choosing "enterprise settings" or anything along those lines would be more suitable.

This is the point - for-profit corporations, by definition, don't want to waste money, and if "high reliability" isn't required (or they don't know about it), they don't waste money. Most of the time, they don't waste the money.

However, if "being cheap" would means billions in lost income (and they know about it), they really want to have reliable, redundant infrastructure and systems around.

I've almost never seen situations where a single request would need more resources available, than the entire server has (outside of large GPT models for text, though maybe that's because I couldn't afford beefy machines for that myself).

Instead, if your monolith needs X resources to run (overhead) and Y resources to serve your current load, then in case of increased load you can just setup another instance of your monolith in parallel with another set of X + Y resources (same configuration) and it will generally almost double your capacity.

Now, there can be some issues with this, such as needing to ensure either stateless APIs or sticky sessions, but both are generally regarded as solved problems (with a little bit of work). Monoliths themselves shouldn't be limited to running just a single instance and aren't that different from a scalability perspective than microservices.

Where microservices excel, however, is that you can observe individual services (e.g. systemd services or them running in containers) better and see when a particular service is misbehaving or scale them separately, as well as decrease that X overhead since each service has a smaller codebase when you have lots of instances running. This does come at the expense of increased operational complexity and possibly noisy network chatter, especially if you've drawn your service boundaries wrong.

However, at the same time I've seen actual monoliths that can never have more than one instance running due to problematic architecture, so therefore I propose the following wording (that I've heard elsewhere):

  SINGLETONS - a monolithic application that can ever only have a single instance running, for example, when business processes are stored in memory for a bit, or have user sessions or something like that stored locally as well; these will ONLY ever scale VERTICALLY, unless you re-architect them
  MONOLITHS - applications that contain all of your project's logic in a single codebase, although multiple instances can be launched in parallel, depending on your needs; can be scaled BOTH VERTICALLY and HORIZONTALLY; they have more overhead though and observability can be a bit challenging
  MICROSERVICES - applications that contain a part of the total project's logic, typically across multiple separate codebases, possibly with shared library code, pieces of your project can be scaled separately, BOTH VERTICALLY and HORIZONTALLY; they are operationally more complex, can involve more network chatter and while you can observe how services perform, now you need to deal with distributed tracing

I don't actually expect anyone to use these particular terms, but I dislike when someone claims that monoliths have the issues of these "singleton" applications when in fact that's just because they've primarily worked with bad architectures. Sometimes they wouldn't need to shoot themselves in the foot with microservices if they could just extract their session data into Redis and their task queues into RabbitMQ. Other times, microservices actually make sense.

On top of that, the tendency to get complacent with unstructured data in a lot of systems is really creating a very complicated lock-in when systems are developed for unique services on each cloud provider... Bowls of spaghetti.

Hire Solutions Architects for dev projects, make your apps future proof... I warn you. Too much microservice customization leads to vendor lock in and expensive operational costs... This is why a lot of apps get sunsetted early.

It might still be completely viable to rewrite something so that it's only 10% as efficient but you can scale it horizontally easily.

Aren't you still writing out to external data stores? If that's the case then its really not a comparison of in process to a RPC, it just two RPC hops now, no?

So instead of a single DB transaction, often with a single SQL roundtrip, microservices can sometimes force through coordination of separate transactions to several DBs.

What happens if you need to redesign the architecture to meet new needs? That's right; it's not going to happen, because people will fight it tooth and nail.

You also cannot produce any meaningful end-user value without involving several teams.

Microservices is just "backend, frontend, and database team" reincarnated.

My take: do Microservices all you want, but don't organize teams around the services!

This isn't easily fixable, but I'd like technologists to at least be able to perceive the extent to which the surrounding business culture has permeated into technical culture. I'm probably considered old AF by a lot of you (41) but I'm becoming increasingly interested in tools/methodologies that enable fewer devs to do a lot more, even if it means that there are sharp edges.

Horizontal scaling always incurs communication overheads faster, and as we know, developers have high communication overhead to begin with.

Human scaling is its own problem set and companies like to ramp projects up and then down again. Most of those people only think about how to get more people on board, not how to run the ship once attrition starts to condense responsibilities and reduce redundancy, especially for unpleasant tasks.

I’ve been trying to beat the idea of ergonomics into a smart but mule-headed developer (too old not to know better) for years now and it wasn’t until a combination of team shrink and having to work with code written by a couple of people who emulate his development style that all of a sudden he’s repeating things I said four years ago (of course with no self awareness of where those ideas came from). It remains to be seen if it affects how he writes code, or whether he starts practicing Do As I Say, Not As I Do, but I’m a short-timer now so it’s mostly an anthropological study at this point.

As projects get larger, the problems become more organizational in nature.

hence, if you want a certain architecture, you likely need to execute a "reverse conway law" org first. To get the org into the target config, the software will follow.

What does not work is splitting ownership of many tings for a long time. It’s too many plates to spin. But some things can be shared by a few people without tipping over into anarchy.

  Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.

  — Melvin E. Conway

Exactly. In order to reap the benefits of modular/Microservices architecture, you need teams to be organized around product/feature verticals.

IMO it’s not about the internals of a product/service architecture, but about encapsulating, isolating, and formalizing inter-organizational dependencies.

Also - talking of redesign should be a major lift for any product/service that has real paying clients. The risk of breaking is huge, and the fact that during that time you won't be delivering incremental value is also going to look bad.

Amazon's warehouses have incredible domain complexity which drives the complexity of the software architecture. It's also one of the oldest parts of the company with a direct lineage going back 20+ years. (for example, our inventory databases still have the same schema and are still using a RDBMS, albeit RDS Postgres instead of Oracle).

About five years ago we started rearchitecting to a centralized orchestration service which is being used to abstract this complexity from process teams. This is to avoid projects where 20+ service teams must be engaged just to launch a new solution.

How often does major reorganization happen? For most companies the answer is never.

The "what if" argument is what leads to all sorts of premature optimizations.

Yeah, didn't get a lot done...

For most companies – regularly.

But the need for them does :)

This is actually the whole point (see: Reverse Conway).

Conway's Law is basically "You don't have a choice." You will cement the design of your app around your organizational design. (At least, beyond a certain org size. If you have only 4 engineers you don't really have the sort of structure in question here at all.) je42 narrowly beats me to the point that if this is a problem you can try to match your organization to your problem, but that takes a fairly agile organization.

"What happens if you need to redesign the architecture to meet new needs? That's right; it's not going to happen, because people will fight it tooth and nail."

Unfortunately, in the real world, this is not so much "a disadvantage to the microservice approach" as simply "an engineering constraint you will have to work around and take account in your design".

Despite what you may think after I've said that, I'm not a microservice maximalist. Microservices are a valuable tool in such a world but far from the only tool. As with everything, the benefits and the costs must be accounted for. While I've not successfully rearchitected an entire organization from my position as engineer, I have had some modest, but very real success in moving little bits and pieces around to the correct team so that we don't have to build stupid microservices just to deal with internal organizational issues. I don't mean this to be interpreted as a defeatist "you're doomed, get ready for microservices everywhere and just deal with it"; there are other options. Or at least, there are other options in relatively healthy organizations.

But you will have code that matches the structure of your organization. You might as well harness that as much as you can for the benefits it can provide because you're stuck with it whether you like it or not. By that I mean, as long as you are going to have teams structured around your services whether you like it or not, go in with eyes open about this fact, and make plans around it to minimize the inevitable costs while maximizing the benefits. Belief that there is another option will inevitably lead to suboptimal outcomes.

You can't engineer at an organizational level while thinking that you have an option of breaking the responsibility and authority over a codebase apart and somehow handing them out to separate teams. That never works long term. A lot of institutional malfunctioning that people correctly complain about on HN is at its core people who made this very mistake and the responsibility & authority for something are mismatched. Far from all of it, there are other major pathologies in organizations, of course. But making sure responsibility & authority are more-or-less in sync is one of the major checklist items in doing organization-level engineering.

If your org doesn’t suffer feature creep willingly, I believe that means you can have a lot more flexibility with respect to Conway’s Law. A low-churn project can maintain a common understanding. Not all of them will of course, but it’s at least possible. Feature factories absolutely cannot, and you shouldn’t even try.

What really kills a lot of projects though is an inverted dependency tree. And by inverted I mean that the most volatile code is at the bottom of the call graph, not the top. In this case every build, every deploy, or in the microservices scenario every request, can have different behavior from the previous one because some tertiary dependency changed under you. Now you need all sorts of triggers in your CI/CD pipeline to guarantee that integration tests are rerun constantly to figure out where the regressions are being introduced. Your build pipeline starts to look like one of those server rooms with loose wires everywhere and a bucket under the AC unit. Nothing is stable and everything is on the verge of being on fire at a moment’s notice.

> Any organization that designs a system (defined broadly) will produce _a design whose structure is a copy of the organization's communication structure_.

Organization is the primary method by which we organize communication (indeed, that's its only point). That's why architecture follows organization. It is possible to "break" that boundary by organizing other forms of communication, for example the feature teams of SAFE (not a proponent of safe, just an example).

The good news is, pretty much everyone has been on at least one side of that balance being broken (that is, having responsibility without authority, or being subject to someone else's authority without responsibility), and a lot of people have been on both, and once you bring it to their attention that they're creating another one there is a certain amount of motivation to avoid it, even if it is a bit more work for a them. You can actually use people's experiences with bad organization to convince them to not create another one, if you are careful and thoughtful.

That will always happen, and I've seen it happen in a monolith too (expressed as module ownership/ownership over part of the codebase).

It's inevitable in almost all organizations.

Long ago I consulted for one company that reconstituted teams for every new project, so people got swapped around every few months, usually ending up in a new chunk of the code. Whatever the theoretical benefits, it meant that nobody felt a sense of ownership for any of the code. Combine that with aggressively short project deadlines and an absolute refusal to ever adjust them and the code rapidly dropped in quality. After all, you'd be on something else soon, so any mess wasn't your long-term problem. And why should you bother given that the code you started with wasn't pristine?

As far as I can tell, the best it gets is organizing around the most durable features of the business environment. E.g., audiences and their real-world needs. Next, long-lived solutions to those needs. And then you find the common and/or large needs of those teams and spin up service teams around them. And in the background, some strong culture around what the priorities really are, so that service teams don't get so self-important that they become the tail wagging the dog.

But I love all the war stories here and would love to hear more, as I think there are no perfect global solutions to organizational problems. I think it's mostly in the details of circumstance, history, and the people involved.

Microsoft, Apple, Amazon I think do a good job at preaching the customer first model and organizing around that.

Look, once upon a time managers designed the system that workers implemented. The factory assembly line, or the policy manual.

But now the workers are CPUs and servers. The managers designing the system the workers follow are coders. I say coders are the new managers.

Now this leaves two problems. The first is that there are a lot of "managers" who are now doing the wrong job. (and explains perfectly why Steve Jobs gets involved in emails about hiring coders.) But that's a different post.

For me the second problem is microservices represent the atomic parts of a business. They should not be seen as a technical solution to anything, and because the first problem (managers arent managing workers anymore) there is no need to have the microservices built along Conways Law.

And so if you want to build a microservice it is not enough to consider the technical needs, it is not enough to automate what was there, you need to design an atomic building block of any / your company. They become the way you talk about the company, the way you think about the processes in a company.

An mostly the company becomes programmable. It is also highly likely the company becomes built mostly of interchangable parts.

This is a misunderstanding of Conway's Law. Your code _will_ reflect the structure of your organization. If you use microservices so will their architecture. If you use modules, so will the modules.

If you want a specific architecture, have your organization reflect the modules/microservices defined in that architecture.

The point ebing is that if workers are CPUs and coders are managers, then why worry about how the managers of the coders are arranged. Get rid of that management layer. Conway is not implying the financiers of the organisation affect the architecture.

This basically means that the microservices a company is built out of should more readily align to the realities of the business model. And instead of shuffling organisations around it would behove executives to look at the microservices.

One can more easily measure orders transferred, etc, if the boundaries are clearer.

Plus conway is just a ... neat idea, not a bound fate.

There is a caveat with the architecture reflecting the organisation of the software teams, but that usually follows the other to-be-automated structure as well.

> Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure.

It doesn't matter WHO defines or does the work (the coders themselves in a team or the manager). It matters how communication is arranged. And communication naturally aligns itself along organizational and team boundaries.

When a group/person/whatever writes an API they are MOSTLY writing a document for other human beings. That is an act of communication. If you meet with someone everyday in a stand-up you're likely to put a less structured API between yourselves than if you meet with someone once a month, or once a year. You're also going to create a finer grained subdivision of the API for them as you're likely working on closely related posts of the problem.

Organization creates communication, and communication creates architecture. Therefore organization creates architecture.

> Plus conway is just a ... neat idea, not a bound fate

It is in fact bound fate. Code is written to communicate with others.

People come together, they decide to solve a problem. They divide up that problem and create functions (an API), objects (an API), modules (an API), and services/microservices (an API). At each later they've written something that communicates with others. The structure of that code reflects the necessary communication to the other parties for their part of the problem. This the code is structured following the lines of communication in the organization.

Organization is how we structure communication (that's it's primary point) and thus organization = architecture + some wiggle room for communication paths defined outside that (say a SAFE feature team).

Just to add it here, how evaluations and goals are set matter too. Those tend to be even more constrained by the formal organization than communication.

I've been a couple levels up the chain at times in my career. When I've seen technical decisions at this level done well is when those people realize that their organizational decisions are inevitably technical decisions because of Conway's law. When the impact of Conway's law is not explicitly considered, things will often look random at the IC level. Too often, organizational leaders don't realize they are dictating technical outcomes with decisions that appear to just be about reporting structure.

An example is when a major new product feature comes along that is big enough to need multiple teams to build/own it for the long term. Where are those teams placed in the organization? Conway's law helps us see that a huge part of the future architecture is being determined by answering that question. Placing those new teams in an existing org mainly responsible for UX will result in a very different architecture than putting the new teams in an org mainly responsible for monetization. Can the teams be split across those orgs? Well, can you envision a reasonable architecture with an enforced divide along those lines? Without recognition of Conway's law, that last question might not be considered and the decision to split or not will be based on other factors.

Unfortunately that requires multidisciplinary thinking (technical architecture, resource allocation, communication structure, people development, ...). Such broad thinking is not always easy. When leadership starts neglecting one or more of these disciplines, more and more of the people doing the work on affected teams start perceiving the decisions as arbitrary and counter productive. Organizational churn (e.g., teams repeatedly changing location within the org over the course of a year or two) is often leadership looking at a different subset of these dimensions over time, yielding different "correct" answers.

But if you chnage the dataflow in a few microservices so that the accounts team no longer deals / works directly with the sales team you have chnaged the organisation.

plus it's way easier to monitor activity etc

There's a third technical problem that microservices solve, and it's my favorite: isolation. With monoliths, you provide all of your secrets to the whole monolith, and a vulnerability in one module can access any secret available to any other module. Similarly, a bug that takes down one module takes down the whole process (and probably the whole app when you consider cascading failures). In most mainstream languages, every module (even the most unimportant, leaf node on the dependency tree) needs to be scoured for potential security or reliability issues because any module can bring the whole thing down.

This isn't solved at the language level in most mainstream languages. The Erlang family of languages generally address the reliability issue, but most languages punt on it altogether.

> The real reason that microservices may make sense is because they keep people honest around module boundaries.

Agreed. Microservices, like static type systems, are "rails". Most organizations have people who will take shortcuts in the name of expedience, and systems with rails disincentivize these shortcuts (importantly, they don't preclude them).

Imagine a world where every pip/nuget/cargo package was a k8s service called out-of-process and needed to be independently maintained. We would have potentially hundreds of these things running, independently secured via mTLS, observability and metrics for each, and all calls run out of process. This is the abysmally slow hellscape that some are naively advocating for without realizing it.

It is relatively obvious what should be a library (json, numpy, NLog, etc), and what should be a service (Postgres, Kafka, Redis, NATS, etc.) when dealing with 3rd party components. It is also obvious that team scaling is not determined by whether code exists in a library or service from this example since all 3rd party code is maintained by others.

However, once we are all in the same organization, we lose the ability to correctly make this determination. We create a service when a library would do in order to more strictly enforce ownership. I think an alternate solution to this problem is needed.

Services can be continuously deployed by small owning teams.

Is this correct or are there other aspects that you are considering?

So does modularity.

"The benefits expected of modular programming are: (1) managerial_development time should be shortened because separate groups would work on each module with little need for communication..."

On the Criteria To Be Used in Decomposing Systems into Modules, D.L. Parnas 1972.

http://sunnyday.mit.edu/16.355/parnas-criteria.html

He's right as far as Conway's Law goes, though.

Edit: failed to spell "I" correctly. Ouch.

The point is that Parnas never conceived of doing it because he was writing about a world where the interfaces between modules were known ahead of time and were static.

They aren't built in, it's just that the need for them is impossible to ignore. Developers (and management) can't help but recognize and respect modularity in microservices because the separation of services and the APIs between them make the modularity obvious. When the separation between modules only exists in code, and even then only when seen through the eyes of someone who understands the modular architecture of the codebase, it is easily ignored, inevitably forgotten, and might as well not exist at all.

If all modules have to be deployed using the same build even though different build of the modules would have been API compatible - you don't have modules.

IMO a module is defined as something that has a self-contained API, and versioning rules for that API if the module is evolving.

With modules you need some sort of wrapper application to bring it all together.

With microservices you need some sort of network layer so that the microservices can talk to each other.

You don't restart the network every time you deploy a microservice.

If we're being technical some languages support hot swapping modules so no restart would be needed. Setting that aside, restarting an application isn't anything that needs coordination today. You wouldn't even restart the application. You'd deploy a new version, swap over, and shut down the old version.

>You don't restart the network every time you deploy a microservice.

No, but something changes in the network configuration so that the other microservices are aware of the deployment.

With microservices, as long as you maintain the contract with caller services, you can deploy whenever you want. If you have some kind of issue, your team is solely responsible. If the caller services do weird things, they are responsible for fixing them.

If you are pushing changes to a module as part of a more monolithic, or wrapper service - if you do a deploy and break the whole big service, you are now responsible for the entirety of any issue, which is now more likely due to integration risk and unrelated changes from others, especially because now there need to be coordination across many teams integrating - hopefully via tests and observability. But this requires a high-degree of maturity for automated quality assurance and site reliability. If you don't have that, the operational risks are very high. So that alternative is having some ops-like function, or other integration team responsible. Or doing more top-down waterfall coordination.

Given the service maturity needed is rare, microservices distributes that operational ownership in a way where there is more accountability.

Infrastructure that someone has to take care of.

In chaotic environments like that, microservices help with defining clear ownership of operational responsibility, though at the cost of clear responsibility for the overall distributed system. This comes at the cost of making it hard to develop, let alone ship, changes that impact multiple microservices (outside the scope of a single team, or closely related sibling teams).

The main point I was making was that, with a more monolithic system, managing that kind to organization disfunction, necessitates top-down, waterfall like control. So moving to microservices enables non-coordinated agility removing one layer of coordination problem (a the cost of other problems).

Developers do a much better job with microservices. I think it's easy for them to respect the seriousness of designing and changing the API of a microservice. In contrast, they often don't respect or even realize the seriousness of making a change that affects the modular design of code.

Language-level support for defining and enforcing module interfaces might help developers invest the same level of care for module boundaries in a modular monolith as they do in a microservices architecture, but I've yet to work in a language that achieves this.

Often people are too lazy to issue API version change on a "bugfix", and take down multiple applications.

Funny thing, a modularized monolith may have identified some of those issues.

If you have a particular piece of the system that needs to be scaled, you can take that module out when it becomes necessary. You can alter your system such that you deploy the entire codebase but certain APIs are routed to certain boxes, and you can use batch processing patterns with a set of code outside the codebase.

You can have an admin application and a user application, and both of them are monoliths. They may or may not communication using the database or events or APIs.

However, you don't make this on the single bounded context guideline.

And I do agree that most people need less of those than they think.

I gave a talk [1] about scalability of Java systems on Kubernetes, and one of my recommendations is that Java-based systems - or any system on a runtime similar to the JVM, like CLR or even Go - should be scaled diagonally. Efficiency is the word.

While horizontal scaling can easily address most performance issues and load demand, in most cases it is the least efficient way for Java (and again, I risk saying .NET and Go), as these systems struggle with CPU throttling and garbage collectors.

In short, and exemplifying: one container with 2 CPUs and 2GB of RAM will allow the JVM to perform better, in general, than 2 containers with 1 CPU/1GB RAM each. That said, customers shouldn't be scaling horizontally to any amount more than what is adequately reasonable for resiliency, or unless the bottleneck is somewhere like disk access. For performance on the other hand, customers should be scaling vertically.

And Kubernetes VPA is already available. People just need to use it properly and smartly. If a Kubernetes admin believes a particular pod should double in number of replicas, the admin should consider: "would this microservice benefit even more from 1.5x more CPU/Memory than 2x more replicas?" and I bet to say that, in general, yes.

[1] https://www.youtube.com/watch?v=wApqCjHWF8Q

My default recommendation has always been to make instances "as big as possible, but no bigger". You may need 3 instances for redundancy, fault tolerance and graceful updates but past that you should probably scale up to near the size of your machine. There are obviously lots of complications and exceptions (for example maybe using most 3 instances uses 90% of your machines so you can't bin pack other processes there so it is better to use 4 instances at 70% as the machines will be used more efficiently) but bigger by default is generally a good option.

Exactly.

But I am seeing more and more of the opposite: make instances as small as possible, and scale out (not up) if/when needed.

Making data available to a thread on the same core has exceptionally high bandwidth. Just that alone will help with scaling.

Even if Microservices are better for scale, most companies will never experience the level of scale of 2013 Twitter.

Are Microservices beneficial at much smaller levels of scale? Ex: 1M MAU

I fully agree with your argument. Then again, as mentioned elsewhere in this discussion, microservices are often not introduced to solve a scalability problem but an organizational one and there are many organizations that have more engineers than Twitter (had).

Personally, I still don't buy that argument because by solving one organizational problem one risks creating a different one, as this blog post[0] illustrates:

> […] Uber has grown to around 2,200 critical microservices

Unsurprisingly, that same post notes that

> […] in recent years people have begun to decry microservices for their tendency to greatly increase complexity, sometimes making even trivial features difficult to build. […] we experienced these tradeoffs first hand

I'm getting very strong Wingman/Galactus[1] vibes here.

[0]: https://web.archive.org/web/20221105153616/https://www.uber....

[1]: https://www.youtube.com/watch?v=y8OnoxKotPQ

I'm not sure at what point the monolith codebase became absolutely enormous, but I would bet that GitHub grew to 1M active users with a monolith just fine.

Micro-services might be necessary for the companies like Github, Twitter, Google that grow to be huge, but monoliths seem to work just fine for the vast majority of other companies.

If a team wants to try out a different language, or hosting model, or even just framework/tooling, those things can be really hard to do within the same codebase; much easier when your only contract is handling JSON requests. And if your whole organization is totally locked into a single stack, it's hard to keep evolving on some axes

(I'm generally against microservices, but this is one of the more compelling arguments I've heard for them, though it still wouldn't mean you need to eagerly break things up without a specific reason)

Not all engineers need to move around - DS and MLEs are generally useless on the frontend, and vice versa.

Each team gets to distribute libraries over repos (COM, JAR, DLL, whatever), no way around that unless they feel like hacking binaries.

Creating an object-oriented API for a shared library which encapsulates a module to the degree a service boundary would is not trivial and it's very rarely done, never mind done well. Most OO libraries expose an entire universe of objects and methods to support deep integration scenarios. Maintaining that richness of API over time in the face of many consumers is not easy, and it (versioning) is a dying art in the eternal present of online services. The 90s aren't coming back any time soon.

If you own a library which is used internally, and you add a feature which needs a lot more memory or compute, how do you communicate the need to increase resource allocation to the teams who use the library? How do you even ensure that they upgrade? How do you gather metrics on how your library is used in practice? How do you discover and collect metrics around failure modes at the module boundary level? How do you gather logs emitted by your library, in all the places it's used? What if you add dependencies on other services, which need configuring (network addresses, credentials, whatever) - do you offload the configuration effort on to each of the library users, who need to do it separately, and end up with configuration drift over time?

I don't think binary drops work well unless the logic they encapsulate is architecturally self-contained and predictable; no network access, no database access, no unexpected changes in CPU or memory requirements from version to version.

There's plenty of code like this, but it's not usually the level of module that we consider putting inside a service.

For example, an Excel spreadsheet parser might be a library. But the module which takes Excel files uploaded by the user and streams a subset of the contents into the database is probably better off as a service than a library, so that it can be isolated (security risks), can crash safely without taking everything down, can retry, can create nice logs about hard to parse files, can have resource metrics measured and growth estimated over time, and so on.

As for scalability, most orgs aren't Facebook nor Google scale, regardless of how much they want it to be true.

All of those issues can be tackled, when proper architecture design is done, instead of coding on the go and then will see attitude.

The important part of microservices isn't just API boundaries, it's lifecycle management. This CAN be done with a DLL or JAR, but it's MUCH harder today.

Sure if one likes to make things harder than they are supposed to be, there are lots of paths down that road.

At my last job, there were quite a few times where being able to scale some small "microservice instance" up from 2 -> 4 instances or 4 -> 8 or 8 -> 12 was a lot easier/quicker than investigating the actual issue. It'd stop production outages/hiccups. It was basically encouraged.

Not sure how that can be done with a "it's all modules in a giant monolith".

> A few ways, the easiest being to scale up the whole monolith with more instance

as far as I know, there's no way to granularly scale up a monolith. if the monolith has 20 or 50 or 100 modules and you need 1 or 2 of them scaled, you have to scale up the entire thing which is huge, expensive, etc.

> Another way is run multiple "services" using the same codebase, so you have workload segmentation, either via synchronous network calls, or async worker systems.

this is interesting. a monolith with some form of IPC? why not do microservices at that point? that sounds like microservices-ish?

Yes, yet people still do it that way. This is tradeoff against the costs of microservices. I'm not saying it is worth it, but yes, sometimes you can just inefficiently throw hardware resources at scaling problems.

> this is interesting. a monolith with some form of IPC? why not do microservices at that point? that sounds like microservices-ish?

Because you are incrementally changing an existing monolith, but still getting some of the benefits of scaling distinct workloads independently. If you do this right, it does lend it self to reworking the new "service" to be become its own microservice. Or you can just keep going with the shared codebase.

Microservice scaling is also an inefficient hardware scaling option.

it brokerages them to subclasses/handlers for CRUD/RPC/plumbing to database/other services

then in a separate process App/main() you have a cron job or a queue of some sort

you bundle these together as a monolith into one main()?

We deploy a few instances of the monolith fronted by a load balancer. There's not much scaling logic here -- when we see p75 request latency reach a certain threshold, we add another monolith instance.

A while later, we realize that the /foo route is getting a ton of spikey traffic, which makes it hard to scale. So, in the load balancer, we set up a new pool that just targets /foo, and in our Maven project we create a new module that will be the deployable for this pool. Now, /foo still requires most of our libs, so the jar we're bundling is still "the monolith" in essence, but maybe it doesn't require a few dependencies, and so is a little slimmer. We deploy that and everything is scaling great!

Then, we discover that in our main monolith, the background threads we were using to send emails are getting really clogged up, which is adversely effecting performance. We decide we want to move this work to an external persistent queue, like Kafka. Now, for our consumer, we create one more Maven module for a new jar. This time, we're lucky. Emails only need 2 of our libraries and so this is actually a pretty small deployable, but it's still being built from the same core set of libraries as the monolith.

my concern would be if you accidentally create bugs by basically creating new mini-monolith flavors as the modules were never expected (in the beginning of their creation) to run in a "partial" context

A cool side-effect is that you can usually run the whole thing in one app for development by just enabling all profiles - as opposed to some microservice architectures where you need dozens of containers and DBs for replicating inter-service bugs.

The benefit though is you get a bit of 'free' scaling. Most of the modules don't even have to consider scaling at all, as long as they are growing ~average or slower. So this saves a lot of operational burden for many teams. Conversely with microservices every team has to consider scaling, no matter how simple their service is.

(If you do have 1 module out of 100 that takes up 40% of the resources, then it may be appropriate to split it up. Monolith doesn't literally mean "exactly one service", after all, just a small number of big ones.)

I could be wrong but doesn't monolith usually refer to one really heavy app? As soon as you go from needing 1 instance to 2 (because 1 of the 100 inner "modules" needs to be scaled), I would guess most of the time there's a high expense (lots of memory) in duplicating the entire monolith? Unless you can scale threads or something instead of the entire process... Or if the entire process doesn't initiate 100 modules as soon as it is started (although I imagine it would in most cases)

No. Backend Server clusters are around almost as long as the internet exists.

>I would guess most of the time there's a high expense (lots of memory) in duplicating the entire monolith?

That's right, but there's a gotcha. Everytime you split your monolith into parts, that doesn't mean that each part will consume 1/n of original monolith startup resource. There will be a floor of memory usage. Consider 1 monolith that eats up 1GB to startup vs 4 microservices that uses 512MB each. Right from start you doubled the wasted memory from 1GB to 2GB. That only gets worse the more services are created. Another problem is that microservice/cloud folks loves to create anemic machines. Usually setting up 512MB RAM and half a core cpu to a service that needs 380mb minimum :) Thats 75% overhead and 25% of working memory. It's bonkers.

not necessarily. yes, you do pay for executable bloat and some memory overhead from loading code into memory, but the argument here is that you can still deploy a "Service B" from the same deployable that only services certain kinds of requests in order to scale parts of the application horizontally.

> this is interesting. a monolith with some form of IPC? why not do microservices at that point? that sounds like microservices-ish?

no, because again, you're still deploying the same single codebase, it's just the instance on the other end of the queue is configured just to consume and process messages.

This happens a lot. Organisational problems conflated for technical ones. Technical solutions to organisational politics. Etc. It's often easier to admit technical challenges than organisational ones. Also, different people get to declare technical and organisational challenges, and different people that get to design the solutions.

There's also a dynamic where concepts are created by the technical 1% doing vanguard or challenging tasks. The guys responsible for scaling youtube or whatnot. Their methods and ideas become famous and are then applied to less technically demanding tasks, then in less technical organisations entirely.

I think if we can be honest at the actual problem at hand, 80/20 fixes will emerge. IE, the "real" value is not the architecture per se, but the way it lets you divide the responsibilities in the organisation.

https://ardalis.com/conways-law-ddd-and-microservices/#:~:te....

I would like someone to spell this out. It seems to me people are claiming that if a single binary serves some CPU-bound requests and some memory-bound requests, and you give it more memory, then the memory gets "wasted" on the CPU-bound part. Or if you give it more CPU, the CPU gets wasted on the memory-bound part. But this kind of assignment of resources to code paths seems to be a consequence of microservices. In a single computer, single binary situation resources should not get used up unless the workload actually wants to use them. A compute-heavy thread doesn't cost heap. A big heap doesn't slow down a compute-heavy thread. What am I missing?

My IDE should be able to easily traverse the call graph. My development environment should be simple to setup.

I’ve worked on microservices that required an insane amount of boilerplate to do simple things. Like 7 layers of controllers, clients, services, data services, etc, just to fetch a simple piece of data. And the developer experience of running dozens of services in a Kubernetes cluster running on my dev machine was awful.

Does anything like this exist?

I only dabbled many years ago but Erlang/OTP comes to mind.

And tRPC for TypeScript calls in browser and server.

> https://github.com/NathanRSmith/lib-courier-js

Interestingly, we're pursuing a monorepo & multi-monolith setup for the next version of our platform. So lib-courier is no longer necessary to stitch it all together. It was fun while it lasted though. Once you understood the routing algo & code patterns, lots of stuff "just worked".

It turns out that "transparent RPC" is basically a contradiction in terms. As soon as you start doing things across process boundaries, and even more so across network boundaries, it requires a very different approach for API design - something that's very cheap locally, like passing objects by reference, becomes expensive and full of footguns.

If you reduce the feature set to the point where it can be transparently mapped to either local or RPC - which is, basically, function calls processing and returning data organized into arrays & trees (but not graphs) - there's still the issue that RPC has so many more failure points that you have to handle that would never light up in local.

This is all still doable; I have my doubts about practical the end result would be, though.

Totally fine to support a limited set of primitives and collections passed by value, and require developers consider the failure modes.

In my experience HTTP+JSON or gRPC are usually lacking in at least one of those respects. If you constrain all the services to be the same language it becomes easier.

Modularization allows development to scale.

Microservices allow operations to scale.

This breaks down when the database is essentially a generic graph. The worst solution I've seen to this is to have another service responsible for generic write operations and any service that wants to write data goes through that service -- you're essentially re-introducing the problem you're purporting to solve at a new layer with an added hop and most likely high network usage. The best solution I've seen is to obviously have the monolith. The enterprise solution I've seen, while not good by any means but nearly essential for promoting team breakdown and ownership, is to just let disparate services write as they see fit, supported by abstraction libraries and SOPs to help reinforce care from engineers.

I’ll go one step further and say that you should treat your data stores as services in and of themselves, with their own well-considered interfaces, and perhaps something like PostgREST as an adapter to the frontend if you don’t really need sophisticated service layers. The read/write pattern you recommend is a great fit for this and can be scaled horizontally with read replicas.

Been there: how do you handle schema changes?

One of the advantages that having a separate schema per service provides is that services can communicate only via APIs, which decouples them allowing you to deploy them independently, which is at the heart of microservices (and continuous delivery).

The way I see it today: everyone complains about microservices, 12 factor apps, kubernetes, docker, etc., and I agree they are overengineering for small tools, services, etc., but if done right, they offer an agility that monoliths simply can't provide. And today it's really all about moving fast(er) than yesterday.

Our data is mostly append-only, or if it's being changed, there is a theoretical final "correct" version of it that we should converge to. So to "get" data, you subscribe to messages about some state of things, and then each service is responsible for managing its own copy in its own db. This worked well enough until it didn't, and we had to start doing true-ups from time to time to keep things in sync, which was annoying, but not particularly problematic, as we design to assume everything is async and convergent.

The optimization (or compromise) we decided on, was that all of our services use the same db cluster, and that if the db cluster goes down, it means everything is down. Therefore, if we can assume the db is always up, even if a service is down, we consider it an acceptable constraint to provide a readonly view into other services db. Any writes are still sent async via MQ. This eliminates our syncing drifting problem, while still allowing for performant joins, which http apis are bad at and our system uses a lot of.

So then back to your original question, the way that this contract can break is via schema changes. So for us, since we use postgres, we created database views that we expose for reading. And postgres view updates are constrained that they must always be backwards compatible from a schema perspective. So then now our migration path is:

- service A has some table of data that you like to share

- write a migration to expose a view for service A

- write an update for service B to depend upon that view

- service B now needs some more data in that view

- write a db migration for service A that adds that missing data, but keeping the view fully backwards compatible

I don't think I understand. You need to update (and deploy) service B every time you perform a view update (from service A), although it's backward compatible?

If you don't need the new column, then you don't need to do anything on service B, because you know that existing columns on the view won't get removed and their type won't change. You only need to make changes on service B when you want to take advantage of those additions to the view.

But hey, every team and company has to find their strategy to do things. If this works for you, that's great!

It's just not a microservice by definition.

And yes, that is correct, we agree that once we expose a view, we won't remove columns or change types of columns. Theoretically we could effectively deprecate a column by having it just return an empty value. Our use cases are such that changes to such views happen at an acceptable rate, and backwards incompatible changes also happen at an acceptable rate.

Our views are also often joins across multiple tables, or computed values, so even if it's often quite close to the underlying tables, they are intentionally to be used as an abstraction on top of them. The views are designed first from the perspective of, what form of data do others services need?

And any benefit of a microservice owning it's own rDB is still, that schema changes aren't easily reversible. Specially when new, non predefined, data has been flowing in.

Stateless microservices are great, in the sense that you don't have to build multiple versions of APIs... but stateful microservices are just a PITA.

And the fact you must keep backward compatibility ( see the OP answer above) at the implementation level shows how fragile this approach is - you will never be able to change a database schema as you wish just because you have consumers that rely on the internal details of your implementation - a table. If you want to change a field from char to int, you can't. How is it important for service B to know that level of detail? An API could still expose a domain model as char, if you want to, and maybe introducing new fields, new methods, whatever way. Or maybe nothing at all, maybe it's not necessary because the database field is never exposed but only used internally (!!).

On the other hand, if you expose a database agnostic API (e.g., http, rpc, ... whatever) you can even swap the underlying database and nobody will notice.

A good rule of thumb is: if I change the implementation, do I need to ask/tell another team? If the answer is yes, that is not a microservice.

something like this

CREATE VIEW table_read_api AS SELECT TO_CHAR(now_int) as was_char FROM the_table;

not sure that the view is an internal structure because it will be exposed via API as it is.

SQL allows to swap the database at least if no specific SQL features are used.

Which proves my point: if I change a field type ( in the table ), I will have to change the view type. I need to change 2 services because one of them changed an implementation detail.

CREATE TABLE the_table (was_char CHAR);

CREATE VIEW table_read_api AS SELECT was_char FROM the_table;

a client consumes data via:

select was_char from table_read_api;

after the change

alter table the_table modify was_char int;

alter table the_table rename column was_char to now_int;

CREATE VIEW table_read_api AS SELECT TO_CHAR(now_int) as was_char FROM the_table;

the client uses the same query

select was_char from table_read_api;

the type of the column has not changed

it is still a char

only "internal implementation" is changed

Microservices allow your permissions to be clear and precise. Your database passwords are only loaded into the process that uses them. You can reason about "If there's an RCE in this service, here's what it can do". Trying to tie that to monoliths is hard and ugly.

And for that matter it can (but not necessarily) limit how much damage a bad code release can do.

Of course you don't get those benefits for free just by using microservices, but impleminting those kind if boundaries in a monolith is a lot harder.

Microservices aren't inherently more secure.

Maybe it is possible with some low-level system calls, but at the very least it is a lot more difficult than using separate VMs or containers.