So, looking at the architecture, it seems like you have some design components that overlap with pieces design components in CoreOS. I'm not clear as to what components are really application components vs. platform operations components, so maybe that's it. But the use of rsyslog in lieu of systemd seems like really going against the CoreOS grain. Is this really a function of trying to mirror Heroku as closely as possible?
My understanding of Deis's design goals is that they will pick and choose their favorite components from any number of sources (including but not limited to CoreOS), but don't plan on locking themselves into any particular distro. Instead they are relying on Docker itself as the point of interoperability.
As a maintainer of Docker I think it's a good thing that you preserve the freedom to run your platform on any number of distros, and are not stuck, for example with having to install systemd everywhere.
I grok the container win (been using containers on non-Linux platforms and OpenVZ on Linux platforms for ages).
I agree a nice aspect of using Docker is that you have the flexibility of using whatever distro you want for each application component... however a heterogeneous container platform doesn't really provide much of a win, and to make that work you just end up implementing your own solution for each of the components that abstract you from the platform.
That's assuming Docker doesn't expand its interface to allow for more parts of the platform to be interoperable :)
The whole point of Docker is to introduce simple interfaces to important components of distributed systems, and get a critical mass of people to agree on it, producing interoperability. So far we did this for packaging and sandboxed execution of individual components, with some success. We plan on doing this for other areas in the future, including clustering, stack composition, application networking, service discovery, authentication and access control, etc. This will allow rival implementations to compete in each of these areas, without creating fragmentation. I don't want a World where "CoreOS containers" can't talk to "Atomic containers" or "Mesos containers" because we couldn't agree on a common interface - we can do better than that.
We are pretty open about that plan. The Deis guys know this and are making the bet that embracing "Docker-native" interoperability is good for them.
I think I didn't present what I was saying clearly.
I think you definitely want to be very open about what the inside of the container looks like and even what the interface is between the containers and the rest of the world.
I was thinking about the platform that the containers run on top of. You don't gain anything by using a given host platform, but building alternate components that handle deploying and wiring up those containers. You're better off either using the existing platform's components or contributing patches to it, and the latter really only makes sense if you're adding something distinctive.
I understand. But from the point of view of Deis, CoreOS is not a platform. It is a Linux distro with a loose collection of tools pre-installed, some of which Deis happens to use. One such tool is Fleet. Another is the systemd log journal. Each of those tools has dozens of competitors. Deis reserves the right to pick and choose from those competitors without locking themselves in. It also reserves the right to use some pieces of CoreOS without using the other. If you think of CoreOS as an end-to-end monolithic platform, this may seem strange. But it is perfectly aligned with the Docker philosophy, which is that big monolithic platforms are not a good thing. What you want is a vast ecosystem of small composable lego bricks.
> Deis reserves the right to pick and choose from those competitors without locking themselves in.
Well of course! It is your own project.
> If you think of CoreOS as an end-to-end monolithic platform, this may seem strange.
Yeah, but I'm not looking at CoreOS as an end-to-end monolithic platform. I'm looking at it as a bunch of components that happen to be bundled together, each of which is pretty nice all by itself. It doesn't mean you have to use them all, but on the other hand, you would hope that if you were going to replace one of them entirely you'd go with something with a significant advantage (and I'd argue in some cases it has gone the opposite way).
> What you want is a vast ecosystem of small composable lego bricks.
Understood. I just don't think you help yourself much if you take out one brick and replace it with one of your own, particularly if the new brick isn't any better than the old one (and might be worse).
> So, looking at the architecture, it seems like you have some design components that overlap with pieces design components in CoreOS. I'm not clear as to what components are really application components vs. platform operations components, so maybe that's it.
Deis was architected before the CoreOS integration. While we love CoreOS, architecturally we need to remain scheduler agnostic. Probably a good opportunity for a blog post...
> the use of rsyslog in lieu of systemd seems like really going against the CoreOS grain
We view platform logging as separate from application-level logging. Aggregating logs through the systemd journal is possible, but it would tie us a little too closely to systemd. Using log sidekick containers (as we do) is more portable and easier to funnel to end developers.
> Deis was architected before the CoreOS integration.
Okay this makes sense. If I were in your shoes, I might be tempted to radically reexamine the architecture in light of CoreOS's design. Otherwise you end up duplicating work or working at cross purposes, and CoreOS's components (systemd, etcd, fleet, etc.) are pretty cleanly separable from both CoreOS and each other already, so you shouldn't end up being married to CoreOS should there be some kind of problem.
Given that you're maintaining a level of agnosticism, what do you see as the main thing you are gaining by integrating CoreOS?
> We view platform logging as separate from application-level logging.
One can certainly take that view, but I'd argue it does lead to more complexity for managing logging.
> Aggregating logs through the systemd journal is possible, but it would tie us a little too closely to systemd. Using log sidekick containers (as we do) is more portable and easier to funnel to end developers.
Fair enough, but if you really want to be agnostic, I'd say have the contract that in the end everything goes in to a syslogd of some kind, but have the default approach to getting that job done be via systemd (with /run/systemd/journal/syslog to get it the rest of the way). If someone wants to use rsyslog or something else to do the job, so be it.
While using the log sidekick approach is popular, it's actually quite inefficient and clumsy. Particularly on magnetic disk, you've got this nice, efficient, append only per-event log writing that doesn't even have to be cached and then you periodically scrape through the old stuff as if it were just a blob of bytes.
The big win though is that you no longer need a sidekick for managing logging from each container. No muss no fuss about where the logs are, etc. Instead you just direct logging to systemd (or if need be, the syslogd socket). No sidekick required!
I just interviewed Deis CTO Gabriel Monroy about Deis and how it works with CoreOS and how it differs from Flynn and Dokku. Great guy, great team, and awesome project.
It is their own nginx HTTP / TCP router, which is backed by etcd and confd. You can check it out at https://github.com/deis/deis/tree/master/router. Essentially, you can fire up multiple of these instances, and Deis uses them to route application, controller, and git traffic to the proper container.
https://twitter.com/bacongobbler