> To optimize cold start overheads, Fission keeps a running pool of containers for each environment. When a request for a function comes in, Fission doesn't have to deploy a new container -- it just chooses one that's already running, copies the function into the container, loads it dynamically, and routes the request to that instance. The overhead of this process takes on the order of 100msec for NodeJS and Python functions.
> You should reduce your server response time under 200ms
Still, I've heard that AWS Lambda may run around 200 ms, so perhaps Fission is doing OK here.
But with modern progress in fast-starting compiled languages like Go and Rust, I'd love to have some easy way to supply small static binaries instead of source code. With compiled web server frameworks, it's feasible to handle over 250,000 (very simple) requests per second, or thousands of requests while Fission is trying to figure out where to send your source code. It doesn't always make sense to pay for interpreter startup overhead on every request (EDIT: this is not the case according to the Fission developers), especially not now that compiled code is so easy.
About that 100msec overhead -- there's a bunch of optimizations yet to be done, and also we think we can reduce the median at the expense of tail latency; but for now 100msec is good enough performance for a good enough set of use cases (but of course we'd love to hear from people for whom it really doesn't work at all).
Also, we could add knobs that let you tune how long a function stays "warm", so you can tune the latency overhead vs. resource consumption trade-off.
> But with modern progress in fast-starting compiled languages like Go and Rust, I'd love to have some easy way to supply small static binaries instead of source code.
We're very interested in this use case. We're redesigning the function environment stuff right now to better support compiled languages, and allowing users to bypass the source-code level stuff is quite possible.
BTW, to be clear, for a compiled language Fission wouldn't compile source code at runtime; there would be separate build stage that would run when you add the function to fission. I'm also not sure why you say we pay interpreter startup overhead on every request (we don't).
So in the old days there was a way to supply a binary rather than source code. That binary could be fast starting. There was even an open standard for connecting those binaries to different web servers. The standard used unix patterns such as reading stdin and writing stdout.
Sometimes it seems like people have created an insanely high learning curve to run their CGI programs. Sometimes it seems like most of the magic and value add of some of these platforms is that they reduce some of the overhead they added.
The cold start impact is quite high, but that's only for APIs which have been idle or are scaling up rapidly.
If you were getting even 1 request per second, the container last used would remain live, so the response time would then be in line with normal responses from a typical NodeJS server running the same code.
I agree though, supplying binaries in something like Go or Rust will be a very nice addition.
They're talking about cold start, not long-term performance. After the initial warm up period, performance should be as fast as Node/Python can execute it. It's the same experience for Lambda and Cloud Functions in my experience.
> But with modern progress in fast-starting compiled languages like Go and Rust, I'd love to have some easy way to supply small static binaries instead of source code.
Isn't that just what "regular" Kubernetes is, except that you wrap a Linux image/container around your static ELF binary? The Stack build tool for Haskell has an option for building as a Docker image, wrapped around the static Linux ELF binary that it would otherwise produce. I assume something similar exists for Go and Rust.
I always thought that things like Fission are best for things that will take long enough that 200ms doesn't make much of a difference such as video encoding, updating recommendations, etc. What reducing the overhead does is make it useful for more things like machine vision for images (eg: is this user uploaded image safe for work) but its still nowhere near good enough for doing a DB query that takes 1ms or serving static content.
In case anyone else was wondering how the auto keep-warm feature is implemented: it's a CloudWatch Event that runs the Lambda every four minutes by default.
While this probably keeps the majority of the requests warm, I would expect that there would still be some cold ones every now and then. Amazon doesn't provide a guarantee in how the containers get recycled.
We did some analysis[1] on cold starts in AWS Lambda, and found that there is a cold start if there's no activity for 5 minutes, or regardless of activity, there will be a cold start every 4 hours from creation of the underlying VM.
We've been planning on supporting other DIY/non-AWS offerings in this space - particularly IBM's OpenWhisk[1]. If the Fission authors are here - can I ask how the two stack up? OpenWhisk's ability to write your own custom even sources seems like a leg up, as does their wider language support, but Fissions's Kubernetes foundation seems preferable. Curious how the two compare!
Hi there, it would be awesome to have Zappa support Fission, happy to talk more about it!
We're working on a detailed comparison with Openwhisk, but basically -- we bet on Kubernetes as a good foundation to build this on, and it seems to be going well so far. It's true that we're at an earlier project maturity stage than Openwhisk is. In particular, we're actively working on integrating event sources and adding support for more languages.
Can you talk more about Fission Router? This seems to be where most of the hand-waving is happening right now :)
AWS API Gateway doesn't get very much love, but it is actually a very important part of the server-less ecosystem.
Why did you write your own Router and not make an nGinx plugin, for instance? How well does it scale? How do you deploy it? What features does it have?
Feature-wise, fission router is very simple right now. It's a service based on gorilla/mux. We had to write our own router because of cold-starts -- basically, the router needs to be able to talk to the pool manager and to get an instance when one doesn't exist, while holding on to the incoming request. (I'm not sure how much of this is possible to write as an nginx plugin.)
I don't have perf numbers just yet, but the router is stateless and you can just run more replicas when you reach its limit. It caches results from the rest of fission components, and most requests are on this fast path.
(We're also thinking about adding some existing feature-ful API proxy in front of the router, to get stuff like authentication, rate limiting, etc.)
It might be the case that docker was just some enabling technology to go back to static binaries even with dynamic languages that IMHO expect a more complicated runtime setup.
So if one goes with statically compiled web services written in e.g. Go, Rust, Haskell or any other language supporting static binaries (most languages support this actually, but not all of them encourage you to do it), the only thing missing is a deployment framework that distributes your binaries to your (potentially virtual) machines. So am I correct that this is the point where Kubernetes comes in?
So how to use Docker in production is a controversial topic and the consensus seems to change rapidly, but in my opinion is mostly used as a weapon against dependency hell, but this problem is mostly solved using static binaries. Other use it for sandboxing, but I am not sure what security guarantees one can expect here.
If one compares static binaries with Docker containers how do they compare from a security point of view if one is hyperparanoid that the code executed may harm the host?
I would add another advantage: You get to escape the steaming mess that is configuration management. Our legacy landscape uses VMs configured with Chef, and our new landscape has containers on Kubernetes. I like the latter approach much better since I don't have to care about idempotency or whatever. I just scrap the container, start a new one, and its startup configuration is handled by a (usually trivial) shell script that `exec`s into the application binary at the end.
But if the shell script is trivial, why not just give the binary its runtime constraints via arguments or environment variables?
IMHO configuration management overlaps with dependency hell, or could be seen as a dependency. You do not have to bake the dependency into the binary, but just have to setup the runtime environment.
Putting a shell script in front of a third-party binary is easier than setting up a build pipeline to build that binary with the modifications that you need, and maintaining those patches across releases. (And that's implying that you have the source code for the binary at all.)
Of course, the perfect solution is to upstream the necessary changes into the binaries that you need, so that you can supply all configuration via args and env.
I'm relatively new to the Serverless/FaaS space. Or to be more precise: I missed AWS Lambda and only heard about it when OpenWhisk was donated to Apache.
I have a couple of questions, maybe someone here has answers for me:
1) In particular I'm wondering about expensive one-time/setup operations (e.g. opening a database connection) that are usually done once at startup of a service. But that doesn't really apply here. How are these things handled?
2) As a Java guy I also wonder which of these tools support Java and how they deal with dependencies and JVM startup times etc.
3) And my last question: How do these integrate into development workflows? Continuous integration? Run this offline/local etc. Will I need a Kubernetes instance on my machine?
2. We don't support Java yet, but we'd like to. The JVM would already be running by the time there's a request, so there won't be a JVM startup in the request path.
3. Dev workflows: Lots to do on that front. A big part of our roadmap is improving the dev workflow, better versioning support, CI/CD.
For running locally, yes, the best way is to run a local Kubernetes instance on your machine. minikube makes it very easy to do that, see the instructions in the README: https://github.com/fission/fission#setup-kubernetes
OpenWhisk actions (equivalent to functions in other frameworks) have two life cycle events: initialization and execution. The very first activation of an action (a cold start), will initiate both events. Future activations of the action, if they're contemporaneous, will only generate execution events. This is the warm execution, which can take advantage of previous activations. To provide scaling, it is necessary to sometimes create new execution environments for the repeated actions, and so that means new initialization events.
While one cannot rely that all executions after a cold start are warm, it's possible to for the action code to check if it's warm and forgo internal initialization as needed. This is a convenient trick that's common to avoid expensive setup.
For Java actions running in OpenWhisk, this can be done with static fields.
It remains to be seen if the abstraction that Kubernetes adds to manager containers will have performance characteristics that serverless demands - considering the need for container reuse, reclamation, pause/unpause, and elastic scaling.
Perfect, thank you. Some of the docs I read made it out so that instances of my functions would never be reused. Or to put it better: I should not rely on them being reused.
If I keep DB connections for example am I now relying on an implementation detail or is this documented behavior? I realize that this will be different for every system.
My job is mainly developing on aarch64 server platforms. Does Fission work on aarch64 machines? If not, are there any plans to provide support? How much work is needed?
While native support is always better, this seems to be implemented similar to AWS Lambda sothe same approach should work there as well. You can have a husk of a nodejs app exec the precompiled binary of your go program. You'd just have to make sure the CPU arch matches up with the k8s infrastructure (i.e. it's going to be Linux x86-64).
Yes, this is very much on the roadmap. We're working on better support for compiled languages. Once we have that, adding a language should be a matter of creating a plugin.
I think it feels like a competition for celery. It fits into those tasks that are too heavy to run on the primary web server, but too small to set up the dedicated deployment/replicaset.
On slightly different note, I am still looking for a good job pattern for my use case. I run jobs on very custom docker images (including deep learning and computer vision stuff). I currently use redis queue with todo items, each pod in deployment picks up items from the queue in the loop, but I dislike some things about it. Fission is not the good fit, but I am wondering if HN community would have some suggestions. I thought about kubernetes jobs, but they have some things I don't like.
If you can model your data flow as pipelines that run when their inputs update, you might interested in Pachyderm (https://www.pachyderm.io/). This uses Kubernetes and Docker images to run multi-stage computations. It's still a bit rough around the edges. (I wouldn't recommend the current version if you work with multi-gigabyte binary input files, though they're currently working on fixing an issue affecting that use case.) They do have some examples (http://pachyderm.readthedocs.io/en/latest/examples/readme.ht...), including one using TensorFlow.
We're experimenting with Pachyderm at work, and the design feels "right" for our use case.
- As mentioned previously, retry count limit to job/pod would be useful.
- Minor nit: Deleting old job with the pod is cumbersome with current python API. Either delete_namespaced_job with orphan_dependencies: False, but I would need to check if job exists beforehand (deleting if job does not exist fails). On the other hand delete_collection_namespaced_job lets me delete job by label selector, but I also need to manually clean up the pod.
- Another nit: When exporting stackdriver logs collected by GKE from a job to big query, each log ends up as a separate logsource.
- Retry control is course grained. IIUC the only option are "no retry at all" or "terminate if job have been active for X seconds". If my jobs tend to be long and failure happens early, second option can lead to hundreds of retries. Each job retry is expensive, as it creates a new pod. I remember ending up with 100s pods.
Not the OP, but I'll chime in with what we're missing when we took a look at it. AFAICT there is no way to tell k8s to retry a job X number of times; you have to futz around with deadlines.
Job ordering/dependencies! Job1a must finish successfully before Job1b and Job1c.
Alternatively, all Jobs matching this query must succeed before this job. Failures can be OK if a configurable threshold of success is met. Exceeding the thresh fails the dependent jobs before they get to run.
You can do all of this with supervisor pods, but since you asked...
Redis itself is fine, but job queue around redis is finnicky.
On one hand, celery and python-rq do not work well big custom jobs (they are more suited for small tasks, but too big to run on the web server). On the other hand, using raw redis queue is finicky if I want to implement custom things like retries, timeouts, etc.
Your article mentions the reason you chose client code vs docker images, which I understand and can agree with. However, is there any plan to support docker images in place of client code?
What you're suggesting [initially] is very similar to what the folks at iron.io did but I'd really want to ship containers all around, not pieces of code. Iron.io addressed that a little too late in my opinion and their feature set around docker leaves a lot to be desired.
IMO, AWS Lambda and Google Cloud Functions are going to support docker in the near future, they have to if they want their ecosystem to be adoptable by all.
So we've started by focusing on improving the whole workflow, which is why we're operating at the source level. But we have occasionally heard the use case for working at the image level, and it has pros and cons, so we're thinking about it. I'd like to learn more about what you're trying to do -- did you consider just using Kubernetes Deployments and Services? Or are you saying you'd like to combine the on-demand start with working at the image level? Is this basically a resource optimization use case?
(not GP) I'm actually building something that would require running large amounts of containers of varying usage (some almost never used, some constantly queried) that serve up webapps and other network services, and being able to run services inside of containers instead of functions from fission would be awesome.
Isn't one of the big benefits of serverless functions being able to scale transparently? Can a kubernetes cluster give you anywhere near the elasticity of something like lambda?
By and large, yes. However, the responsiveness is highly variable. Both pods and nodes can automatically scale, though it is intended that this is somewhat gradual, by design (you don't want to spin up 4000 nodes because you have a 30 second spike). Because the functions tend to be _very_ small, you can (likely) pack many of them together without having to scale.
Even if it could spin up machines, you also need an auto-scaling database, and an autoscaling event infrastructure. otherwise it's not really serveless in any meaningful sense, since you'd then have to have separate, manually operated servers.
I'm a long time software developer, have recently implemented a series of web application servers on Heroku using Node.js, and I have no clue what this article is about. My suggestion is a better introduction might broaden it's readability.
I had the same problem with docker.com a few years back. I read through pages of "successful developer stories" and shit, and was sitting there like, "That's great and all, but what does it actually do?"
On the upside, this has led me to a good practice: When I do a website for one of my apps, the first few sentences will state in the utmost clarity what the software actually does.
It's for when you want the benefits of serverless computing for your engineers, but your organization is too big (too many requests) for lambda to be financially feasible. Or you project things getting to that point and don't want to be locked into the AWS ecosystem.
I was talking to a financial company about aws lambda and they wanted control over the underlying containers/VMs that is running the aws lambda, unfortunately AWS lambda doesn't give you that control and 'fission' does
Kubernetes gets you pretty far along the path of not messing with servers (especially if you use GKE). And we think there's other advantages to Functions-as-a-Service on Kubernetes -- it simplifies the dev workflow, improves server utilization, interoperates well with other services on K8s, etc.
> To optimize cold start overheads, Fission keeps a running pool of containers for each environment. When a request for a function comes in, Fission doesn't have to deploy a new container -- it just chooses one that's already running, copies the function into the container, loads it dynamically, and routes the request to that instance. The overhead of this process takes on the order of 100msec for NodeJS and Python functions.
100 ms of overhead is pretty steep. Google recommends a limit of twice that for the entire server response (https://developers.google.com/speed/docs/insights/Server#rec...):
> You should reduce your server response time under 200ms
Still, I've heard that AWS Lambda may run around 200 ms, so perhaps Fission is doing OK here.
But with modern progress in fast-starting compiled languages like Go and Rust, I'd love to have some easy way to supply small static binaries instead of source code. With compiled web server frameworks, it's feasible to handle over 250,000 (very simple) requests per second, or thousands of requests while Fission is trying to figure out where to send your source code. It doesn't always make sense to pay for interpreter startup overhead on every request (EDIT: this is not the case according to the Fission developers), especially not now that compiled code is so easy.