I think I've said this before, but the caching advice:
> HTTP provides a built-in caching framework!
Conflicts with the authentication advice:
> each request should come with some sort authentication credentials
One of those gotchas in API design and implementation is how to provide good performance against authenticated resources.
My personal take on this is to not follow the standard HTTP caching patterns as this encourages the cache to be external - browser, proxy - to your authentication.
Instead I choose to cache behind my authentication layer, and this either means:
1. I use a plugin to Nginx or Varnish to call the authentication check before serving a cached resource.
or
2. I put my resources in a memory cache within the application.
The former has the advantage of being as HTTP pure as possible, but the disadvantage of maintaining the server extension/plugin.
The latter has the advantage of code maintainability (just the stuff in your app) but the disadvantage of not using existing tools for the caching.
Either way, if you are mixing authentication and caching the rule should always be that you cache behind your authentication layer and everything that passes through the authentication layer is explicitly marked as not-cacheable.
Awesome points; didn't know about the Link header (RFC5988).
Regarding pagination: a few years ago I switched to using "skip" and "limit", instead of "page" and "per_page" or a variation of the two. Instead of thinking about pages (an abstract idea) it forces me to think about the concrete list of objects being requested from the resource, how many of those I want and how many to skip from the start; making it easier to reason about pagination (at least for me).
I thought about this one when writing the post. Although I think the differences between the two are marginal, my conclusion was that it's "simpler" to increment a page number by 1 in code than incrementing by the page size.
Dealing with pages can get trickier if you allow the user to change the number displayed per page when not on the first page. It's not particularly hard to figure out, but what the right thing to do when the boundaries don't line up is a question that needs to be answered, as well, as well as dealing with user confusion about what page they are then on. Dealing with number ranges instead of pages solves these problems, at the cost of some readability.
A topic I rarely see (at least in depth, or in discussion) in stories/blogs about APIs is ID generation, i.e. best practices for creating access_token, resource ID and especially handling collisions.
The Flickr blog post [1], which featured a simple implementation of a ticket server, is so far one of the easiest and most secure to use.
Personally I don't feel comfortable using "hash of i++", "hash of ...", "i++", as they all "fall apart" when you need ids of different specifications (e.g. a 8-char ID, a 24-char token ID).
Anyone have two cents on this? I have a difficult time imagining that larger APIs just "live with" the chance of collision (even though it can be very low) - they must mitigate it somehow, right? combined probability (such as the nonce check and tokens on oauth1 requests)?
Actual ticket servers (e.g. [2], [3]) either introduce added complexity (and possibly latency) or needs a secondary code base. Regardless, the issue of scaling and latency is rarely touched upon, so if anyone have some input here, I'd greatly appreciate it.
EDIT: I've found two interesting links on the problem, [4] [5] (snippit'ed here [6]).
They start with a timestamp and are fully distributed. In the official linux client library, the "3 byte machine id" is the first 3 bytes of the md5 hash of the hostnames. As long as you can guarantee uniqueness there, there are no collisions.
For a large API, one could generate their own machine ID's and provide a strong guarantee on uniqueness.
The one criticism with MongoDB's approach is that the ID is 12 bytes and doesn't fit very well in column types offered by other DBs
My opinionated design principle for RESTful APIs is to keep in mind that REST is just a style sheet of good practices and patterns as in, do not apply it blindly and just use whatever makes sense.
From there, I have conflicting emotions with this:
Although the web generally works on HATEOAS type principles (where we go
to a website's front page and follow links based on what we see on the
page), I don't think we're ready for HATEOAS on APIs just yet.
I do not think "being ready" is an argument to use or not use HATEOAS. Sure, HATEOAS as defined by Roy Fielding contains a lot of clutter, but there's nothing bad if it makes sense to assume a client that does not know how to construct URLs. Generally I find it nice to have something similar to:
There's a sliding scale of HATEOAS-ness. I've seen a much more extreme form of it described elsewhere. Mark Seemann (http://blog.ploeh.dk/2013/05/01/rest-lesson-learned-avoid-ha...) describes an approach that encrypts all URLs so that clients have no choice but to follow links because generating URLs from templates is impossible.
And you'd have to get that URL out of the response to another request you'd already made. I never liked this idea, and the linked article provides a really strong counterargument to this approach to HATEOAS.
When browsing a website, decisions on what links will be
clicked are made at run time. However, with an API, decisions
as to what requests will be sent are made when the API
integration code is written, not at run time. Could the
decisions be deferred to run time? Sure, however, there isn't
much to gain going down that route as code would still not be
able to handle significant API changes without breaking.
About the counterargument, yes, it does raise a good point (in fact, one that I skimmed through). Specially about changes and breaking. For instance, in the following document, what is to change in the href attribute?
Surely, not much. Either /fooes/ changes to /foos/, in that case, the call to /fooes is broken anyway, or the id has changed, and needs to be handled with an HTTP 301 anyway. So no, there's not much to gain.
But then again, I was just saying that the href attribute might make sense in some cases, with not much extra effort. Nobody is asking the clients to consume the href attribute, it just makes understanding the whole API easier.
As of images, what would make more sense, to add an endpoint /images/10/view or directly use an href attribute pointing to http://s.foo.bar/1337.png ?
My point is, in some cases it makes sense, just use it when it does. Just as one should not be encrypting URLs just because Mark Seemann feels it is within the holy grail of the Level 3 API as defined by Leonard Richarson [1], we should not be avoiding to make our APIs navigable.
[1]: Don't get me wrong. What they say might make sense in non-trivial APIs, as always, REST is not an specification.
I think we spend too much time trying to craft beautiful, descriptive URLs, where we should be spending our time crafting beautiful, descriptive responses. We try to make the URL describe to the developer what is being returned, when in reality, this should be dictated by media types and link relations.
Take for instance the example above. Whether the URL is /images/10/view or /images/10.png, the developer has no idea what kind of resource that is without any media types or relations (the file extension != to the media type). In other words, you could use both URLs to return the same exact resource, and the client shouldn't care. You could even change back and forth between those URLs and the client should never flinch.
This is actually a good example. Say you are serving a static PNG file at /images/10.png, linked in your server response (i.e. in some href like above). Say you want to change things up to do some type of logging on how many times the image has been viewed, so you write some code, create a response at /images/10/view, log views, and return the image with an image/png media type. You just did a really big API change and the client didn't have to be changed.
Now if your client was dumber, and was crafting URLs as /images/{id}.png, you would break all of your clients if you did the above changes. This of course is really just the tip of the iceberg of what flexibility HATEOAS can bring.
His argument makes sense in the current world of very stupid clients (as in, fragile to changes) and non-standard formats.
Current clients are like those factory robots that can only do a fixed series of movements, it's not really worth it to add ARTags and computer vision to the system if the arm joints can't do any other movement any way.
But on the other hand, if we don't add those tags (or in the API world, use HATEOAS and standard data formats) now, how are those smarter robots ever going to be made?
What about adding a field, removing a field or updating the type of data stored on a field? HATEOAS doesn't offer anything for that. This is why I feel we need more experimentation, more standardization, more time before some practices stand out enough to consider them 'best practices'
Isn't HATEOAS more about just URL changes, though? It is about letting the responses provide available responses to the client. This requires a commitment to creating media types or using existing ones, along with using link relations. Not using REST correctly leads to such complicated best practices that recommend adding .json to a URL to get the JSON representation. You shouldn't have to go to the documentation to figure out have to craft that URL to get a representation of a resource.
Than maybe it's too soon to have any kind of best practices. We agree that more experimentation is needed, but when is that going to happen when everyone is building according to the REST-but-not-really best practices and the systems trying new things are toy services?
Mutations in the resource are addressed by changes in the media type. You're talking about it as if it was rocket science, while everything is very well addressed.
While the author of that post is advocating a level 3 API, you could still have unguessable URLs without any hypermedia involved. If the client and server shared a secret key, or if you used random UUIDs as resource IDs, then you could achieve the same thing.
This feels like a good place to get some opinions out there about an app I am working on.
Some team members are advocating a very dogmatic approach to our eventually public facing REST API design where each REST end point of /foo results in a response of a list of some ids; each url can then be hit to fully populate the list.
eg
/foo returns something like [1,2,3,4]
I then have to do 4 more http requests to /foo/1, /foo/2, /foo/3 and /foo/4.
Further, if a /foo/1 model contains for example a user, then /foo/1 would additionally contain userid of eg 101.
I would then have to do a /user/101 and ensure the foo model was populated.
This can mean a simple list page with 10 on screen list items can easily create 1 + 10 + 10 = 21 http requests at a minimum. This is starting to add a lot of code complexity to managing the lists for sorting/filtering/updating purposes and managing the async requests in our AngularJS application.
The way I have worked before is to build a single end point of /foo with a paginator offset, and then that would return the entire object in an HTTP request, eg
[
{fooid : 1, user : {id : 101, name 'bob'}, orders : [{id: A},{id : B}]},
{fooid : 2, user : {id : 102, name 'alison'}, orders : [{id: C},{id : D}]}
]
This approach is less RESTful, but presents fewer issues when writing the API consuming code and is much more responsive from a latency point of view.
What does HN think about these contrasting approaches?
I don't see why would the latter be any less RESTful than the former; as long as the "foos" still have their canonical URL, I don't see any constraint being bent or broken by having their representations be sent in another request/response.
In fact, I'd say the latter approach is actually more RESTful, since the former requires the client to build URLs, which breaks if the server changes them.
You could add a get parameter defining which resource you want to get (no values meaning just getting the id/href) and you put something like ?expand=user,orders if you want to get them in one query
21 requests is indeed too much to display 10 items I think
Personally I don't see why your approach makes it less RESTful. In fact, doing 21 requests instead of 1 seems like an extremely chatty application and reminds me of SELECT N+1 database problems.
I stumbled upon this post recently and I love it. It summarises very nicely my own feelings.
I went down the path of exploring HATEOS and Collection+JSON and it all started to feel like enterprise architecture style over complication for very little benefit.
I believe it's actually the other way around, where HATEOAS is the least "enterprisy" of the options. When I think of enterprise or over-complication, I think of a system where logic is so deeply embedded in the server and client that if the client or server changes in any way, everything breaks. REST was designed to better handle changes between a client and server. For instance, if you built your API to use something like Collection+JSON, then it could be instantly read and understood by any Collection+JSON client (much like in the way the HAL browser works [1]). You could change things, move things, even redo URLs, and the client will still work.
If, on the other hand, you don't conform and use HATEOAS, you have to go look at the documentation on how to build the URL to make a call to the API. Your client can never explore an API, but must be built to make certain calls to certain URLs that are built a certain way.
True REST with HATEOAS was what HTTP 1.1 was built around. It may take some extra work to implement, but it was designed to make communication over the web simpler (and it's worth the trouble). Unfortunately, this article is not really endorsing what REST is, but rather a very common RPC that has been popularized by Rails and labeled as REST.
Great ideas and tips. As I read through it, I found myself mentally saying "yep, did that" for most of the sections. One thing I disagree with was what to do when a new resource is created. I think the Location header is a great idea (which I'm not doing) but I also think returning the new entity in the body is a good thing in that it saves another request.
The new tips for me were the Link: header for pagination and using 422 Unprocessable Entity for validation errors. How did I overlook 422 before? I've been using 400 for validation problems, which I wasn't very happy with but didn't know what else to use.
of course 'dont use xml'; unless your data mainly looks like documents, needs to handle mixed content, need to represent richer types and/or if you already have a robust, full featured, stable XML toolchain.
XML gets a lot of flack, and I think part of it is that "enterprise" solutions have mis-used and abused it so much (and part of it might be that people started out with only DTDs for specifying schemas).
Today, I think it's obvious that xml shouldn't ever really be written by a human, it should be used for machine generated interchange of data, much as html -- markdown and rst are reasonable languages to write in -- html "requires" a gui/wysiwig editor -- just as it really isn't pleasant to work with eg: xml config files directly. Or ant-scripts. It just isn't. Yes, you can hide the ugly with a smart editor, but it's still pretty ugly.
Now, html might be pleasant compared to writing in Postscript -- but it's not really pleasant to write (hence things like sparkup).
I believe the issue that many people have with XML goes a bit deeper ... using XML means you are creating a bit of distance between yourself and the host programming language. XML stands alone, apart, that distance comes with costs in terms of ease of use (which could have been addressed by proper tooling), as well as many benefits.
HTML may not be pleasant to write documents with and I would say exactly the same thing of json ... show me the equiv of a Word document in json and I would argue that only a madman would like to directly edit that.
Really, James Clark said it right, 'json for data, xml for documents'.
Today's 'ease of use' of manipulating a native data structure of the 'language du jour' will eventually be viewed a bit more negatively in the years, decades to come. But really this is a storm in a tea cup ... XML has embraced json and has lots of support for generating it so I am fine. For others, they will eventually find the boundaries of json and start redeveloping the same concepts.
However, all the above being said/done I do think json is the right 'over the wire' format for data today; I just think we should embrace heterogeneity instead of 'knee jerk' cargo cult advice of 'don't use XML'; that's the equivalent of throwing tools out of one's toolshed and that's just plain stupid.
The strength of xml is xsl. The ability to declaratively transform the content into another form. I'm not aware of any way to do this with json other than by hand-rolling code.
I would say that the strength of XML is XPath - forget the rest of XSL (XSLT and XSL-FO) - and I would still avoid it unless there was a real requirement for "document like" content.
I'm not entirely certain that is a plus... Is writing a compiler/translator in xsl really more pleasant/easier to maintain than writing it in a "proper" language?
Following your examples, one of the ways of searching is by issuing a GET to /search?params.
The problem with this, is that is easily to reach the maximum size of the url. For example, if you want to know the data of your facebook frriends you will issue a GET /data?user[:fb_id]=123&user[:fb_id]=534...
There is a solution for this less ugly than using POST, is to sent data in the GET request. This is not forbidden by any RFC. Is just not so common. Most of the client libraries supports that.
That's Best Practices for Designing an HTTP API. Of 22 points, the author deals with HTTP implementation details in 14, and REST is supposed to be protocol agnostic and leverage on the standard. In 6 points he goes directly against the REST principles, and in 2 other points he directly contradicts them.
Sorry, but if you follow his guide, you won't have a RESTful API at all. I don't understand how can someone sell that as a "best practices" manual.
One thing the (otherwise thorough) article doesn't mention is whether to provide a machine-readable schema for the API or not. Some integrators do seem fond of being able to generate stub client code automatically ...
Is there a good way to declare a REST API and automatically have a cli created from it? Having a simple GUI would also be great but that seems like a much less likely chance.
We don't have enough standardization for that yet.
Old enterprisey API's had things like WSDL[1] which made auto-generated clients possible. But added their own overhead and complexity. REST API's tend to be much simpler and don't have anything like this at the moment.
In a standard CRUD application, PATCH should be used for "U"pdates. So you should be able to change attributes of a resource without having to send all parameters to it. It gets a little confusing when you think that most people use POST to create resources, and PUT to do updates (partial or not) and leave PATCH without too much practical use.
So, if you want HTTP purity, think of PATCH as "the verb to use for partial updates without having to send all parameters in the form".
We use for almost all representation mutations. When representations have automatically updating fields (for example: timestamps), it makes a lot of sense to use PATCH.
> HTTP provides a built-in caching framework!
Conflicts with the authentication advice:
> each request should come with some sort authentication credentials
One of those gotchas in API design and implementation is how to provide good performance against authenticated resources.
My personal take on this is to not follow the standard HTTP caching patterns as this encourages the cache to be external - browser, proxy - to your authentication.
Instead I choose to cache behind my authentication layer, and this either means:
1. I use a plugin to Nginx or Varnish to call the authentication check before serving a cached resource.
or
2. I put my resources in a memory cache within the application.
The former has the advantage of being as HTTP pure as possible, but the disadvantage of maintaining the server extension/plugin.
The latter has the advantage of code maintainability (just the stuff in your app) but the disadvantage of not using existing tools for the caching.
Either way, if you are mixing authentication and caching the rule should always be that you cache behind your authentication layer and everything that passes through the authentication layer is explicitly marked as not-cacheable.