Transitioning to a REST API in support of a modern front-end

What is this post?

This post identifies a growing technical challenge that some companies might experience after working with a large monolith (such as Rails or Django) that is tightly coupled to a front-end application (JavaScript).

I'll provide an overview of the problems you might be facing today, and what those problems might lead to. I'll cover the major pieces of the back-end-to-front-end relationship, and what each of them handles.

Finally, I'll talk about how you might fix these problems by moving to a front-end heavy web application (if that makes sense for your application), by implementing a common and consistent REST API on the back-end.

The role of the back-end

Let's assume Rails is your back-end application framework of choice. It's responsible for most of the heavy lifting and integration work of our product. It handles data:

• storage
• retrieval
• relationships
• caching

Your Rails back-end is the source of truth for the business-logic of your product. It defines terminology and behavior (what is an application? what is an organization?), and it's charged with processing and massaging that data into more useful data, such as graphs, aggregation, association, etc. It also provides a platform for processing data outside of the typical user-to-product relationship:

• Email delivery
• Transactional messaging
• Third-party associations: Twitter <-> applicationFacebook <-> application

In support of these many responsibilities, your Rails back-end might interface with several systems:

• Web browsers via HTML rendering and JSON endpoints (currently Rails resource endpoints)
• Our datastore via database connections
• Third-party APIs such as Zuora, NetSuite, SendGrid, Salesforce, etc.

The relationship between the Rails application and web browsers is overloaded. Delivering HTML, JavaScript, and resource data to the browser and then patching business logic via tightly-coupled JavaScript applications is what this post is intended to address.

The role of the front-end

The role of the front-end in a modern web application is to deliver the visual representation of your product, and its purpose, to users. It portrays how your application looks and how it works.

The components of a web front-end are:

• HTML rendering
• CSS styling
• JavaScript for application and business logic

The JavaScript part of your front-end is a piece that gets a large amount of attention, as significant amounts of business logic have been moving from the back-end to the front-end in recent years. Some examples of application and business logic that JavaScript might now be responsible for are:

• Code that combines resources (data) according to known relationships, and presents that information to the user.
• Input validation from a user before sending data to the back-end.
• State management: application viewsstate storage in URLs, local storage, etc.

Differences between back-end heavy and front-end heavy applications

When web applications were in their infancy, they tended to be back-end heavy. Servers would send HTML to a web browser, respond to user input, crunch some data and send back more HTML. This traditional request/response cycle was useful and necessary in the early days of the web. It had some benefits:

• Browsers had poor support for handling much more than the typical request/response cycle.
• Servers were more powerful than personal computers (this one is making a comeback).

There are still some significant benefits to writing back-end heavy applications today. They tend to shine best when users of your application will typically only do a small number of things during their visit. You can do all the back-end processing on a powerful server and deliver a single response with everything the user needs to do that one thing. Consider Wikipedia as an example here: the majority of visitors are landing on the page from a Google search, and it makes sense to deliver that page as fast as possible with no additional overhead than what's needed to display that page.

But with more technologically advanced browsers came an opportunity for application developers to build systems that could front-load processing on a user's device and only request smaller updates from a server incrementally, instead of making large requests on every single view. This shift drove the rapid development of front-end application frameworks such as jQuery, Backbone, Angular, Ember, and React.

Let's imagine two cycles: one back-end heavy and one front-end heavy:

Back-end heavy cycle

1. User visits a page that lists the comments on one of their blog posts.
2. The server looks at the request, sees the supplied cookie, and starts hitting the database: Who's this user? Get the user's blog post that matches this request / URL. Get the blog post's comments that match this result. Get any extra data you might need to render the page (say, user's organization, a count of all blog posts, etc). Do a templating cycle that renders a page based on any supplied templates (headers, footers, blog post, comments, etc). Compile the templates into a string. Send the HTML back to the browser.

This is all fine and well, until the user visits another view. The back-end heavy application has to do this all over again. This is computationally expensive both on the server and the client, and is a waste of resources.

Front-end heavy cycle

1. Do all of the things above, but include a heap of JavaScript that sets up an application that will then handle any future route changes via pushState.
2. When a route changes or an action happens on the page, the browser can ask the server "can I have this user's blog posts?", and the server can deliver just the blog posts, not a payload that requires an entire cycle of extraneous data, templating, compiling, etc.

Of course, this introduces a new layer: processing on the client. It means that the first time the page loads, the browser on the client now must do some processing to establish code that will handle future actions and route changes.

Why do this? If your users will typically visit more than one route or perform more than one action, you'll save measurable cycles on both the server and the client. This doesn't just save cycles for the current session - for the entire time the user is logged-in and has cached JavaScript, you benefit from this lightweight data-transport approach.

When you have an application that encourages users to visit multiple views or perform many actions, and you provide a back-end heavy implementation, you will eventually deal with performance bottlenecks that will eat engineering time. You can always layer on more caching, but it's far easier to cache raw data than entire request/response cycles and the complexities around that. This is why front-end heavy applications have become commonplace.

Where do you go from here?

There are short-term wins that you can start doing to help facilitate the switch to a front-end heavy application:

• Instate new front-end build tools (Grunt, Gulp, etc.)
• Separate the front-end from Rails. This means HTML, CSS, and JavaScript. Say goodbye to .erbtemplates.
• Use modern development practices: switch to ES6, adopt a stricter component architecture, and streamline tests.

In the long-term, you should hope to have clear separation between the Rails application and the front-end. This will allow you to hire specialists on both the front-end and the Rails side, and help you strengthen your codebase at a faster rate.

How do you change?

You change by discussing this post as an organization and proposing incremental actions. As mentioned above, you might start with:

• Improving the JS and CSS build process with tools such as Gulp.
• Supporting ES6 with Rails 4 + Sprockets (or similar workflows for other back-ends)

Your next steps are to identify exactly how you make technical and workflow changes, in addition to what you've already done.

A common, consistent, and scalable API.

What is the point of an API? It's an interface for data, which systems (and humans) can use. A good API is one which employs consistency, intelligible interfacing, and complete documentation.

A common interface is one which is guessable, across resources:

• /api/users
• /api/organizations
• /api/users/1
• /api/organizations/1
• /api/users/1/organizations

HATEOAS

We're all familiar with REST APIs, and HATEOAS has gotten a fair amount of coverage over the years. HATEOAS is a specification of a REST implementation that primarily differentiates how resources are represented and linked to via the system. HATEOAS is not an excuse for a lack of documentation, and it does provide usability benefits for both human and computer driven clients. It allows the server to define resource locations dynamically and seamlessly upgrade resource relationships. An example of a HATEOAS-style resource endpoint:

/api/users/1

{
id: 1,
username: 'nick',
organization: 'https://api.app.com/organizations/60',
applications: 'https://api.app.com/users/1/applications'
}

The endpoint provides enough information for us to subsequently access additional related resources, but doesn't make assumptions about what we want to do with the resource.

Endpoints could also filter via querystring:

/api/applications?user=1

[
{
id: 123
}
]

HATEOAS-style APIs can reduce computational load on the API server by reducing the number of relationships which must be calculated and retrieved on endpoints. It's up to the client to make subsequent requests to retrieve the data it needs to paint a complete picture of resources and their relationships. This means a greater number of requests from client-to-server, but ideally the lighter data load on the API (as far as relationships go) should allow us to more easily optimize the performance of individual endpoints.

A happy side effect for front-end developers building on top of a HATEOAS-style API is the ability to click through an API via standard web browser. Surprisingly this can make debugging even production systems much more pleasant (ie., share the resource URL via the API instead of a full UI).

Complex and non-traditional endpoints

While HATEOAS/REST specifications dictate that a strict coherence to resource structure must be followed, in the real-world there are sometimes edge cases for complex requests. Any divergence from the traditional REST structure makes the API less consistent, less discoverable, and more difficult to develop with. The edge case must present a strong-enough benefit to diverge (such as performance, security, etc), but sometimes that might happen.

Some examples of non-standard endpoints might include:

• /api/applications/1/usage-graph?from=01-01-2015
• /api/applications/1/recent-actions?include=users

Any endpoint that returns data that would not strictly be considered a "resource" would be a non-traditional endpoint and should be treated with care. As mentioned, exceptions can be made, but only when the trade-offs are clear and the endpoint is well documented.

Transactional endpoints

Sometimes it's necessary to build an endpoint that takes a request, performs some background action, and returns the result. An example of this might be uploading a user's public key via a PATCH to the endpoint:

PATCH /api/users/1

Request body:

{
ssh_key: 'ssh-dss AAAB'
}

You might want to take that key and store it on S3, perhaps. What happens if there's an error uploading the file to S3, though? Instead of having the endpoint respond immediately, one alternative is to respond after the upload is finished, but this introduces latency with the API and can create disconnects between the server and client (e.g. the user closes their browser). One potential solution is to respond with a request identifier that the server maintains and that both the client and server can work with later. An example response to our sample request above:

{
status: 'working',
message: 'Uploading key to S3.',
request: 'https://api.app.com/requests/12345'
}

This response can be instantly returned - even before we begin any background process. Later, the client can ask the server for information on the request:

/api/requests/12345

{
status: 'complete'.
message: 'Key uploaded successfully.'
}

This type of request handling is non-standard within the REST/HATEOAS spec, but again, sometimes exceptions must be made to ensure performance and data consistency between systems.

Authentication

With a completely separated front-end and back-end, authentication and authorization become the primary mechanisms for identifying who a user is and what they can do. With a typical Rails app, this is handled via cookies / session data stored on the server. With an API you need to authenticate first, then give that client a secret they can use later to identify themselves with every call. This is most often done via OAuth.

Mocking API endpoints

One of the primary benefits a common API provides to a front-end team is the ability to mock endpoints between the client and the server. This can be very useful during development, as the front-end team can build features against an API that will eventually exist.

It can also be helpful for the back-end team, in that they can see exactly what the front-end team has built a feature against. There are many libraries that exist to facilitate the creation of a mock API, and it is trivial to build one from scratch, as well. Being able to create mock endpoints is pointless, though, without a common and consistent API that exists for the production back-end.

Documentation

Along with a clean and consistent API comes a responsibility for writing equally clean and consistent documentation for that API. As mentioned, HATEOAS is not an excuse for a lack of documentation. An API is only as good as its documentation.

Every endpoint should be documented, clearly explaining the following:

• Acceptable request methods.
• Response format for each method.
• Available request parameters.
• Any special cases and/or caveats.

Additionally, each endpoint in the doc should provide examples of request/response pairs, and explain (generally) the usage and purpose of the endpoint.

Many types of API doc frameworks exist, but from a publication standpoint, GitHub's API documentation is an excellent example (and is open-source).

Why should we change?

Velocity

A strong separation of concerns allows teams to move faster through the use of improved documentation, clear boundaries, and better cross-team communication.

Testability

By cleanly separating the front-end from the back-end, each respective codebase can adopt best practices for testing with regards to build tools, testing platforms, etc.

Scalability

In a world where the front-end and the back-end are completely separate, systems engineers can more appropriately delegate resources for each system. Front-ends and back-ends have different hardware requirements, and keeping them separate allows us to more appropriately delegate those resources. It also makes fine-tuning performance easier - the front-end need not be burdened by the computational load of the back-end.

When you have them properly separated, the front-end can deliver a controlled experience to the user faster - and we can manage load-time expectations from the back-end much more quickly.

Hire-ability

Front-end and back-end engineering have been going through a specialization phase in the industry for a few years. Front-end engineers have increasingly grown fond of working only with JavaScript, interfacing with an API somewhere. Back-end engineers have begun taking roles exclusively building APIs for applications.

By separating the front-end from the back-end, we're not precluding full-stack engineers from working on either piece - you're making it easier to hire specialists and in return also lowering the barrier for engineers with cross-platform interests to work between the two.

Workflow changes post-API

Once you've developed a stronger separation between the back-end and the front-end, you'll begin to see shifts in what specific types of engineers are interested in working with.

Rails engineers can focus on data, business logic, storage, caching, tests, permissions, etc. Front-end engineers can focus on things such as routing, defining stricter view components, handling data/resources from the API, and more fine-grained tests based on the assumption of a common, consistent API.

From a project management perspective, resources may eventually become more specific. You'll likely have dedicated front-end engineers and dedicated back-end engineers working on their respective pieces. Project managers will need to re-allocate engineering resources according to how workflows eventually shift. Ideally, the separation of concerns between the front-end and the back-end will make individual engineers more effective and efficient with their work.