REST services best practice

There are different opinions out there on best practices for developing RESTful services. Projects have different needs, developers have different opinions and HTTP has its limitations just like most other technologies.


In this blog I'm listing some of the best practices and guidelines that I could find, sprinkled with my own opinion.

Key API requirements

According to Vinay Sahni's article, there are some key requirements for a good REST API.

1. Use web standards when they make sense. Or the inverse, do not use (certain) web standards for your API if the don't make sense in your context.
2. Your API should be friendly to the developer (your API's client) and be browser-explorable. 
3. It should be simple, intuitive and consistent.
4. It must be efficient and accomplish the task easily.
5. Your REST API must match the domain you want to expose and be flexible to change.

API design best practices

Use plural nouns, not verbs: /books and not /getbooks

REST as an architectural style differs from SOAP webservices in that it works on resources, using http mechanisms like POST to indicate an action to be taken. Instead of the action being described by the API (/getbooks), the action is a combination of the resource (/books) and the http method (GET).

It is important to highlight that we use the plural form. It keeps the API consistent and simple, as you don't need to worry about pluralisation of resources.

JSON vs XML vs...

I use JSON when I'm expecting a web client to use the REST service, typically a javascript client.

XML works well for inter-system contracts or when you really need to be strict on the structure (you can also use JSON schema).

Use other return types based on your needs. Octet stream works for returning documents or images directly.

GET and query parameters should not alter state

You wouldn't have an API method called getBook(bookId) to alter the state of the book, so don't do it for your REST service either. GET and query parameters should not alter the state of the resource.

POST vs PUT

Use a POST when creating a new entry. A call to POST will, every time, create a new resource with a new identifier (if applicable).

PUT on the other hand creates a new entry at first, and updates it thereafter.

POST books/book (with appropriate form data or json)
PUT books/book/123 (note the addition of the ID)

Use sub-resources for relationships: /library/books/

When you need to show sub-relationships, build it in to the URI. The rule of thumb here is to not go deeper than 3 levels with this. If you need to go deeper, maybe create query aliases or provide more direct access to the lower-level resources.

Use HTTP headers to indicate serialization formats

Make use of Content-Type to indicate the (return) type of the content, and Accept to indicate what the client wants to accept.. Some developers modify the Content-Type to include versioning information. Nazar Annagurban shows how you can use Content-Type to indicate versioning (maybe for a minor version): application/myapp.v1+json.

Typical examples:
Accept: application/json;
Content-Type: text/plain

Use HATEOAS (hypermedia as the engine of application state) to assist navigation

Although not necessarily used everywhere, there are a lot of proponents for adding urls to your results to aid the client in navigating your resources. For example, URLs are added as part of your JSON to allow pagination or accessing related resources. Again, use web standards where they make sense.

Provide filtering, sorting, field selection and paging for collections

Make it easy for the client of your API to browse collections of resources.

• filtering: filter resources on some field, for example GET /books?category=horror
• sorting: sort resources in forward or reverse order, for example GET /books?sort=title (sort ascending on title) or GET /books?sort=-title (sort descending on title)
• paging: if the collection is large, allow pagination using offset and limit, for example GET /books?offset=20&limit=20 to get the next 20 books, starting at book number 20. A good idea is to add the URIs (HATEOAS) for paging to navigate previous, next and so on.
• field selection: For resources with lots of fields, allow the client to retrieve a subset of fields, for example GET /books?fields=title,isbn

Version your API in the URL: /myapi/v1/

It is seen as best practice to add a simple version in the URL so it's easy to browse your API in the browser between versions. You can also add the version number in the header (see Nazar Annagurban's article). Some APIs add the major version in the URL and the minor version, or a timestamp, in the header.

Sending multiple parameters (range of values)

When you need to specify a range of values in a query, you have two options:

1. application/x-www-form-urlencoded standard: /books?titles[]=title1&titles[]=title2
2. Widely used JAX-RS-happy standard: /books?titles=title1&titles=title2

So, don't do /books?titles=title1,title2 or something along those lines.

Handle errors with HTTP status codes and return results

When something happens, good or bad, use the HTTP status codes. Also return a payload listing the error. The returned 'error' JSON should be consistent across your API.

Some common uses of Http error codes are listed below.

200 OK - Response to a successful request. Use this whenever your GET, POST, DELETE etc was successful.

201 Created - Response to a POST that resulted in a successful creation. You need to also return a response body pointing to the unique ID or location (url) where the new item can be retrieved.

204 No Content - Response to a successful request that won't be returning a body (like a DELETE request or a GET query with no results). I've used this as well when no results were found in a search query.

304 Not Modified - This means the resource has not been modified since the version specified by the request headers If-Modified-Since or If-None-Match.

307/308 Temporary/Permanent Redirect - Use these if you rolled out a new version of your service and want the client to go there instead (older service is now depricated but still functioning).

400 Bad Request - The request is malformed or something on the client is perceived to be in error.

401 Unauthorized - Authentication is required by the client. After authentication, the request may be executed successfully.

403 Forbidden - This means the client was authenticated but not authorized.

404 Not Found - The requested resource couldn't be found (but may be found in the future).

405 Method Not Allowed - When e.g. a POST is sent to a resource that only supports GET.

410 Gone - See also 307/308: you can use this to indicate your service is now removed, not just deprecated.

415 Unsupported Media Type - If the client does a PUT or POST, with Content-Type that the service does not support. For example, you're posting xml while the service wants JSON.

422 Unprocessable Entity - Used for validation or semantic errors.

429 Too Many Requests - When a request is rejected due to rate limiting.

500 Internal Server Error – When your service has an internal error (some exception you can't handle gracefully), use this code. Your service should trace-log it out but send very little information to the client to avoid exploitation.

For more detail you can have a look at the wikipedia list of http status codes.

Conclusion

Although there are different opinions on the RESTful-ness of a service, the fact is you're OK if

• You think Resources (plural);
• You think Http standards (GET, PUT, Headers, Error codes, queries etc) to be the engine of your service and describe the actions;
• You version your service.

And finally, don't overkill your solution if it's not required.

Using IBM WCM 8.0 Query API - sample code

For our project we needed to make use of IBM Web Content Manager as a 'lightweight' document store, or Library, so users can search for their documents using search criteria. To allow us to do this we create a prof of concept using the Query API.

For the purpose of this blog I'm assuming you have a good understanding of how to create WCM ContentItems, Authoring Templates and the like, and that you're familiar with the terminology. You must also have a licensed Portal 8.0 with WCM 8.0 installation to play with.

Accessing WCM content

IBM Web Content Manager 8.0 has different means of accessing WCM Content:

• Via the WCM REST API (out-of-the-box): WCM comes with a REST API that gives you access to most of the concepts, including content items, site areas and the like. For WCM 8.0 though, no facility is available to access or search by Categories.
• Using the Java API: The Java API allows a lot more flexibility, including searching by categories. It is also a lot faster than the 'chatty' REST service. It includes Workspace 'findxxx' methods, or for more advanced cases, a Query API.
• Using a WCM Content Viewer Portlet with Presentation template: This solution works fine for 'normal' content on the site. We wanted a completely custom UI though, and also surface the items in other standalone applications.

Content (library) requirements for the POC

We created some sample content in WCM to showcase the following:

1. Search for content by keywords, or
2. Search for content by categories, or
3. Search for content by name(s), then
4. Search for or limit results based on custom elements (e.g. text element)

Demo portlet

To make it easy to play with, we created a JSR286 Portlet with a simple JSP, deployed as a WAR directly on a Portal 8.0 environment (same virtual portal/portal as the WCM Library).

The demo code listed at the end of the blog demonstrates the following usage of the WCM Query API and Workspace.

1. Use a WCM Workspace for a specific logged in user;
2. Limit search results to a given WCM Library;
3. Search for Content (Items) by category(ies), keyword(s) or name(s) - these are built-in fields or profile abilities of ContentItems;
4. Filter the results on a custom Element (a text element) value (the Query API doesn't seem to support searching on Element in Documents directly).

Getting the project to work in your environment

Content

Before creating the portlet, be sure to have some WCM ContentItems with keywords, categories and the like available in a WCM Library where the user has access.

WAR

You will need to create a WAR with a JSR286 Portlet, with the appropriate Portal WCM jar (ilwwcm-api.jar), which should come with your Portal installation and be available in RAD or eclipse with Portal 8 Development stubs. So, your project should either have the correct maven dependencies or the Portal 8 libraries attached.

Once set up, you can copy the code into your portlet java class and your own JSP, deploy it and test.

References

WCM Query API:

http://www-10.lotus.com/ldd/portalwiki.nsf/dx/WCM_Query_API

Using the more base Workspace find facilities (code recipes): 

http://saranshibmblog.blogspot.co.za/2012/12/wcm-api-document.html

Code listing

You can get the code as Gist here as well.

Making a success of microservices

I've been reading quite a lot on microservices architecture, especially the team and organisational culture required for this approach to be a success. More specifically, I found it just as interesting to see when NOT to use this approach, than when it's appropriate. 

So without further ado, let's run through some findings and thoughts, followed up by reading material for your perusal.

Essential rules of thumb

In order for you, your team and your organisation to even think about, let alone succeed, with microservices, think about the following.

1. Know when it's appropriate to apply it: microservices architecture isn't a one-size-fits-all solution and can actually hurt you, if you don't apply it with eyes wide open. Sometimes monoliths are quite appropriate;
2. Know your team culture: your team need to be willing to do what it takes, and be patient. Microservices architecture ride heavily on team maturity, tooling and techniques to be able to deliver it effectively;
3. Know your organisational culture: Conway's Law states "organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations". Essentially, your organisation's structure and culture should be able to, or willing to, give autonomy to teams and have less governance. The more red tape, the less likely microservices are to succeed; 
4. Be willing to trial-and-error, and grow into it rather than big-bang: companies like Google, Amazon and more didn't start off with microservices as an architectural approach; they all started with a monolithic approach and grew into it, to solve real architectural and business challenges. 

Tips on your road to success

To get you on your way, here are some tips and tricks gathered from the experts (see the list at the bottom). 

Make sure it's a fit for your project(s)

Before you apply this architectural style to a project, make sure:

• you're not applying it because it's the latest cool kid on the block. Sometimes a monolithic approach is a perfectly good solution.
• it will meet your client's needs in terms of performance, functionality and effort required;
• you have a real customer problem or business case that is hard to solve in a monolith, and apply microservices architecture there.

Team and culture

Your team must be willing to own a product (one or more microservices) top-to-bottom and be mature enough to be able to do the following.

• Decide on the technology to use with little or no governance, and be autonomous as far as possible;
• Have cross-cutting skills in the team, so the team can address from the database right through to the UI if need be;
• Your team should either be willing to, be moving towards, or already be doing Continuous Deployment. Microservices imply you're going to have lots of them, so you really need to be efficient with how you develop, test and deploy them;
• The team must be held (and want to be held) accountable for that product and own it right up to production support (DevOps).

Deploy, Refactor, Reuse Recycle

Microservices need to be boxed (see Domain Driven Design's bounded context) and simple to reason about.

1. You must be willing to retire, add and modify them separately and often. This means you need to be on top of your versioning game;
2. Reuse other teams' microservices before you build your own. This means strong collaboration between teams;
3. Use Domain Driven Design, paying special attention to the Bounded Context principle, where appropriate, to make sure your microservices don't end up being semi-monoliths.

So, when does it make sense to use microservices as an architectural approach?

I found some tips from Adrian Trenman (Gilt) very informative. Use microservices as an architectural approach when

1. you can isolate a piece of domain functionality that a single service can “own”;
2. the service can fully own read and write access to its own data store;
3. multiple teams are contributing to a monolithic system but keep on stepping on each other’s toes;
4. you want to implement continuous deployment;
5. you favor an emergent architecture rather than atop-down design.

Conclusion

The take-home from this article, if  you remember nothing else, is that you need to be mature as a team before embarking on the microservices journey. It requires trust from your team and organisation for it to be successful.

In addition, you also need to understand that it is a tool in the toolbox of architectures, and not a magic bullet. 

Lastly, don't rush it. Be OK with trial-and-error and make sure your organisation understands that.

References

Thank you to the following resources and contributors to those resources. I highly recommend you read up on these articles if you want to know more.

 InfoQ microservices architecture e-book where most of my info comes from. People like Eric Evans, Martin Fowler and Randy Shoup (plus some others from Gilt) were contributors.

From monolyth to microservices by Randy Shoup (slideshare).

InfoWorld's article on how to succeed with microservices.

SmartBear's article on what is microservices architecture.

How to JUnit test post-login code in Websphere Application Server 7+

Quite recently I wrote a WAS 7 utility that needed to access the JEE security Subject (read more on JEE Security) and its private credentials. The project, an auditing tool, accesses the Subject via IBM's WSSubject feature and uses some of its methods.

While trying to unit test it I ran into several problems, especially around trying to mock container-provided classes like Subject or LoginModules, and especially getting WSSubject to work.

Solving post-login unit test nightmares

When dealing with container-provided services like JAAS and the security model it can be very hard to write unit tests that exercise the code out-of-container, aka automated-build-tool-friendly tests. You can most probably start up an embedded container, but even so, setting up security to the point where it will populate your Subject can be tricky.

After a lot of research and trial-and-error, I created a test utility that allows you to:

1. Let WSSubject return correct values for getRunAsSubject(), getCallerSubject() and getCallerPrincipal();
2. Allow a unit test to add test private credentials to the Subject before testing a code unit that uses it;
3. Allow a unit test to add a custom LoginModule and/or test custom LoginModules without the need of a container (not even a lightweight one);
4. Allow the whole lot to run nicely from e.g. Jenkins in an automated build process.

Get to the code already

The samples contain 3x java classes:

• A LoggedinTestContext with helpers to facilitate the login in out-of-container unit testing scenarios;
• A MockLoginModule to show how to test JAAS Modules and/or add stuff to a Subject;
• Sample unit test (LoggedinSubjectTest) to illustrate the LoggedinTestContext's usage and what it can provide.

The example code is not a full project as it depends on your own setup and is specific to your IBM WAS installation. The test data is also samples added as private credentials, but you can add anything your unit of code requires out of the Subject.

Dependencies needed

The code samples require you to (either in maven dependencies or elsewhere) have the following libraries available.

 WAS (7+) dependencies

All the jars below are relative to your WAS install root (e.g. IBM/Websphere/AppServer).

• /lib/bootstrap.jar
• /plugins/com.ibm.ws.runtime.jar
• /runtimes/com.ibm.ws.admin.client_<version>.jar
• /lib/j2ee.jar
• /plugins/com.ibm.ws.emf.jar
• /plugins/org.eclipse.emf.ecore.jar
• /plugins/org.eclipse.emf.common.jar
• /java/jre/lib/ibmcfw.jar

Where  <version> is your WAS version, e.g. 7.

JUnit dependencies

For the rest of the dependencies, the test code sample uses normal junit dependencies (junit mockito etc).

Code

I've added comments in the code so you should be able to follow along. I included the Gist listing in the blog, but you can it also get it from here.

Happy hunting!

Make a JSR 286 Portlet work with AngularJS in WS Portal

In an ideal world, you would create a web application (an angularjs app for example) that connects to some REST services written by either yourself or some other team. No need to worry about other UIs rendering right next to you (*cough* Portal *cough*).

Portlet developers, however, have interesting challenges not often encountered by most web and java application developers. We know we can use JSP, JSF or some other server-side framework quite easily out of the box; we also know we can add some javascript (not a lot though :)). Things get more hairy if you're running EJBs or other non-REST services, need to have a responsive (async) ui and all the modern bells-and-whistles while still playing nicely with other portlets on the page.

In this article I'm going to demonstrate how to create a simple portlet, add angularJS, and make an ajax call to post some JSON to our portlet. We will therefore use our portlet as a simple REST-like service so we can call in to our backend.

AngularJS

AngularJS needs no introduction, but if you're new to it have a look at the site. It allows you to create some really cool single-page applications (SPAs) but can definitely also be used instead of JSF or custom-rolled js to build your portlet UI.

IBM Websphere Portal Script Portlets

A note on IBM Websphere Portal script portlets: script portlets essentially wraps your javascript in a portlet. This is relatively new, so please have a look here if you're interested.

Sample project ws-angular-portlet

I have created an example portlet WAR you can clone in github from here. Please have a look, as I'm going to use snippets from there to illustrate the key points. You can also use it as a starting point for your portlet application.

Note that the sample project does not contain the maven repositories needed to get the artifacts listed; this is because of the proprietary nature of Websphere Portal. Please add your own repos in the pom.xml.

Let's dig in

Make sure you've either copied the sample code, or have your project/portlet open with a javax.portlet.GenericPortlet and the relevant portlet.xml.

Handle Requests and Responses in the Portlet

 private ObjectMapper objectMapper = new ObjectMapper();  
      /**  
       * Simply forward to the JSP to bootstrap angularjs.  
       */  
      @Override  
      protected void doView(RenderRequest request, RenderResponse response) throws PortletException, IOException {            
           PortletRequestDispatcher dispatcher = getPortletConfig().getPortletContext().getRequestDispatcher("/WEB-INF/jsp/AngularPortlet.jsp");  
           dispatcher.forward(request, response);  
      }  
      /**  
       * Receives a JSON String (jsondata) that gets parsed into a codingglass.portlet.Request object.  
       * For illustration, this method then 'echoes' the request information in a codingglass.portlet.Response object as JSON.  
       */  
      @Override  
      public void serveResource(ResourceRequest request, ResourceResponse response) throws PortletException, IOException {  
           Request fromJson = objectMapper.readValue(request.getParameter("jsondata"), Request.class);  
           Response responseToClient = new Response("OK", "Call OK for ID " + fromJson.getId() + " with data " + fromJson.getData());            
           objectMapper.writeValue(response.getPortletOutputStream(), responseToClient);            
      }       

1. In the doView, point to the jsp that will bootstrap your angular (code to follow);
2. Override the serveResource() method in the portlet to handle ajax calls.
3. For the incoming request, mapped to our custom parameter 'jsondata', parse it into a java object (Request.class in this case) using Jackson.
4. Create the Response object, passed via Jackson to the response outputStream as JSON.

Create the JSP

Create the portlet JSP (as referenced from your portlet class) as follows.

 <%@ page contentType="text/html" pageEncoding="UTF-8" language="java"%>  
 <%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>  
 <%@ taglib prefix="portlet" uri="http://java.sun.com/portlet_2_0"%>  
 <portlet:defineObjects/>  
 <link rel="stylesheet" href="<%=request.getContextPath()%>/resources/css/app.css">   
 <!-- html container for your angularjs application. This container will be manually bootstrapped, as ng-app does not work for > 1 app  
 on a page  
  -->  
 <div id="<portlet:namespace/>angularjsApp">  
 <h1>Example AngularJS app</h1>  
      <call-portlet></call-portlet>  
 </div>  
 <!-- Create portlet/Portal constants (urls etc) to be used in the angular app via injection. -->  
 <c:set var="angularPortlet" scope="request">  
   <portlet:namespace />Portlet  
 </c:set>  
 <script type="text/javascript">  
      var ${angularPortlet} = (function() {  
                var portal = {};  
                portal.portletName = "${portletName}";  
                portal.portletNamespace = "<portlet:namespace/>";  
                portal.resourceURL = "<portlet:resourceURL/>";                 
                portal.contextPath = "<c:url value='/'/>";                 
                return portal;  
        })();         
 </script>  
 <script src="<%=request.getContextPath()%>/resources/js/jquery-1.11.3.min.js"></script>  
 <script src="<%=request.getContextPath()%>/resources/js/angular.min.js"></script>  
 <script src="<%=request.getContextPath()%>/resources/js/app.js"></script>  
 <!-- Manually bootstrap angularjs application. -->  
 <script type="text/javascript">  
           angular.module('angularjs-App').constant("PORTAL", ${angularPortlet});            
           angular.element(document).ready(function() {  
         angular.bootstrap(document.getElementById("<portlet:namespace/>angularjsApp"), ['angularjs-App']);   
       });                  
 </script>  

1. After the normal scaffolding taglibs, define a div to be used as the application container, using the portlet:namespace tag to make it unique;
2. Add our directive (can be angularJS view or whatever) inside the div;
3. Set the portlet name as a variable so we can use it in our js;
4. Create the constants to be used in our angular application. The constants include server-side-rendered URLs to the serverResource and other entry points;
5. Manually bootstrap the angular application (ng-app doesn't work for more than one portlet on the page).

Create the angular application

Our example project consists of a single input box with a button. You type in what you want to send to the portlet and the result is echoed next to the button.

Disclaimer: for large apps you typically break down your controllers, services, directives etc into separate files; I grouped it for simplicity.

 angular.module('angularjs-App', [])  
 /**  
  * A simple service that'll call the portlet serveResource with our data.  
  * Note the setting of the Headers and data as parameters: this is required to make sure your data ends up in the resource request getParameter()  
  */  
 .factory('callPortletBackend', ['$http', 'PORTAL', function($http, PORTAL) {  
      return {  
           /**  
            * @param {String} an ID you want to pass to the portlet. This allows the portlet to handle >1 types of requests.  
            * @param {String} the data you want to pass in as a js object.  
            * @param {Object} The callback function once call is completed. The function takes in the result object etc (see angularjs $http).  
            */  
           submitToPortlet: function(id, data, handler) {                 
                var objectToSubmit = {id: id, data: data};  
                console.log('calling portlet with data ' + data);  
                $http({  
                     method: 'POST',  
                     url: PORTAL.resourceURL,  
                     headers: {'Content-Type': 'application/x-www-form-urlencoded'},  
                     data: $.param({                           
                          'jsondata': JSON.stringify(objectToSubmit)  
                     })  
                }).then(handler);  
           }  
      }  
 }])  
 /**  
  * Simple directive with input text and a button, to submit some stuff to the portlet.  
  */  
 .directive('callPortlet', ['callPortletBackend', 'PORTAL', function(callPortletBackend, PORTAL) {  
      return {  
           templateUrl: PORTAL.contextPath + '/resources/html/templates/callportlet.html',  
           restrict: 'E',  
           scope: false,  
           bindToController: true,  
           controllerAs: 'callportletCtrl',  
           controller: function() {  
                var thisCtrl = this;  
                thisCtrl.resultMessage = 'Nothing yet...';  
                thisCtrl.submitToPortlet = function() {  
                     callPortletBackend.submitToPortlet('myData', thisCtrl.dataToSubmit, function(result) {                                                    
                          thisCtrl.resultMessage = result.data;  
                     })  
                };  
           }  
      }  
 }]);  

1. Create a service that calls the portlet's serveResource (resourceURL), passing in the data as JSON as a url parameter called 'jsondata'. Note how the PORTAL constant is used to get to the resourceURL.
2. Create a directive that uses the service to post whatever you typed into the input (the template can be found here)
3. Note the use of PORTAL.contextPath to load the directive's template from the portlet project serverside.

Deploy and test

That is it! The angular $http service will call the portlet serveResource with some JSON and display the JSON results back to the user. Forgive the styling.

Conclusion

We demonstrated how to create a portlet, running in (at least) Websphere Portal 8.x with an angularjs UI, using the portlet serveResource to make AJAX calls.

If you're running Liferay, please have a look at this project from planetsizebrain.

My first post: what can I expect from this blog?

Welcome

Welcome to my new blog and also my first post! My name is Roan Bester, I work for a company called Dynamic Visual Technologies (DVT) doing lots of different software development projects.

I'm starting a blog that looks at coding and software development differently. Differently in the sense that it will include stuff that's hard to find by Googling it, making sense of sometimes hard-to-read Wikis and generally de-mystifying programming.

What you can expect

I've been busy doing projects in IBM Websphere (WAS. Portal), and as many of you know it's hard to find a good post or blog that can help you to get the best out of a licensed product. So, my aim will be to share some tips, tricks and my experiences in this regard, sprinkled with more 'general' programming posts.

Tech you can expect to find here

I do a pretty broad range of development so you can expect some exciting articles!

1. IBM Websphere Application Server
2. IBM Websphere Portal 8.x
3. General Java and JEE (5+)
4. HTML, CSS
5. Javascript including AngularJS, NodeJS and more.

What not to expect

There are a lot of very smart people blogging and writing articles on specific topics, for example Java 8 features, that I won't necessarily go into in detail. Same goes for deep AngularJS or deep NodeJS as I'm not a specialist in these areas.

I will, however, be sure to reference the good stuff when needed.

So what's next?

Please bear with me as I write my first 'real' blog entry: Make a JSR 286 Portlet work with AngularJS in WS Portal.

I hope to add value to your life soon!