Sunday, August 7, 2016

Creating and article list (or Blog) with IBM WCM 8.0

For those of you finding youselves struggling a bit with IBM 's Web Content Manager (8.0) toys, and would like to see a nice and big example, I've posted something on DZone.

For a recent project I was tasked with creating a small Blog platform for IBM Websphere Portal 8.0. The requirements weren't for a full-blown blog, complete with user comments, but rather for a place where authors could create and share business-approved articles.

As this is not a trivial task if you do not know WCM, I decided to document how to do this via a series of five DZone articles. The reason for doing it on DZone is so that it can reach a bit more people. On this blog I'll just list the URLs to those articles if you're interested.

Part I sets the scene and creates the WCM Library;
Part II creates the article authoring and presentation templates, the site area for our articles and discusses Element tags.
Part III creates the list view for the articles.
Part IV adds some authoring tools for external authors.
Part V concludes the series by adding everyting to Portal pages.


Friday, February 19, 2016

The mindset of Agile: why we struggle in enterprises


As a developer I've wondered why we struggle so much to get along with business people in a classical enterprise or big corporate. With the whole "agile" movement, we are now supposed to work a lot closer. So why do we still struggle?

My suspicion is that we think fundamentally differently, like a right-brained artist versus a left-brained bean-counter. In this article I'm exploring these differences to try and discover how this affects agile adoption.

The Developer


We software developers are used to thinking in an agile way. The nature of our business is to think about both the big picture and sweat the details. We need to understand the business needs as well as how to realize it in software.

To be able to do our job, you need to keep on top of technology, and therefore learnt to learn really quickly. In my case, the downside of this is that my memory and attention span became shorter, in no small measure due to the volumes of information my brain is fed daily.

So, put a team of us together and you get a bunch of hyperactive kids who don't like sitting still and has the attention span of a 2-year-old. But we can get the job done, learn really quick, get kicks out of well-written code and new toys, enjoy cool UI design and have a drive to excellence (I hope).

What we struggle with is structure, meetings, lots of Excel spreadsheets or thick requirements docs.

The developer's idea of agile is to adapt often, learn and deliver quickly and to be a bit chaotic. The time-frame for this tend to be one to three weeks max before we lose interest.

The Business person


With a degree in, let's say, financial planning, banking or an MBA, the typical business person is trained in an industry and practices refined in the past 50 years or more. Efficiency, creating order from chaos and planning and the tooling that goes with it is a staple. You learn to follow the rules laid out by your industry. These rules do change, but typically only a small percentage of it, and only once a year.

Put a bunch of business people together and they hold meetings, create plans, hold road-shows, create documentation explaining rules, or listing financial plans.

So essentially generating contracts, and plans to execute these contracts. 

The business person may also use some software to assist them to assist a client or do their work, depending on their role or industry. In general though it is a linear process, well-known and oft-trodden.

What they struggle with is delivering value quickly, with a minimal plan, and postponing details until the latest. They struggle to not write too much and not plan too much.

A business person's idea of agile is to measure and adapt to client needs relatively quickly and to get to market faster than the competition. The time-span is measured in a few months or a financial quarter/year.

The mindset of Agile

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

To paraphrase the agile manifesto, which I believe is about behavior and a mindset and less about a SCRUM or SAFe framework, it is all about collaboration and teamwork, trust, accountability and fail-fast/learn-fast/deliver fast.

So, essentially it fits developers like a glove. Not so much the business person who was trained to do process, documentation, contract negotiation and following plans.

SCRUM and SAFe help mending bridges


SCRUM helps to create a little bit of order in the chaos that is software development and the developer's mind. It helps the business people to plan a little bit at least and understand (by virtue of visual elements like the board, burn-down and demos) where they're at and get a better grip on what they'll get.

SAFe does nothing for developers but lets higher-up business people feel like they're doing agile. Hopefully after a while they will actually be thinking agile and not just do it lip-service.

None of this makes any difference if business do not trust developers, developers don't trust each other, accountability isn't given where it should be and collaboration is still a once-a-quarter meeting.

SCRUM and SAFe then become marketing tools and status symbols instead of assisting an agile way of work.

The enterprise and HR

Often times in an enterprise, HR (and the enterprise as an entity) don't really understand or know what to do with this new way of work.

The company has ceremonies to award individuals and bonuses are given out to (those perceived to be) star players. Quarterly or yearly performance, or "360", reviews revolve around individual performance, and exclude or minimize the key values of teamwork and the agile mindset or expected culture. In essence the company then rewards an individualistic culture, while on paper it states that it is an agile culture.

I believe this is a result of business and HR following the world they know, which is indeed individual-based. HR themselves aren't necessarily schooled in the arts of being agile or the new "startup-style" of work.
Maybe they should also be included in the same SCRUM and SAFE training or agile bootcamps as the rest of us?

Conclusion

By understanding the way we think vs the way business and even HR or the enterprise as an entity thinks, we can be more aware and tolerant of each other and become "agile-ish" enough to satisfy both the chaos of software developent, and the realities that are Business.

In a future post I'm going to list some key points of research on exactly why agile (could) fail, what gets in the way of making it work and a road to get to agile nirvana.

Tuesday, January 5, 2016

IBM WCM 8.0 - virtual-portal-scoped actions

A challenge we came across when building our REST service for using WCM as a document library was: how do I target a virtual portal? WCM content libraries are created in a virtual portal. This means that, while you're working on the same VP as your content library, you can happily access WCM content via the Java API (see my example code in an earlier article). The problem comes in when you want to access content in a library on one VP, while your client code sits on another VP or are completely run outside of the Portal (as is the case in a REST client).

To make it possible to access content across virtual portals, IBM released a fixpack (CF13) that includes something called a VirtualPortalScopedAction and some additions to the Repository. This API comes standard with Portal 8.5, but for Portal 8.0/WCM 8.0 you need to upgrade to the right fixpack.

Diving in to a sample

I've created a Gist with some sample code, which is listed here.

The code uses the WCM Query API to fetch Category items (from the WCM API) on another virtual portal and library, using a hypothetical portlet. The main players in the code are:
  • VirtualPortalScopedAction interface to implement. In the run() method you can access the WCM API as if you are running in the correct virtual portal.
  • Repository.generateVPContextFromContextPath which is a CF13+ method to generate/get a context for a virtual portal different than the one you're currently on.
  • Repository.executeInVP(VirtualPortalContext, VirtualPortalScopedAction) executes your implementation of VirtualPortalScopedAction on the required virtual portal.

Monday, November 2, 2015

REST services best practice

There are different opinions out there on best practices for developing RESTful services. Projects have different needs, developers have different opinions and HTTP has its limitations just like most other technologies.


In this blog I'm listing some of the best practices and guidelines that I could find, sprinkled with my own opinion.

Key API requirements

According to Vinay Sahni's article, there are some key requirements for a good REST API.
  1. Use web standards when they make sense. Or the inverse, do not use (certain) web standards for your API if the don't make sense in your context.
  2. Your API should be friendly to the developer (your API's client) and be browser-explorable. 
  3. It should be simple, intuitive and consistent.
  4. It must be efficient and accomplish the task easily.
  5. Your REST API must match the domain you want to expose and be flexible to change.

API design best practices


Use plural nouns, not verbs: /books and not /getbooks

REST as an architectural style differs from SOAP webservices in that it works on resources, using http mechanisms like POST to indicate an action to be taken. Instead of the action being described by the API (/getbooks), the action is a combination of the resource (/books) and the http method (GET).

It is important to highlight that we use the plural form. It keeps the API consistent and simple, as you don't need to worry about pluralisation of resources.

JSON vs XML vs...

I use JSON when I'm expecting a web client to use the REST service, typically a javascript client.

XML works well for inter-system contracts or when you really need to be strict on the structure (you can also use JSON schema).

Use other return types based on your needs. Octet stream works for returning documents or images directly.

GET and query parameters should not alter state

You wouldn't have an API method called getBook(bookId) to alter the state of the book, so don't do it for your REST service either. GET and query parameters should not alter the state of the resource.

POST vs PUT

Use a POST when creating a new entry. A call to POST will, every time, create a new resource with a new identifier (if applicable).

PUT on the other hand creates a new entry at first, and updates it thereafter.

POST books/book (with appropriate form data or json)
PUT books/book/123 (note the addition of the ID)

Use sub-resources for relationships: /library/books/

When you need to show sub-relationships, build it in to the URI. The rule of thumb here is to not go deeper than 3 levels with this. If you need to go deeper, maybe create query aliases or provide more direct access to the lower-level resources.

Use HTTP headers to indicate serialization formats

Make use of Content-Type to indicate the (return) type of the content, and Accept to indicate what the client wants to accept.. Some developers modify the Content-Type to include versioning information.  shows how you can use Content-Type to indicate versioning (maybe for a minor version): application/myapp.v1+json.

Typical examples:
Accept: application/json;
Content-Type: text/plain

Use HATEOAS (hypermedia as the engine of application state) to assist navigation

Although not necessarily used everywhere, there are a lot of proponents for adding urls to your results to aid the client in navigating your resources. For example, URLs are added as part of your JSON to allow pagination or accessing related resources. Again, use web standards where they make sense.

Provide filtering, sorting, field selection and paging for collections

Make it easy for the client of your API to browse collections of resources.
  • filtering: filter resources on some field, for example GET /books?category=horror
  • sorting: sort resources in forward or reverse order, for example GET /books?sort=title (sort ascending on title) or GET /books?sort=-title (sort descending on title)
  • paging: if the collection is large, allow pagination using offset and limit, for example GET /books?offset=20&limit=20 to get the next 20 books, starting at book number 20. A good idea is to add the URIs (HATEOAS) for paging to navigate previous, next and so on.
  • field selection: For resources with lots of fields, allow the client to retrieve a subset of fields, for example GET /books?fields=title,isbn

Version your API in the URL: /myapi/v1/

It is seen as best practice to add a simple version in the URL so it's easy to browse your API in the browser between versions. You can also add the version number in the header (see Nazar Annagurban's article). Some APIs add the major version in the URL and the minor version, or a timestamp, in the header.

Sending multiple parameters (range of values)

When you need to specify a range of values in a query, you have two options:
  1. application/x-www-form-urlencoded standard: /books?titles[]=title1&titles[]=title2
  2. Widely used JAX-RS-happy standard: /books?titles=title1&titles=title2
So, don't do /books?titles=title1,title2 or something along those lines.

Handle errors with HTTP status codes and return results

When something happens, good or bad, use the HTTP status codes. Also return a payload listing the error. The returned 'error' JSON should be consistent across your API.

Some common uses of Http error codes are listed below.

200 OK - Response to a successful request. Use this whenever your GET, POST, DELETE etc was successful.

201 Created - Response to a POST that resulted in a successful creation. You need to also return a response body pointing to the unique ID or location (url) where the new item can be retrieved.

204 No Content - Response to a successful request that won't be returning a body (like a DELETE request or a GET query with no results). I've used this as well when no results were found in a search query.

304 Not Modified - This means the resource has not been modified since the version specified by the request headers If-Modified-Since or If-None-Match.

307/308 Temporary/Permanent Redirect - Use these if you rolled out a new version of your service and want the client to go there instead (older service is now depricated but still functioning).

400 Bad Request - The request is malformed or something on the client is perceived to be in error.

401 Unauthorized - Authentication is required by the client. After authentication, the request may be executed successfully.

403 Forbidden - This means the client was authenticated but not authorized.

404 Not Found - The requested resource couldn't be found (but may be found in the future).

405 Method Not Allowed - When e.g. a POST is sent to a resource that only supports GET.

410 Gone - See also 307/308: you can use this to indicate your service is now removed, not just deprecated.

415 Unsupported Media Type - If the client does a PUT or POST, with Content-Type that the service does not support. For example, you're posting xml while the service wants JSON.

422 Unprocessable Entity - Used for validation or semantic errors.

429 Too Many Requests - When a request is rejected due to rate limiting.

500 Internal Server Error – When your service has an internal error (some exception you can't handle gracefully), use this code. Your service should trace-log it out but send very little information to the client to avoid exploitation.

For more detail you can have a look at the wikipedia list of http status codes.

Conclusion

Although there are different opinions on the RESTful-ness of a service, the fact is you're OK if
  • You think Resources (plural);
  • You think Http standards (GET, PUT, Headers, Error codes, queries etc) to be the engine of your service and describe the actions;
  • You version your service.
And finally, don't overkill your solution if it's not required.



Friday, October 16, 2015

Using IBM WCM 8.0 Query API - sample code

For our project we needed to make use of IBM Web Content Manager as a 'lightweight' document store, or Library, so users can search for their documents using search criteria. To allow us to do this we create a prof of concept using the Query API.

For the purpose of this blog I'm assuming you have a good understanding of how to create WCM ContentItems, Authoring Templates and the like, and that you're familiar with the terminology. You must also have a licensed Portal 8.0 with WCM 8.0 installation to play with.

Accessing WCM content

IBM Web Content Manager 8.0 has different means of accessing WCM Content:
  • Via the WCM REST API (out-of-the-box): WCM comes with a REST API that gives you access to most of the concepts, including content items, site areas and the like. For WCM 8.0 though, no facility is available to access or search by Categories.
  • Using the Java API: The Java API allows a lot more flexibility, including searching by categories. It is also a lot faster than the 'chatty' REST service. It includes Workspace 'findxxx' methods, or for more advanced cases, a Query API.
  • Using a WCM Content Viewer Portlet with Presentation template: This solution works fine for 'normal' content on the site. We wanted a completely custom UI though, and also surface the items in other standalone applications.


Content (library) requirements for the POC

We created some sample content in WCM to showcase the following:
  1. Search for content by keywords, or
  2. Search for content by categories, or
  3. Search for content by name(s), then
  4. Search for or limit results based on custom elements (e.g. text element)

Demo portlet

To make it easy to play with, we created a JSR286 Portlet with a simple JSP, deployed as a WAR directly on a Portal 8.0 environment (same virtual portal/portal as the WCM Library).

The demo code listed at the end of the blog demonstrates the following usage of the WCM Query API and Workspace.
  1. Use a WCM Workspace for a specific logged in user;
  2. Limit search results to a given WCM Library;
  3. Search for Content (Items) by category(ies), keyword(s) or name(s) - these are built-in fields or profile abilities of ContentItems;
  4. Filter the results on a custom Element (a text element) value (the Query API doesn't seem to support searching on Element in Documents directly).


Getting the project to work in your environment

Content

Before creating the portlet, be sure to have some WCM ContentItems with keywords, categories and the like available in a WCM Library where the user has access.

WAR

You will need to create a WAR with a JSR286 Portlet, with the appropriate Portal WCM jar (ilwwcm-api.jar), which should come with your Portal installation and be available in RAD or eclipse with Portal 8 Development stubs. So, your project should either have the correct maven dependencies or the Portal 8 libraries attached.

Once set up, you can copy the code into your portlet java class and your own JSP, deploy it and test.

References


Code listing

You can get the code as Gist here as well.



Friday, September 25, 2015

Making a success of microservices

I've been reading quite a lot on microservices architecture, especially the team and organisational culture required for this approach to be a success. More specifically, I found it just as interesting to see when NOT to use this approach, than when it's appropriate. 

So without further ado, let's run through some findings and thoughts, followed up by reading material for your perusal.

Essential rules of thumb

In order for you, your team and your organisation to even think about, let alone succeed, with microservices, think about the following.

  1. Know when it's appropriate to apply it: microservices architecture isn't a one-size-fits-all solution and can actually hurt you, if you don't apply it with eyes wide open. Sometimes monoliths are quite appropriate;
  2. Know your team culture: your team need to be willing to do what it takes, and be patient. Microservices architecture ride heavily on team maturity, tooling and techniques to be able to deliver it effectively;
  3. Know your organisational culture: Conway's Law states "organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations". Essentially, your organisation's structure and culture should be able to, or willing to, give autonomy to teams and have less governance. The more red tape, the less likely microservices are to succeed; 
  4. Be willing to trial-and-error, and grow into it rather than big-bang: companies like Google, Amazon and more didn't start off with microservices as an architectural approach; they all started with a monolithic approach and grew into it, to solve real architectural and business challenges. 

Tips on your road to success

To get you on your way, here are some tips and tricks gathered from the experts (see the list at the bottom). 

Make sure it's a fit for your project(s)

Before you apply this architectural style to a project, make sure:
  • you're not applying it because it's the latest cool kid on the block. Sometimes a monolithic approach is a perfectly good solution.
  • it will meet your client's needs in terms of performance, functionality and effort required;
  • you have a real customer problem or business case that is hard to solve in a monolith, and apply microservices architecture there.


Team and culture

Your team must be willing to own a product (one or more microservices) top-to-bottom and be mature enough to be able to do the following.
  • Decide on the technology to use with little or no governance, and be autonomous as far as possible;
  • Have cross-cutting skills in the team, so the team can address from the database right through to the UI if need be;
  • Your team should either be willing to, be moving towards, or already be doing Continuous Deployment. Microservices imply you're going to have lots of them, so you really need to be efficient with how you develop, test and deploy them;
  • The team must be held (and want to be held) accountable for that product and own it right up to production support (DevOps).


Deploy, Refactor, Reuse Recycle

Microservices need to be boxed (see Domain Driven Design's bounded context) and simple to reason about.
  1. You must be willing to retire, add and modify them separately and often. This means you need to be on top of your versioning game;
  2. Reuse other teams' microservices before you build your own. This means strong collaboration between teams;
  3. Use Domain Driven Design, paying special attention to the Bounded Context principle, where appropriate, to make sure your microservices don't end up being semi-monoliths.

So, when does it make sense to use microservices as an architectural approach?

I found some tips from Adrian Trenman (Gilt) very informative. Use microservices as an architectural approach when
  1. you can isolate a piece of domain functionality that a single service can “own”;
  2. the service can fully own read and write access to its own data store;
  3. multiple teams are contributing to a monolithic system but keep on stepping on each other’s toes;
  4. you want to implement continuous deployment;
  5. you favor an emergent architecture rather than atop-down design.

Conclusion

The take-home from this article, if  you remember nothing else, is that you need to be mature as a team before embarking on the microservices journey. It requires trust from your team and organisation for it to be successful.
In addition, you also need to understand that it is a tool in the toolbox of architectures, and not a magic bullet. 
Lastly, don't rush it. Be OK with trial-and-error and make sure your organisation understands that.

References

Thank you to the following resources and contributors to those resources. I highly recommend you read up on these articles if you want to know more.

 InfoQ microservices architecture e-book where most of my info comes from. People like Eric Evans, Martin Fowler and Randy Shoup (plus some others from Gilt) were contributors.

From monolyth to microservices by Randy Shoup (slideshare).

InfoWorld's article on how to succeed with microservices.

SmartBear's article on what is microservices architecture.

Wednesday, September 16, 2015

How to JUnit test post-login code in Websphere Application Server 7+

Quite recently I wrote a WAS 7 utility that needed to access the JEE security Subject (read more on JEE Security) and its private credentials. The project, an auditing tool, accesses the Subject via IBM's WSSubject feature and uses some of its methods.


While trying to unit test it I ran into several problems, especially around trying to mock container-provided classes like Subject or LoginModules, and especially getting WSSubject to work.

Solving post-login unit test nightmares

When dealing with container-provided services like JAAS and the security model it can be very hard to write unit tests that exercise the code out-of-container, aka automated-build-tool-friendly tests. You can most probably start up an embedded container, but even so, setting up security to the point where it will populate your Subject can be tricky.

After a lot of research and trial-and-error, I created a test utility that allows you to:
  1. Let WSSubject return correct values for getRunAsSubject(), getCallerSubject() and getCallerPrincipal();
  2. Allow a unit test to add test private credentials to the Subject before testing a code unit that uses it;
  3. Allow a unit test to add a custom LoginModule and/or test custom LoginModules without the need of a container (not even a lightweight one);
  4. Allow the whole lot to run nicely from e.g. Jenkins in an automated build process.

Get to the code already

The samples contain 3x java classes:
  • A LoggedinTestContext with helpers to facilitate the login in out-of-container unit testing scenarios;
  • A MockLoginModule to show how to test JAAS Modules and/or add stuff to a Subject;
  • Sample unit test (LoggedinSubjectTest) to illustrate the LoggedinTestContext's usage and what it can provide.
The example code is not a full project as it depends on your own setup and is specific to your IBM WAS installation. The test data is also samples added as private credentials, but you can add anything your unit of code requires out of the Subject.

Dependencies needed

The code samples require you to (either in maven dependencies or elsewhere) have the following libraries available.

 WAS (7+) dependencies

All the jars below are relative to your WAS install root (e.g. IBM/Websphere/AppServer).
  • /lib/bootstrap.jar
  • /plugins/com.ibm.ws.runtime.jar
  • /runtimes/com.ibm.ws.admin.client_<version>.jar
  • /lib/j2ee.jar
  • /plugins/com.ibm.ws.emf.jar
  • /plugins/org.eclipse.emf.ecore.jar
  • /plugins/org.eclipse.emf.common.jar
  • /java/jre/lib/ibmcfw.jar
Where  <version> is your WAS version, e.g. 7.

JUnit dependencies

For the rest of the dependencies, the test code sample uses normal junit dependencies (junit mockito etc).

Code

I've added comments in the code so you should be able to follow along. I included the Gist listing in the blog, but you can it also get it from here.

Happy hunting!