Every organisation in every sector is dealing with digital disruption in today’s volatile, fast changing and uncertain world.  Businesses need to transform to stay competitive or be in danger of going the way of once great brands like Nokia, Blockbuster or Kodak who saw the writing on the wall but didn’t act fast enough to adapt.  Too often digital transformation efforts fail with a regular cause being the organisation seeing the task as a project with a beginning and an end.  Those organisations leading in this era of disruption recognise that transformation needs to be continuous and businesses need to think of being in a permanent state of reinvention.  But often, the key barrier to change is the siloed nature of most organisations.  In today’s connected world, that needs to change, and we believe the way to solve the problem combines different thinking in terms of people, culture and architecture, as well as a new approach to systems integration, making use of the API Economy.

There is plenty of research exploring how business has evolved over the 19th and 20th centuries.  The various parts of a typical organisation often fail to work together with a shared sense of mission.   We would argue that the structural issue of divisions is a natural result of the command and control and hierarchical management approach of most, and particularly larger, organisations.  Most large companies have divisions, or even groups and functions within divisions, that operate in silos.  Even the word “division” itself highlights the problem.  People are, by their nature, territorial.  Those functional teams that grew with an objective of efficiency and process simplification in the beginning, have created issues around territory and mistrust that can derail the cross functional thinking and new ideas that are required in the 21st century of the Fourth Industrial Revolution.

In looking at the people and culture issues we are guided by Professor Vlatka Hlupic and the research behind her book The Management Shift.  She has investigated companies who have been tackling these big shifts over a number of years.  She references more than 20 companies using her approach and leadership model.  They are from small to large, in various sectors and include a FTSE 100 Company.  She has categorised their management styles in 5 stages or levels from Traditional to Emergent.  The traditional companies haven’t moved beyond command and control and silos.  The smart, successful companies have an emergent management style characterised by an unlimited mindset, strong team cohesion, unbounded culture, inspirational leaders, a strong sense of purpose, and a passion for the work.  These are the characteristics we need in our 21st century leaders and managers to break down barriers, encourage cross functional teams and foster the right mindset for collaboration rather than conflict.

We also think that the silo problem is a manifestation of Conway’s Law where organsations are constrained to design systems which are copies of the communication structures of that organisation.  We need to be thinking, communicating and doing things differently.

Given the ubiquity of IT in the way enterprises are organising their business, one cannot tackle breaking down silos in an organisation without addressing it at a technology level as well as people level.  Data silos are  the result of cultural, organisational and technical choices that were made long ago, either for strategic reasons or because of technical limitations. They reduce productivity, they make it difficult to have a global view of your business, and they make it difficult to leverage the new technology available today.

Over the past 10 years, the start-ups who managed to create new business models managed to do it because they leveraged the new technology available to them and did not have the legacy to deal with.  They could build everything from the ground up, at a speed unheard of before.  15 years ago, no organisation would have had the resources to develop geo-localisation, mobile apps, mobile payment, booking system, and scale as Uber did in such a short amount of time.  Equally they had no organisational barriers to deal with. They were purpose built and organised from the ground up. They had an idea and leveraged the cloud to pick and choose what functionality they needed to make this idea reality.

In order to stay current, to re-invent themselves and stay relevant to their ever more demanding clients, enterprises need to be agile and break down the silos that they built over the years. To achieve this, they need to be able to develop applications extremely rapidly, matching what the business needs, and ready to iterate to deliver fast.  However today, it is still considered that 50% of these projects fail because of integration issues.

Historically, integration teams have always been very centralised, being themselves one of the silos they should be contributing to break with integration technology. We are used to refer to the SOA team, or the ESB team. Integration was not something a developer would do on an ad-hoc basis, it was a full-time job, needing deep expertise in integration tools. This became very acute when the service-oriented architecture was put in place. It forced the creation of a centralised team to create the service layer that had to be used by the developers to developer their applications. The problem is that the integration team did not understand the application their services were created for and it created friction and finally slowed down the pace of developing new applications. It was clear that the best approach would have been to let the application team own the creation of the integration services, but technology did not allow that.

Over the past few years, new techniques have allowed us to re-think the way we tackle integration.  Let’s take a quick look at some new concepts and how they help moving towards a decentralized integration team.

Fine Grained Integration & Microservices

Breaking up your enterprise wide deployed integration hub into right sized containers provides improved agility, scalability, and resilience.  Agility, because many teams can work on integrations without having to defer to a centralised integration team. Scalability, because individual flows can be scaled on their own.  Resilience, because isolated flows cannot steal shared resources such as CPU from one to another.

Microservices are a software development technique that allows you to decompose an application into smaller de-coupled services. They provide greater agility because they can be changed independently from one to another, they are scalable because we can tie their usage with the resources they need, and they provide better overall application resilience because they are independent from one to another.

As we have seen, fine grained integration architecture and Microservices are providing similar advantages, and once brought together they bring the developers the environment they need to be fast, to be independent and to be able to concentrate on their part of the application.

APIs

API solutions have come a long way and today provide the tooling to be offered and consumed easily. They provide tools to be easily discoverable, they allow the provider to secure them and control the on-boarding of users.  They provide analytics so you can monitor them and control their usage, they can be promoted to third parties and they now can also be monetised.  The API economy is here, and the companies adopting the approach are more successful – research shows it adds more than 10% to the firm’s market value.

APIs therefore provide a very simple way for the provider to “offer” access to its data, and to the consumer to get to the service he or she needs. Based on a modern integration architecture, they are the key to unlock the data new technologies need to deliver on their premise. AI is only as good as the data it can be trained on. Innovative mobile application are only worth it if they allow the end user accessing and manipulating meaningful data.

The combination of Microservices consuming APIs to get to integration points can give an organisation great prospects in terms to speed and agility to respond to changing business needs.

People

We have seen that technology can change the way integration teams are organised, and give more autonomy to application developers.  But technology should also provide non-technical teams access to data. Take the example of the HR department that decided to subscribe to a Workday SaaS service. It is likely they did this without involving IT much – remember shadow IT – (at least during the choice of the solution). They did this because they wanted access to that application simply, without having to wait for a long IT development cycle, and were ready to adapt to get to the functionality. Now the HR department is using Workday and they need to access some specific information and want to receive an email alert when there is a specific change. For them, for simple integration requests like this, reverting to asking IT is out of the question. Modern integration tools should have “ready to use” connectors allowing them to perform no-code integration tasks.

Of course, technology used to create the silos we are trying to break. Over specialisation created barriers between the business and IT.  Within IT, it created barriers between integration specialists and developers, and it certainly didn’t facilitate communication between an enterprise and the “outside” world. Today, the need for data to fuel new technologies such as AI, Blockchain and other emerging technologies forces us to break down these barriers. And that’s what new technology and techniques allow us to do. It gives greater autonomy to the developer, it allows business users to be self-sufficient for their simpler needs, with a new level of controlled self-service thanks to APIs and the API Economy.

In summary, for today’s organisation to stay ahead of the competition it needs both a new mindset and a new approach to technology addressing architecture, technology and people. It needs more open leadership that recognises cross functional teams are necessary and better teamwork is required at all levels.  It needs a more agile approach to management as well as technology.  In terms of the technology deployed to support transformation, it needs to recognise that integration is the key driver, and the creation of APIs to open up company data reduces friction, drives new business models and creates new revenue opportunities.

Contact us if you want to find out more about making integration and APIs work for your business.

This post is a collaboration co-authored by Emmanuel Treny, Director Sales Europe – IBM Cloud Integration and David Terrar, Founder & CXO of Agile Elephant.