Does your company have the right technology platform for a successful data transformation? At first glance, this seems like an innocent enough question. Surely if you have a modern data platform comprising best of breed technologies, then the answer is a resounding ‘yes’.

In actual fact the answer isn’t quite that simple. Technology is just a tool after all, and for any tool to provide value, you need the skills to use it, and it needs to be right tool for the job (which means that you also have to know what that job is). Having a DEWALT DW729KN does not make someone a master carpenter, and conversely, an experienced carpenter with the right tools and skills still needs to know what to make before they can create something valuable.

It’s the same story with data transformation – before you even start thinking about what technology platform you need, you must make sure that you understand the job that platform has to do. Ideally, you need a data strategy which first sets out the roadmap for how data can help achieve your goals and objectives, and then describes the role that technology needs to play. At a minimum, you need a set of business goals and objectives as your starting point. In any event, you should avoid driving your data transformation journey from a pure technology angle – there are too many examples of organisations who have implemented excellent data platforms without a clear view of the role the data needs to play. The result is a delayed – or worse, a failed – data transformation journey, with no value to show for the effort put in.

Once you have worked out what you want to achieve with data and why you want to achieve it, you are finally ready to start thinking about technology. But how do you pick the right platform to make sure you succeed? In the third of Ascent’s Data Conversation webinar series, I spoke to Alex James, Ascent’s Chief Technology Officer, to look at the key do’s and don’ts in this area, and signpost the path to success.

Where to start: those all-important initial decisions.

The initial decisions you need to make revolve around the key points in the data lifecycle. What technologies are you going to use to acquire data, transform and store it, analyse it and finally serve up information and insight derived from that analysis?

In the last few years, there has been a move towards cloud-based data technologies (think Azure, AWS and others) offering increasingly similar capabilities through commoditised ‘as a service’ solutions. So, deciding which platform to use is no longer about finding the one that can do what you need - usually all of them can.

Instead, you need to focus on your existing technology ecosystem, and look at the potential points of integration with the data platform, like extracting data from key operational systems.

Once you have this big picture view of the data platform and its role, you can then start deciding on the approach to take at each integration point, asking questions like:

• How can we make best use of the technology we already have and so minimise spend on anything new? How can we make sure everything will fit together – legacy and new? Should we integrate with, replace or modernise components in the existing ecosystem?

• If we have multiple sources for the same data, why is this? Can we combine or remove any sources? Can we get rid of any operational silos?

• How can we shorten the path to value? What initial objectives should we focus on, and what slice of the data platform do we need to implement for data to be able to help achieve them?

• Are we set up to develop our technology platform in an agile, iterative way, ensuring regular delivery of value and ROI and making sure we don’t spend a lot of time and money building the wrong thing?

Focusing on integration instead of capability in this way helps make sure that the key technology decisions that you make are more likely to be the right ones (in other words, that you are picking the right tools for the job).

How to start: moving from decision to action.

Once you’ve decided on the technologies you’re going to use, how do you turn those decisions into reality?

Historically you would probably have spun up a team of software or platform engineers to implement the technology platform, then let your data engineers loose on the platform to start moving data through it from source to target. Each team would have their own distinct jobs to do, with project management oversight making sure that everything came together successfully.

This siloed approach to developing technology platforms and data capabilities sounds reasonable on paper, but in practice it has several drawbacks. Each team only feels responsible for its own part of the whole. The relationship between the teams is that of customer and supplier, which naturally leads to a non-collaborative way of working. Even worse, it’s all too easy for one team to blame the other when things inevitably don’t work out right the first time. Siloing teams in this way turns out to be one of the biggest reasons why the technology aspects of data transformation fail to deliver the value they should.

The trend now is to have a single team to develop technology platforms and data capabilities comprising the complete set of roles needed to successfully implement the technologies and then push data through them to deliver information, insight and value. These teams typically include software engineers, data engineers, data analysts and scientists, and business analysts.

Having all these different disciplines working together towards a common goal, with each person understanding how their role fits into the bigger picture, solves the drawbacks of the siloed approach. Everyone understands why they are doing what they are doing, everyone feels a responsibility for the success of the whole solution, and a collaborative way of working becomes the natural approach.

How to avoid a governance nightmare along the way: encouraging data citizenship.

If you get the technology platform right, then you open up a world of data capability that enables everyone in your organisation to more easily use data and insights to achieve their goals and objectives. Low code, no code, self-serve and data democratisation all play their part in removing the historical barriers to data use.

But there is a cost. More data capability leads to more data, and more data requires more data governance. As organisations improve their technology platforms, they move from having data they can’t govern (for example, information in a locally produced spreadsheet shared through email) to data they can govern (for example, information in reports and dashboards shared online). Ultimately this is beneficial, of course, as the end state is a set of governed information and insight resulting in more confident decisions based on trusted data.

Reaching this end state for data governance is challenging though – there is no magic bullet to take you there, and technology is only part of the answer. Technology can help you identify what you need to govern (for example, by generating a list of dashboards and who has access to them), but it can’t (yet!) tell you how to govern it (for example, which dashboards duplicate information, or whether the people who can access them are the right people).

To fill in this gap, you need to look at process and, more importantly, people.

The first step is establishing a data governance framework, comprising policies and procedures together with a team who can help ensure that these are followed. This takes care of the process part of the equation, but to get the people part right starts with recognising that human nature is to take the path of least resistance when it comes to data governance, and that enforcing policies and procedures is probably the least effective way of establishing the governance that’s needed.

That’s where the idea of data citizenship comes in. Show people where they fit into the organisation’s data landscape, how they rely on others upstream to provide them with the right data of the right quality, and how what they do with data in turn affects others downstream. In other words, show people what it means to be a good data citizen, how it benefits them and why it’s important to others. This completes the data governance equation, and helps ensure success by working with human nature instead of against it (encouraging rather than enforcing).

Bringing everyone along for the ride.

Finally, having the right technology platform able to serve up the right data to the right people in the right way is still only part of the story. To achieve value, people need to be actively engaged with, and using, the technology and the insight it provides.

Remember that not everyone will be excited by the in-depth details of how everything works. As we said at the outset, technology is just a tool, and too much focus on the technology aspects of your data platform risks building apathy towards it, which is far from ideal.

Instead, focus on how the technology benefits people - by making sure they know how to use it, and what to use it for. Once people see how data and insight helps them achieve objectives and goals, they’ll buy in to the platform and ultimately become champions and advocates for it. Word of mouth anecdotal evidence will start to augment official communication and messaging, leading to a virtuous circle that guarantees that the technology platform delivers the intended value from the data it provides.

So, do you have the right technology platform to succeed?

To close, let’s revisit the question we started with: does your company have the right technology platform for a successful digital and data transformation?

If you’ve followed the above recommendations then you’ll have the right technology implemented in the right way. The data in the platform will be governed effectively, and the technology will be actively and consistently used by people in your organisation to provide them with tangible value.

And the answer will be a resounding ‘yes’!

Over the last 18 months, organisations have had to evolve to stay competitive, or even just to survive. Initiatives that were typically off-agenda or low priority pre-2020 have become mission critical, including the adoption of new services.

The evolution of the workplace has leaned heavily on technology, demanding digital modernisation, often at pace and with varying success. It has forced organisations to re-evaluate their trajectories and priorities, and as a result revisit their ‘digital roadmap’ with fresh eyes. Increasingly, the legacy application is no longer sustainable, and cannot be worked around. It is a serious impediment to business success - a ‘blot on the landscape’ and an obstruction to that critical evolution. Legacy applications and systems are often monolithic by design with unsupported binaries and platforms. When they fail, they fail hard, damaging productivity, user satisfaction and other dependent services. Their support and maintenance is often a costly and labour-intensive drain on high-value resources.

Savvy IT leaders know that there are options to bring legacy in line with an evolutionary strategy. These options exist along a continuum and are characterised by the degree (and cost) of change they represent. Each change must be considered against the potential value of the outcome. We typically see 3 main application change options – migrate, modernise or rebuild.

Migrate: Replatform

Migration involves the lift and shift of the legacy system, with minimal change to application functionality. The application still ‘looks’ like the original, but the infrastructure location and compute source will change, for example, moving from on premise physical or virtual servers to cloud Infrastructure as a Service (IaaS) virtual machines (VMs).

Replatforming naturally leverages some native benefits, such as scalability and availability. This benefit can only be realised however, if the application tolerates the new platform functionality. Whilst migrating to a new platform may promise lower cost of ownership and increased performance, little business value is added and any problems caused by the application code will persist, just on a shiny new infrastructure.

Also note that this option is rarely available to systems that are run on infrastructure that is way past end of life (for example running on a mainframe) – these are effectively impossible to get into the cloud. If this is you, scroll down for other options!

Modernise: Refactor or Rearchitect

Modernisation involves some form of re-engineering of the existing application - the scale of which is dependent on your requirements. Changing and optimising the application could involve using cloud-native services.

Refactoring

You could start off small and make minimal changes to the application: swapping out a server-based database for a cloud-based Database as a Service instance, or putting a web front end in a container or in a WebApp. The architecture of the application remains the same, but a portion can leverage some cloud native benefits, potentially creating efficiencies in terms of cost, scale and availability.

Rearchitecting

Taking this a stage further would involve Rearchitecting the application. Here we break down the application into its component parts, decouple capabilities and deploy them on to appropriate services. An example would be to look for common services that could be modernised into functions or containerised. These new constituent parts can be kept as separate concerns that talk to each other through documented APIs, easing maintenance and development. This obviously requires significantly more effort for it to be successful than the replatform or refactor solution, however, the longer-term benefits and overall business value are often greater.

Rebuild

The final approach is to Rebuild, and for many there is huge appeal in starting fresh. You can develop a new application that has all the known problems and mistakes designed out - and design in new features and capabilities that make it extensible, scalable, and modular. Similarly, a cloud-native application is likely to create the agility to release and update faster with lower risk and less stress and expense. You can also build in better governance, compliance and security adherence. You can build to your exact requirements today with the tolerance to accommodate tomorrow’s directional change. Should business requirements evolve or priorities switch, the platform, development, and support staff are able to pivot and reprioritise releases with agility. This represents the highest cost to implement, however the relative benefit is potentially massive and far reaching.

So, what’s the right approach?

It depends on several factors: your success horizon (what you need to achieve in the short versus long term), the time and resources you have available and their capability. It depends on budget and how you are going to measure the return on investment - and the cost of doing nothing. You need to start by understanding the business problem that needs to be solved and consider the potential long-term value. There are numerous other factors that should also be considered to make the decision on whether to migrate, modernise or rebuild.

Some of the challenges our customers typically bring to us include:

• Frameworks and operating systems not being supported by the business or having passed end of life

• Applications that do not ‘play’ with the other apps and services

• Feature improvements and new feature requests that are not feasible

• Limited internal capability to understand, manage and support

• Old tech and apps that are not interesting to developers, making recruitment difficult

• Unavailable hardware

• Expensive patching and updates in terms of downtime, fraught with uncertainty

• Benefits decay: severely diminished from its initial proposition

• Uncompetitive user experiences that negatively affect productivity

• Data centre migration mandates (in many weeks…)

• Duplication of data and lack of a true source of truth.

The decision to migrate, modernise or rebuild is really about right-sizing the degree of change your business can tolerate in line with readiness, budget and business impact. Even if you just lift and shift your app to a new cloudy platform like Azure, you have just taken a first step into the world of application modernisation. You have made a change for the better and opened up the opportunity for improvement at a pace that you can control. And trust us - your business, customers, users and stakeholders will thank you for it.

A Proof of Concept (PoC) is a high-value component of software development projects.

It’s a time-bound, fixed-scope exercise that tests the feasibility of a technical concept using measurable goals and success metrics. PoCs act both as a way for organisations to discover and learn about new technologies, and a way for organisations to manage the risks associated with exploring new technologies. However, as with any project, PoCs entail inherent limitations and pitfalls that can create cost, time and resource implications. Here’s my top 3.

1. Lack of business readiness.

One of the main pitfalls with a PoC is the inability of the broader business to adapt to the new capability it demonstrates. In practical terms, this can mean that a level of business transformation is often required beyond the original scope of a PoC, to adopt to a new technical system or process. It may be as simple as individual employees changing their workflows, or as complex as reconfiguring multiple business units. This can lead to a sort of ‘false positive’ from a successful PoC: the concept may be technically feasible, but broader adoption may present obstacles.

It’s vital to consider business process change when implementing a PoC into production, and is a crucial consideration in how an MVP (Minimal Viable Product) should be scoped.

2. Defining the wrong success metrics.

Technology is just a tool. Selecting the right tool for the right job is dependent upon having identified the right job to begin with.

Creating a PoC to answer a broad technical question - rather than applying technology to identify a specific business question - is a common pitfall. Consider the statement:

“Let’s see if we can use Machine Learning in Azure to help our business”

This is a difficult thing for a PoC to prove as it has no clear outcome, and it pre-supposes that Machine Learning is the right tool to solve this undefined need. Instead, we could add clarity to our goal by reframing:

“Let’s see if we can use Azure Data Services to help reduce time spent producing financial reports.”

This is a good starting point, and we can quickly refine the key technologies that need to be proved and simple ways of measuring the factors that define success. Good problem statements for PoCs are as specific as possible, often dealing with the viability of very low-level technical challenges such as integrations or data migrations with very narrow success criteria.

3. Building a PoC for production.

It is common wisdom that a PoC should not be put into production. Although this is almost completely true, it is an often-misunderstood statement. A PoC is immediately compromised if it is built to be productionised. This is because the additional considerations and work entailed in creating a production-grade solution adds significant overhead without knowing if the solution is even feasible, often increasing costs and risk against an uncertain outcome. However, if a PoC is deemed a success, that does not mean all the work undertaken needs to be thrown away by default. It means it can be objectively evaluated, and any viable component parts can be reused and carried forward into a follow-on project to build a production-ready MVP.

Moving to a production-grade solution means a much greater focus on code quality, reusability, security, testing and NFRs (non-functional requirements) among other things. Whilst all imperative in production systems, these will often add little value to the goals of a PoC. However, it is always wise during development to adhere to best practices to maximise the potential for possible expansion and refactoring, preventing the solution from becoming entirely disposable.

A good PoC will be rooted in a business value-based outcome and undertaken to prove that the proposed capability through a new application of technology or data is not only realistic, but the proposed approach is also feasible.