Every company is a digital company. No longer is it a question of if your product will become digital – as was the case with music, newspapers, TV, movies, etc. – it is a question of how the experience of your product (and your company) changes even if your product isn’t digitized.
eCommerce, digital marketing, social, CRM, and content technology and strategies are critical. You will need to invest in those technologies – but underpinning all of those technologies is data – that data is the currency of your digital transformation.
I’m not always sure people always know what they mean when they talk about Big Data – and even when they do know, I’m not sure they can contrast this new Big Data thing from Data’s previous incarnation.
So let’s see if we can clear it up.
Prior to big data the amount and content of the data you had access to was limited – in technical terms you had to deal with a limited information domain. Why? Because obtaining and storing data was expensive and, more importantly, most data was locked up in the real world and never entered the digital (binary data living in computational systems) realm. That obviously has changed.
This flip – from only generating and storing data directly relevant to operating a business to having access to, collecting and storing massive amounts of data which may or may not be relevant to operating a business is the state change.
The first big problem was tooling. The systems and technologies to collect and store data were designed for the relatively small amounts of strictly modeled data relevant to running our business. Moreover, they were designed to strictly control adding to it, because that was expensive. This was the problem we needed to address first – which is why when we talk about Big Data we invariably talk about technologies – Hadoop, MongoDB, Spark, Kafka, Storm, Cassandra…
But, for business leaders this is misleading, because implementing any (or all) of those technologies will not make the business effective in a Big Data context. These technologies will not provide you magical data which supercharges your business. You will not suddenly have insights your competitors do not; you will not – overnight – find the clarity required to dominate your market.
The key is to combine those tools and capabilities with data driven practices and culture.
Let’s start by avoiding the mistake made with Big Data – let’s clearly talk about what has changed and why data driven is different than what came before.
I’ve worked with organizations – from startups to enterprises – that have robust reporting and systems of operational metrics they use to run the business. They review reports and dashboards regularly, perform regular operational reviews focused on those metrics and target resources and budget toward those that are under performing. Invariably they suggest they are already data driven – because they leverage data to run their business.
They are not. They are optimally operating in the pre-Big Data model – where the universe of data was fixed, the metrics long lived and stable and information outside that realm unobtainable – those insights beyond reach.
A Data Driven organization still does those things – metrics, operational reviews, targeted investments based on under performing metrics. But, they also leverage the larger universe of data to openly question the validity of those metrics; they develop processes to evaluate that universe for new metrics and insights; they allow the data to lead them to opportunities and the identification of threats.
This practice almost always feels like a radical shift – and it is. Organizations must shift from the practice of only focusing on the known knows and embrace this new ability to examine and gain insight from the known unknowns and unknown unknowns.
Reports that say that something hasn’t happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – the ones we don’t know we don’t know. And if one looks throughout the history of our country and other free countries, it is the latter category that tend to be the difficult ones.
When these Data Driven processes and practices, extending and augmenting your metrics driven operational practices, become part of the culture the real value of all that data and all those tools can be realized.
Humans are very, very good at rapid pattern recognition. It is the basis of the flight or fight response and based on our ability to see past events in current and future situations.
… humans are amazing pattern-recognition machines. They have the ability to recognize many different types of patterns – and then transform these “recursive probabalistic fractals” into concrete, actionable steps …
This fact is leading to a number of advances in AI leveraging big data approaches. It enables us to understand what is happening right now or what might happen in the future based on recognizing patterns found in historical data. And this is good – and bad.
In stable systems – businesses that dominate their markets in particular, but also in political parties, social groups and non-competitive systems – cognitive biases can make your pattern recognition superpower your kryptonite. How? By convincing you that new data – competitors, market behaviors, demographic shifts, and disruptions – are false.
Too often the reaction to these leading indicators is disbelief or even retrenchment. In institutions that lack high quality data driven practices confirmation and conservatism biases often become the norm furthering the notion that the old patterns still apply. All too often this results in, what appears to be, a sudden collapse.
The key to avoiding this fate is to consistently apply solid data driven approaches which negate the biases and our very human tendency to dismiss data that doesn’t conform to our known patterns. Acknowledging the reality of the data “as it is” and attempting to validate the data via consistent, unbiased best practices enables us to recognize changes in the underlying patterns more rapidly.
That ability – to be open to questioning your pattern recognition and the biases inherent in it – can become your real superpower. That ability to be data driven; to continuously evaluate the data to understand reality in an objective way and apply what is learned is the superpower of enduring, innovative organizations.
I engage with a lot of engineering teams (and leaders) that are stuck. They know full well they need to do something to enable thier product, service or business team – but they can’t get started.
In almost every case I find a culture of resistance – which can be best summarized as:
I don’t think we can execute that – I mean, we’ve never done it before and have no idea how to do it.
I sympathize – I really do – but only signing up to execute what the team already knows how to do is accepting defeat. Yes, I understand, you’ve never used a document database; nope – you’ve never used a horizontally scalable messaging infrastructure; true, you don’t have any experts in WebRTC; yes, I get that you don’t even know what technology to think about to solve this problem.
My prescription in these cases is simple:
Figure out the smallest valuable thing you can implement – and implement it.
If you can’t decompose the feature/problem find someone who can help you do it – they all decompose.
The people you have are smart – assume they will figure it out. Your confidence in them generates their confidence in themselves.
If they can’t you have to up-skill your team! Get outside help in the interim.
Be relentlessly unafraid to fail!
First, you will even if you are afraid – unless, of course, you just stay stuck and do nothing. Second, failure is an outcome – and outcomes are good – we can learn from all of them.
Go back to #1 and repeat.
Two more quick points:
You don’t have to be formal leader (engineering manager) to do any of the 4 above. However, if you are and you don’t support these activities your team will stay stuck.
Be 100% transparent with your stakeholders (product manager, business partner, engineering leadership, etc) about where you are on your journey from stuck to “we got this”.
The consistent application of this prescription – in my experience – leads to teams that rarely get stuck. More importantly you have created the foundation any engineering team needs to become high functioning and deliver consistently for the business.
As I’ve worked with teams engineering teams big and small – in both enterprise and startup contexts – over the last 20 years I’ve noticed two distinct patterns in leadership and their impact on the culture and productivity of those teams.
Pattern: Risk Focused Leadership
Risk focused leadership emphasizes the up front identification and mitigation of risk in any program or project. It attempts to know as much as possible before committing and rewards engineers who can identify and articulate risks.
Impact on Culture
Since the engineer’s perceived worth is derived from her ability to identify reasons things won’t work – or more precisely to avoid mistakes – the culture tends to favor inaction and exhaustive research and analysis.
Impact on Productivity
Predictably, these teams tend to have low output. Generally the output they do generate is both expensive and highly reliable. In enterprise contexts there tends to be a reliance on proven vendors – usually with a bias toward those with long market histories which can be analyzed.
Pattern: Opportunity Focused Leadership
Opportunity focused leadership emphasizes the potential gain of any program or project. It – often aggressively – attempts to capture opportunities as they present themselves. These leaders reward engineers who can grasp the opportunity and rapidly implement solutions which might capture the opportunity.
Impact on Culture
Since the engineer’s perceived worth is derived from her ability to create solutions which may capture the opportunity – or more precisely move quickly with imperfect information – the culture tends to favor rapid cycles of activity and an ability to “change gears” rapidly.
Impact on Productivity
Predictably, these teams tend to have very high output, however, much of that output goes unused. Generally – but not always – the output is proof of concept quality with a bias toward open source tools, frameworks and platforms. Since the long term viability of the opportunity and feature/product were not exhaustively analyzed teams learn to implement low cost solutions which can be “thrown away”.
What should be obvious by now is that neither is bad or good – each is appropriate in certain contexts and, more often than not, a project, program or organization requires a well defined, understood and articulated balance between the two leadership focuses.
Some leaders are naturally opportunistic, some are risk managers. As a leader of a engineering or product development team it is your responsibility to understand:
The opportunity/risk profile of the program/project/product.
The relative opportunity/risk propensities of your team.
Most importantly, you must ensure that the opportunity/risk profile is articulated and the stakeholders understand and agree with the inherent tradeoffs for any program, project or organization. Failing to do that is the unacceptable risk you must avoid at all costs.