Emotions & Facts – Becoming data driven by overcoming bias

Humans are very, very good at rapid pattern recognition. It is the basis of the flight or fight response and based on our ability to see past events in current and future situations.

… humans are amazing pattern-recognition machines. They have the ability to recognize many different types of patterns – and then transform these  “recursive probabalistic fractals” into concrete, actionable steps …

From: Humans Are the World’s Best Pattern-Recognition Machines, But for How Long?

This fact is leading to a number of advances in AI leveraging big data approaches. It enables us to understand what is happening right now or what might happen in the future based on recognizing patterns found in historical data. And this is good – and bad.

In stable systems – businesses that dominate their markets in particular, but also in political parties, social groups and non-competitive systems – cognitive biases can make your pattern recognition superpower your kryptonite. How? By convincing you that new data – competitors, market behaviors, demographic shifts, and disruptions – are false.

Too often the reaction to these leading indicators is disbelief or even retrenchment. In institutions that lack high quality data driven practices confirmation and conservatism biases often become the norm furthering the notion that the old patterns still apply. All too often this results in, what appears to be, a sudden collapse.

The key to avoiding this fate is to consistently apply solid data driven approaches which negate the biases and our very human tendency to dismiss data that doesn’t conform to our known patterns. Acknowledging the reality of the data “as it is” and attempting to validate the data via consistent, unbiased best practices enables us to recognize changes in the underlying patterns more rapidly.

That ability – to be open to questioning your pattern recognition and the biases inherent in it – can become your real superpower. That ability to be data driven; to continuously evaluate the data to understand reality in an objective way and apply what is learned is the superpower of enduring, innovative organizations.



“Only doing what we can execute now” is a terrible strategy – a prescription for unsticking your engineering team.

Photo Credit: Radarsmum67 on Flickr

I engage with a lot of engineering teams (and leaders) that are stuck. They know full well they need to do something to enable thier product, service or business team – but they can’t get started.

In almost every case I find a culture of resistance – which can be best summarized as:



I don’t think we can execute that – I mean, we’ve never done it before and have no idea how to do it.


I sympathize – I really do – but only signing up to execute what the team already knows how to do is accepting defeat. Yes, I understand, you’ve never used a document database; nope – you’ve never used a horizontally scalable messaging infrastructure; true, you don’t have any experts in WebRTC; yes, I get that you don’t even know what technology to think about to solve this problem.

My prescription in these cases is simple:

  1. Figure out the smallest valuable thing you can implement – and implement it.
    • If you can’t decompose the feature/problem find someone who can help you do it – they all decompose.
  2. The people you have are smart – assume they will figure it out. Your confidence in them generates their confidence in themselves.
    • If they can’t you have to up-skill your team! Get outside help in the interim.
  3. Be relentlessly unafraid to fail!
    • First, you will even if you are afraid – unless, of course, you just stay stuck and do nothing. Second, failure is an outcome – and outcomes are good – we can learn from all of them.
  4. Go back to #1 and repeat.

Two more quick points:

  1. You don’t have to be formal leader (engineering manager) to do any of the 4 above. However, if you are and you don’t support these activities your team will stay stuck.
  2. Be 100% transparent with your stakeholders (product manager, business partner, engineering leadership, etc) about where you are on your journey from stuck to “we got this”.

The consistent application of this prescription – in my experience – leads to teams that rarely get stuck. More importantly you have created the foundation any engineering team needs to become high functioning and deliver consistently for the business.

The Message We Do Not Hear

Image Credit: David Mulder

Over the last few days I’ve attempted to write a post here explaining why data driven analysis and action is so important in relation to recent events. It isn’t hard to find data and high quality analytical analysis that speaks to the tragic events in Minneapolis, Baton Rouge and yesterday in Dallas.

It also isn’t hard to connect the resistance to analytical analysis in corporations to the same phenomena in recent events.

Having said all that, however, I can’t make myself complete or publish that post; because it as correct and timely as it may be I can not help but feel any such post would be inappropriate at this time.

So, for now I will simply say:

  • Reality and your world view are not the same thing.
  • Reality exists in both the antecdote and the aggregate – respect both.
  • Bias exists with and without intent.
  • The solutions will come through understanding, not tribalism, hostility and violence.


Enterprise Architecture – Build versus Buy

Photo Courtesy of Steven Depolo

In the bad old days (pre all the aaS’s) enterprise architecture revolved around build versus buy. The crux of the decision was – does this capability provide us significant and lasting strategic competitive advantage and therefore justify the “build” choice?

Today, the cost of “buy” is arguably hirer than ever before – with SaaS per user pricing and recurring contracts, and the cost of “build” is lower than ever before with IaaS and PaaS solutions driving down the cost of development and maintenance.

Moreover, with more companies offering services over the top of product as strategic competitive advantage (e.g., Zappos, Apple Genius Bar, Prime Now, Telemedicine, Online School Access, etc) the old “front office” out of the box best practices solutions aren’t worth buying unless you invest in making them serve your service differentiation.

All of that, taken together, means enterprise architects need to be thinking about the optimal mix of:

  • Buy
  • Rent (SaaS/IaaS/PasS)
  • Build

to enable and sustain durable competitive advantage in an evolving marketplace.

Stop Calling Yourself a Developer

First and foremost, let’s be clear, the entire tech industry suffers from a distinct lack of specificity when it comes to roles and titles. That, however, is no reason to market yourself as a Developer.

Developer implies that you just generate code. That, however, isn’t what (most) of us do. We create systems that generate value by solving meaningful problems. What Developer fails to describe is the focus on the system – which includes, but is not limited to, the code.

Good technologists generate different code for AWS then they would for a physical server in a datacenter. They generate different code for 1000 users then they would for 1.5 billion users.

What we really do is engineer systems based on needs, available tools and desired outcomes. So don’t refer to yourself as a developer – unless of course that is really all you do.

Leadership: Risk or Opportunity Focused

imagesAs I’ve worked with teams engineering teams big and small – in both enterprise and startup contexts – over the last 20 years I’ve noticed two distinct patterns in leadership and their impact on the culture and productivity of those teams.

Pattern: Risk Focused Leadership

Risk focused leadership emphasizes the up front identification and mitigation of risk in any program or project. It attempts to know as much as possible before committing and rewards engineers who can identify and articulate risks.

Impact on Culture

Since the engineer’s perceived worth is derived from her ability to identify reasons things won’t work – or more precisely to avoid mistakes – the culture tends to favor inaction and exhaustive research and analysis.

Impact on Productivity

Predictably, these teams tend to have low output. Generally the output they do generate is both expensive and highly reliable. In enterprise contexts there tends to be a reliance on proven vendors – usually with a bias toward those with long market histories which can be analyzed.

Pattern: Opportunity Focused Leadership

Opportunity focused leadership emphasizes the potential gain of any program or project. It – often aggressively – attempts to capture opportunities as they present themselves. These leaders reward engineers who can grasp the opportunity and rapidly implement solutions which might capture the opportunity.

Impact on Culture

Since the engineer’s perceived worth is derived from her ability to create solutions which may capture the opportunity – or more precisely move quickly with imperfect information – the culture tends to favor rapid cycles of activity and an ability to “change gears” rapidly.

Impact on Productivity

Predictably, these teams tend to have very high output, however, much of that output goes unused. Generally – but not always – the output is proof of concept quality with a bias toward open source tools, frameworks and platforms. Since the long term viability of the opportunity and feature/product were not exhaustively analyzed teams learn to implement low cost solutions which can be “thrown away”.

Leadership Lessons

What should be obvious by now is that neither is bad or good – each is appropriate in certain contexts and, more often than not, a project, program or organization requires a well defined, understood and articulated balance between the two leadership focuses.

Some leaders are naturally opportunistic, some are risk managers. As a leader of a engineering or product development team it is your responsibility to understand:

  • The opportunity/risk profile of the program/project/product.
  • The relative opportunity/risk propensities of your team.

Most importantly, you must ensure that the opportunity/risk profile is articulated and the stakeholders understand and agree with the inherent tradeoffs for any program, project or organization. Failing to do that is the unacceptable risk you must avoid at all costs.

Big Data – Empowering the Age of Agile Analytics

Big data is a buzzword, no question. Given that it is incumbent on practitioners – in particular architects – to tie the new Modern business conceptpatterns available in a “big data” enabled infrastructure to practical business benefits.

While there are a variety of business benefits that are enabled by big data infrastructure the single most tangible is Agile Analytics (also known as self service BI and data discovery and exploration). Here’s why:

1) Your business users never wanted reports.

What they really wanted was to be able to leverage data to answer questions. Traditional BI infrastructure did that well, provided you knew what questions you wanted answered in advance.

The problem is, you don’t. The world moves too fast to create a set of KPIs and instrument your business by those alone for 10 years.

2) Data Driven decision making requires empowered business users.

Business users must be empowered to use the the data directly – without intermediation by technical staff – in order to realize the benefit of data driven decision making.

This isn’t to say the technical staff doesn’t have a role – they do. They provide the platform and advanced support enabling business users to use data directly.

3) The prepared data can only answer known questions.

Business users need to follow the data to the important insights. They have the business knowledge to derive insights that matter, but they can only base those insights in data if they are empowered to explore in the data to follow the information to the value.

This means all the data – from the raw data, through each transformation or aggregation to the KPIs and rolled up analytics.

Big data infrastructure – properly deployed and governed – can provide a platform which solves the key problems preventing business users from engaging directly with the data and discovering valuable insights.

Data Architects – no one cares about your ERD…

Data Architecture has changed – massively – and most data practitioners haven’t caught on yet.

Modern business conceptIn a SaaS world – where business users can sign up for and use powerful platforms by simply signing up we have to radically rethink what data architecture means.

It certainly isn’t about your ERD – your elegant descripton of the data at rest

It is, however, about how that data is acessed, shared, and moves about the technology toplology enabling business agiltiy and experimentation.

So stop using your ERD as a substitute for what a modern data architecture actually requires – a domain model and APIs that enable simple data integration at the point of need.

How to spot a fake Big Data Engineer

Photo Credit: counterfitsoner on Flickr

I interview a lot of candidates… I mean a LOT.

And every resume I get these days has a “Big Data Project” listed.

So, naturally, my first question is – what is it that made it a “Big Data” project?

Top five immediate disquallification answers are:

  1. We were using Hadoop
  2. We had TONS of data
  3. We were running map reduce jobs
  4. The data was unstructured
  5. It wasn’t in our data warehouse

The truth is, no one knows what you mean when you say it was a “Big Data” project – and we all know it is on your resume as keyword search fodder – but if you are going to have it on there you better come with a better answer than one of the five above.

Back to Basics

Back to basics

Sometimes, you have to get back to basics. It is too easy to get caught up in the size of the opportunity, an open IPO window or high expectations. 

Great companies are built with an intense focus on customers and problems. Great businesses are built by maintaining an intense focus on the metrics that matter – revenue growth, churn, net promoter, cost of goods sold and customer acquisition costs. 

Great engineering organizations build systems that solve customer problems – without creating new ones.

Great customer success organizations are laser focused on making customers successful.

Great product organizations are obsessed with the market, customers and – ultimately – fall in love with the problem they solve and commit themselves 100% to solving that problem… again, again and again.

When you strip everything else away and get back to basics – prioritize what really matters – you achieve clarity of purpose and message.