Why Success Is Often Elusive at the Highest Echelons

During recent discussions with friends, one common theme that crops up very frequently is how success has remained rather elusive to recent higher level hires at their companies.

By “higher level hires”, I’m referring to not just senior managers, directors, and VPs, but also very senior engineers (think principal level) who’re tasked with shaping or steering the organization’s engineering innovativeness, productivity, and culture. As often as not, these hires have a solid track record of success over the course of many years at elite tech companies (think a 6–8 year stint in senior engineering or engineering leadership at Netflix or Google), have solid references, some might even have a prominent public profile, and are generally competent and smart people.

Yet, none of these past laurels help them replicate this success at their new workplace, where they struggle to make any meaningful inroads, and often exit (or are tacitly pushed out) within 18–24 months. Sometimes they go back to their old companies who welcome them back with open arms, or they end up somewhere new.

It’s extremely rare to hear about case studies of engineering leaders who’ve been wildly successful in an organization replicate the same success every time they change jobs. I don’t have any hard data on how frequently this happens across the industry, but anecdotally speaking, I can attest that this is extremely rare. The few notable examples might indeed be the exceptions that prove the norm.

The purpose of this post is to posit some theories behind why this might be the case.

At The Highest Levels, You Need To Innovate And Execute

I recently came across an interesting tweet, which I think is applicable here.

This is especially true at the highest echelons of engineering — it’s a sine qua non to be able to dream and execute. This is applicable to both the IC and management track. Successfully manoeuvring both innovation and execution often results in the highest level impact. These are two very separate problems, requiring two very separate skillsets, which are seldom found in conjunction.

Innovation Requires A Support Structure

Being truly innovative is hard. Looking at the current technological landscape, very few companies can be said to be true pioneers in their field.

Objectively, the most purely technical innovative realm today is artificial intelligence, where we’re seeing truly groundbreaking and revolutionary technology being developed. This is true, albeit to a somewhat lesser extent, when it comes to virtual reality, hardware and cloud infrastructure.

However, most such breakthroughs are generally coming from large tech companies (a few notable exceptions notwithstanding). It also typically comes from people who’ve generally spent a very long time (often at the same company) working on a small problem domain. Such people (on the IC track) with deep subject matter expertise tend to have a narrow focus, stay in their lane, generally stick to the IC track, and are few and far between. They know how to execute within the confines of their organization with the support of managers and leadership they’ve been working with for a long time.

Parachuting such people into different cultural milieux where they are left without the scaffolding they’ve implicitly grown to expect and depend on often leads to underperformance. Some adapt, of course, but many flounder, especially when hired at very senior levels with daunting short-term expectations.

Context Is Queen

Innovation, especially when it comes to innovation that makes the organization more productive, is impossible without context into the organization’s history and culture. It’s one of the reasons why building or adopting tooling that’s successful at other companies doesn’t necessarily solve all the problems faced by one’s organization, because a finished product doesn’t tell the story behind how the tech evolved, what overarching problems it aimed to solve, what implicit assumptions were baked into the product, and more.

Building products to solve problems without a full awareness of the context in which the problem exists in the first place often results in failure. It takes time to learn the context — by “learn”, I mean not just become aware of it, as is possible from meetings or reading up about past decisions etc, but to truly get an intuition for why and why not something might work in the future, given what’s been tried and tested in the past. This kind of intuition is indispensable for innovation of all sort, but especially so when it comes to the kind of innovation that solves problems at the sociotechnical axis.

Culture Can Have A Steep Learning Curve

Being able to execute requires being very well-versed in the culture of an organization. You have to know how your organization works in order to get it to work diligently like a well-oiled machine to execute on the higher level strategic vision.

The larger, older and more fractured an organization (as is the case with large tech companies), the harder it is and the longer it takes to learn the culture. And by “learning the culture”, I don’t mean simply parroting the shibboleths or the oft-repeated mantras, but to truly internalize it to an extent where it informs every small decision and discussion. At the best of times, that’s difficult and takes time, particularly so when there isn’t a single monolithic culture to learn, but myriad microcultures indigenous to various different parts of the organization.

It’s more difficult for people who haven’t done it frequently enough in the past (i.e., every so often changed jobs or organizations where they’ve had to re-learn the cultural ropes). In the worst case, where an organization has a culture diametrically opposite to that of the previous workplace, “learning the culture” also requires un-learning almost everything that led people to get to their current level in the first place. Not all newly hired senior leaders are entirely committed to or feel comfortable turning themselves into the leader the organization truly needs, rather than the leader they’ve grown to be over the past years.

Many leaders take the opposite approach or trying the mould the organization in their image or the image of the past workplace. Engineering leaders brought into embattled organizations tasked with stabilizing the chaos are often heavily incentivized to do this. Many a time these folks, in my experience, tend to fail harder and more often than those who try to learn the organizational ropes and tailor their leadership style to fit the organizational culture.

Changing the organizational culture wholesale is an extraordinarily hard undertaking for precisely the same reasons why changing oneself to fit into a new culture is hard. Changing the culture of an organization, in effect, requires every single member of the organization to subject themselves to the aforementioned process of un-learning their current cultural practices, and internalizing the newer cultural tenets to the extent of informing every small decision they might take or discussion they might partake in. This is much, much harder to do at scale.

While changing culture at scale is certainly doable with enough forethought, effort, time, patience, the right people and most importantly, the right reward structure, it also requires the very rare sort of leader with enough personal credibility, charisma and tenacity who can engender the form of trust needed at all levels of the organization (upper-management, middle management to rank-and-file engineers) to successfully pull this off. Such leaders are true unicorns and are incredibly hard to find, which is why we don’t hear more case studies of leaders successfully adapting to environments to replicate their past success over and over again.

Conclusion

We’re living through the era of “The Great Resignation”.

While this might work well enough for rank and file engineers (it’s not a particularly negative thing for a software engineer, especially those early in their career, to have held 3 jobs within the course of 5 years), I suspect a different narrative might begin to take shape for engineering leaders or very senior engineers who fail to produce any meaningful impact at a string of companies they’ve worked for.