Solving Solved Problems — Lessons from my journey

When I started my professional journey, the technology landscape looked completely different, yet the patterns in which people operate within this ever-changing landscape keep repeating themselves.

It’s interesting and somewhat frustrating that the challenges I experienced leading small teams in small companies are the same challenges in leading teams of hundreds in large enterprise businesses!

The interesting part is recognizing the patterns shared across all these experiences and creating solutions that scale up and down to accommodate the context. The frustrating part is how seemingly solved problems keep repeating themselves, and developers keep falling into the trap of reinventing the wheel.

No-one actually has a foolproof plan; people that claim they do are either faking it or miscommunicating their ability to identify patterns. This is especially true in software technology. Our industry‘s half-life of knowledge is rapidly shrinking from a few years to a few months, as the rate of change and innovation keeps increasing. And while change is constant for us, so are the patterns!

Or as popularized by Mike Tyson’s quote: “Everybody has a plan until they get punched in the mouth.”

In almost every team that I led, the following “solved problems” keep coming up: user authentication/authorization, logging & monitoring, data warehousing & BI, data management & security, infrastructure automation, etc …

These are but a small set of “solved problems” that kept on being reinvented, despite many products and service providers that exist solely to solve these problems for you! And I often found myself in the position of being “that guy” who has to pull the brakes and say, “how about we don’t waste time reinventing the wheel here!”

One of the fundamental principles I try to instill in every team is “we should only build things that ONLY WE CAN build.” While that might seem obvious on the surface, it’s hard for many development teams to realize that they don’t have to reinvent the wheel and roll out custom authentication systems, deployment tools, or monitoring solutions…

Our industry’s communities tend to believe that certain companies, which are seen as “leaders” or drivers of “innovation”, somehow have all the answers and know exactly what they are doing. I’m here to tell you that they don’t. Humans tend to put people and companies they admire on pedestals and then blindly copy or mimic them. This is dangerous, especially for technology teams.

Technical Debt and Reinventing the Wheel

My first exposure to a wildly popular company and product was at Viigo, a BlackBerry app that predates the iPhone and Android smartphone world. In this world, everybody had a BlackBerry in their pocket, and on that BlackBerry, they had Viigo installed. The app was so popular, it topped the BlackBerry App Store charts for years, even outranking Facebook in downloads and popularity!

I remember getting on subways and streetcars in Toronto, and seeing everybody on their phones using the app I worked on! The local tech community ended up putting the company and development team on a pedestal as if we were creating some groundbreaking technology

In reality, it was a glorified RSS reader with some additional functionality and hackery to get around the limitations of the BlackBerry OS. Most of the content was funnelled through some bat-shit-crazy XML/XSLT processing in PHP that I put together and threw on a couple of EC2 servers … there was no reason this janky XML/XSLT/PHP combo should have scaled to support tens of millions of users and a vast amount of rich media content and real-time sports events (we even covered the Olympics live with nothing but an XML feed and XSLT!). Still, it worked, and it served millions of users across the world! I led the Web Services team at the time, which consisted of myself and two developers mucking around with XSLT templates all day long!

Years later, I joined Mashape (now known as Kong Inc.) as VP of Engineering where we began with an API Marketplace that grew from 50 users during our Alpha launch in 2010 to over 350,000 users, powering billions of real-time API calls! We later built out a standalone API Management Proxy called Kong, which took all the experience we gained to build and maintain the Marketplace into a standalone product meant for Enterprise-scale API management. The company has since shifted entirely to operating as Kong Inc. and focuses solely on the Enterprise API product space, with multiple products and services.

When we would go to conference events, we would have a line at our booths that sometimes stretched across the entire booth area, much to the dismay and irks of fellow booth exhibitors! People wanted our cute gorilla mascot t-shirts, and they wanted to chat with us about the “amazing” products we were producing that made their lives easier.

At the time, the engineering team was no more than ten developers that built and maintained four products with massive adoption and massive amounts of traffic—the secret technology behind it all: Nginx and Redis. There was nothing fancy or exceptional about the stack, and we barely managed to keep our heads above water as we proxied billions of business-critical API calls for our users through our systems.

My most recent role was as CTO of npm, Inc. With a massive worldwide community of users that looked to us to provide them with critical services that directly affect their productivity and daily work. Every web developer in the world used our products, with over 1.3 million open-source packages, and serving ~125 billion requests at a whopping six petabytes per month!

A small team of ~25 developers were responsible for this large-scale & critical product, dealing with the same challenges and technical debt problems as any other startup or enterprise team. If it weren’t for a CDN partner taking care of the majority of the traffic at a HIGHLY discounted flat rate, there would have been no way a small team like that could maintain that level of scale!

All these teams were burnt-out. None of them were happy. But we were and still are proud of the large-scale work we accomplished with such small team sizes.

The other common thread in all these experiences was that they were all massively popular technology products mired with technical debt and reinventing-the-wheel syndrome. Some of that debt was created by me back when I didn’t know any better, the rest I inherited. This debt usually comes with a breakdown of technical culture and business growth challenges. While not simple nor easy, the remedy was always about focusing on only building things that only the company can build.

Viigo was acquired by BlackBerry in 2010, Kong sold the API Marketplace to RapidAPI in 2017, and npm was just recently acquired by GitHub.

Enter the Enterprise

Before joining npm and soon after the Viigo acquisition, I went into the Enterprise world for a change of pace. In 2013 I joined the CBC as Development Manager, Digital Operations leading a team of 35, and in 2017 I was Chief Architect at TELUS leading all technology practices and teams spanning over 450+ technologists. And while I’ve had many years of experience in servicing and selling technology products to Enterprise companies, my years inside the Enterprise were perhaps the most eye-opening to our industry’s patterns!

The main takeaway:

There are no differences in technology operations and software development practices between Enterprises and Startups; the only differences that exist are in communications and processes, which are fundamentally human—not technological—problems.

Enterprise teams are also stuck solving the same “solved problems” and accumulating technical debt, rather than focusing on building the things only they can build. The difference, though, is while these were the same problems, they were distributed across departments and repeated across many teams, which ultimately adds up to hundreds of millions of dollars in waste.

Lacking centralized technical leadership and unified vision in the Enterprise world is the root cause for teams to fall into these traps and continue going deeper into the cycle of reinventing the wheel and creating technical debt.

In both worlds, Enterprises and Startups require a centralized model of technical leadership. A clear and measured technical vision to avoid reinventing the wheel and minimizing technical debt can only be achieved by investing in people first, ensuring you have the right leaders, the right members, and the right tools to help them succeed. The responsibility to establish such leadership and vision lies in the hands of the CTO.

Are there any answers?

Spoiler Alert! there is no ONE answer to address these challenges, and that’s okay! If there was just one answer, then the entire industry will fold into itself, and we’d all be working for Microsoft!

And to pay respect to Mike Tyson’s wisdom again, I can’t suggest having a “plan” to address these challenges, as the next “punch” life delivers will throw any plan out the window!

There are, however, some key patterns that we share and pitfalls we know to avoid. My focus will be on building and sharing a body of knowledge in these areas so that CTOs, Founders and Engineering Leaders can benefit from and contribute to each other’s success!

I’m planning to use this space to start tackling some of those patterns one by one, and I’m looking for the community’s feedback & contribution to surface our collective body of knowledge higher and higher!

Need help now?

While writing and sharing lessons are helpful, it takes time, and it may not be the thing you need RIGHT NOW. But don’t worry, I might still be able to help you!

I’ve started taking on Fractional CTO roles to help startups and small businesses dig out of technical holes, accelerate product roadmaps, and strengthen technical team culture and best practices. Get in touch to discuss any challenges you’re dealing with today.