The Modern Application Model

SaaS (Software as a Service) and Cloud Computing have taken center stage, accelerating all aspects of our lives even more quickly due to the COVID-19 pandemic. This year alone, it is enabling world wide communication, productivity & education under lockdown, solving complex protein folding challenges and developing new therapies and better understand emerging diseases!

While software technology has been a catalyst for these incredible transformations, the foundational protocols, standards, and systems we rely on to build modern software have not seen the same level of transformation. Their reliability and consistency makes the rapid acceleration on the topmost layers of software development possible.

For example in the realm of web development, foundational technologies such as HTTP, JSON, HTML, SSL, Networking, and more… create the stable bedrock for solving higher order problems in dependent layers. Problems that are closer to their diverse user experience needs, than they are to underlying data algorithms and communication protocols.

Layers of Abstraction

The Open Systems Interconnection model (OSI model) is perhaps the most well-known example of thinking in “layers”.

The model is split into seven different abstraction layers: Physical, Data Link, Network, Transport, Session, Presentation, and Application.

The OSI model characterizes computing functions into a universal set of rules and requirements in order to support interoperability between different products and software.

This method of abstraction and separation of concerns has been essential to allow acceleration and innovation across all layers of the model over the years. It is most evident by the meteoric rise of cloud computing, which completely encapsulates all layers of the OSI model. Cloud computing abstracts away the complexity, and even the knowledge, of these fundamental technologies to nothing more than an API call to deploy a server, or a configuration option to tweak application specific behaviour.

Layers of Knowledge

In the same way that one does not need to know how to build a modern CPU from scratch, nor build an entire operating system to host one’s application, we no longer build our applications from scratch. To be a successful software developer, or to build a successful software technology business, requires little knowledge or concern for the underlying foundational systems. The foundational systems have been abstracted away by cloud service providers who have the domain experience and dedicated resources to build, maintain and operate the systems needed for the rest of us to build our software.

In most cases we even interact with foundational technologies indirectly through frameworks, tools and SaaS to construct and glue together the necessary components for our applications.

To build software today, is to build on the shoulders of many layers of abstractions, built and maintained by companies we may not even know exist. Building from scratch is no longer viable, nor even logically possible.

“If you wish to make an apple pie from scratch, you must first invent the universe.” ― Carl Sagan

Defining The Modern Application Model

Most of today’s software development is occurring within the Application Layer as defined by the OSI Model. However that definition is a bit dated and does not exactly reflect today’s technology.

The OSI Model was created at a time when network computing was in its infancy. After years of debate it was finally published in 1984 (which makes it one year older than me) and primarily described contemporary Network Architecture.

Since then we’ve abstracted away the entire network architecture and trusted it’s operations to 3rd party providers. We need a new model to visualize what we, Application Software Developers, are building.

Zooming into the Application Layer, and keeping with the abstraction layer thinking model, we can identity 4 new sub-layers of modern application development:

7.1 Infrastructure Layer

Abstract the OSI Layer model and offers software capabilities “as a service”, this is most commonly referred to as “The Cloud” and is serviced by vendors such as: AWS, Azure, GCP, IBM, etc…

7.2 Transaction Layer

Abstract and process low-level transactional needs of Applications, such as Payments, Emails, SMS, Image Processing, Data Transfer, etc…, serviced by Vendors such as: Twilio (Communications), Stripe (Payments), SendGrid (Email), Segment (Data), Cloudinary (Media), etc…

7.3 Integration Layer

An emerging layer created by the cloudification of Infrastructure and Transactional Systems. Application development relies on this layer to process data between Transaction Systems, and to facilitate the abstraction of the Logical Layer, where most business specific logic is built.

7.4 Logical Layer

Human-computer interaction layer, where applications can access services, and data sources. This is where companies build their intellectual property, business logic and distinguished offering.

A Modern Example

Let’s bring a practical example into this model.

Your team is tasked with building a home services marketplace, where homeowners can publish projects, receive bids, and hire top local contracts, some of basic and fundamental requirements are known already:

• The Marketplace will be served as a Web & Mobile application
• Homeowners & Contractors will interact with the marketplace with their individual, family or business accounts
• Homeowners will pay the Contractors through the Marketplace, which takes a commission per project
• Contractor are only paid when the Homeowner signs off on the project’s completion
• Product research need to track and analyze how the marketplace is used to better improve the experience
• Must secure customer data, and potentially adhere to regional data compliance laws

These are a few very basic and fundamental requirements in any software application built today. Let’s take a look at the development journey the software team will undertake and how it maps to the Modern Application Model.

Step 1. Choosing Infrastructure Provider

Unless there is a critical business case, or some highly specific compliance requirement which forbids you from hosting your application infrastructure on the cloud, you’re most likely going to pick an Infrastructure Layer provider from one of the big players (Google, AWS, Azure) or smaller, yet more targeted providers (Netlify, Vercel, etc.).

There is very little rationale for a build-vs-buy discussion at this layer. Even if you’re not going to the cloud, you’re still not building a data center from scratch - you’re going to rent server space from a local one and entrust them to operate and maintain it for you.

Your compliance and security needs will partially be addressed at the Infrastructure Layer, (e.g. deployment region of your cloud application satisfies your regional laws) but it will also surface at every subsequent layer, and you are legally responsible for your customer’s data across every vendor it passes through.

Step 2. Choosing Transaction Providers

Your team might initially choose to use whatever programming framework they picked to do basic transaction processing needs, such as sending emails, image processing, processing credit card transactions, etc… - but soon enough, and certainly as your product grows, it becomes increasingly inefficient to own the cost of building and maintaining Transactional logic.

Email service providers and Image processing providers exist, and you’ll never be able to deal with the complex world of financial transactions systems. The vendors that abstract these services can operate at a scale and experience your team will never reach, nor should they! After all, your business is not competing with these providers, so why would you direct resources and internal investment to do so?

The build-vs-buy decision here, while not obvious at first, becomes clear over time. Your team should only build things that only your team can build. Let others who are best suited and experienced with concerns outside your business serve you at a scale you’ll never accomplish.

In our Marketplace use-case, we will need to handle: Email, Analytics, Payments, Media Processing, etc… While these needs might already be provided by our programming language or framework of choice, it’s best to pick SaaS providers who can fulfill these needs at scale:

• Email: SendGrid, Mailgun, Mailchimp, …
• Analytics: Google Analytics, Amplitude, Mixpanel, …
• Payments: PayPal, Stripe, Chargebee, …
• Media Processing: Cloudinary, Imagekit. Imgix, …

Typically teams will create a comparison matrix, taking into account: cost, features, ease of use, compliance & security needs.

It’s not immediately obvious to teams who have not experienced scaling challenges, why to even outsource something such as email delivery.

Most teams tend to measure how much time it will take them to set up a “hello world” email template and use that as their baseline, however, the full set of requirements and the total cost of ownership for building, maintaining, and scaling something like email over the lifetime of your product is much more costly than that initial estimate, so a comparison matrix needs to take account the total cost of ownership as the baseline and ask the question: can we really build X better than a company dedicated to it AND at a lower total cost?

Step 3. Building the Integration

While abstracting away concerns to vendors and service providers is a wise decision and a clear path to accelerate your team and product development, the reality is that with each cost savings through abstraction comes an additional cost penalty through time & knowledge.

Your team now has to get familiar and integrate with Transaction Layer providers. This takes time, and requires custom built software.

There is virtually no build-vs-buy discussion to be had here, at least not today. You might get lucky and find a Transaction service provider which has partnered with another of your choice, and each can fulfill a portion of your use-case. But this is very rare, and if present, does not satisfy the specific business needs of our Marketplace.

In our Marketplace example, we want to pay the Contractor upon completion of the project, and only after signoff by the Homeowner! Your chosen Payment System (Stripe), Email System (Send Grid), and Analytics System (Amplitude) might magically have some nicely packaged integration, where upon invoicing in Stripe, an Email is sent to the “buyer” and an event is tracked in Amplitude, a very typical “cart checkout” experience that might be standard for an e-commerce workflow … but that is not at all what our Marketplace needs.

We want to pre-collect the funds from the Homeowner, prior to the project starting, then upon compilation, the Contractor will input some additional data in our Marketplace, which will trigger generating an Invoice (in Stripe), then we email the Homeowner (in SendGrid), and track an analytics event of “invoice_sent” (in Amplitude), then wait for the Homeowner to click “approve” before the payment is released to the Contractor (in Stripe), followed by an email notifying both parties that payment is on it’s way (in SendGrid) and a further analytics event “payment_approved” to track (in Amplitude)…

We’re still not done! You have to listen to webhook events from Stripe to track when the payment was actually sent, store the event in Amplitude as “payment_sent” and if it got rejected “payment_rejected” (or any other financial transaction issues) then trigger further emails and followups to address the issue(s).

This is custom code that integrates between 3 Transaction Vendor systems, each with their own proprietary API interface, that your team has to learn, integrate with, and deal with all and any failures / edge cases for each.

Transaction providers do not have any deep integration with each other to facilitate this level of integration. This deeper integration is a cost to your business, and your team has to absorb this work as a normal part of the modern application development model.

While this was a rudimentary example, the reality is modern application development time is spent in majority building custom integrations between 3rd party tools and services.

More common examples:

• Check user’s subscription status and tier every time a paid interaction occurs to verify if they have access
• Payment System: Stripe
• Notify user when another user sends them a chat message
• Chat System: Sendbird
• Email System: SendGrid
• Push Notifications: Airship
• On cart checkout: process payment, notify warehouse, prepare a shipment label, email invoice to customer, update CRM!
• Payment System: Stripe
• Warehouse System: Fulfil.io
• Label System: Shippo
• Email System: SendGrid
• CRM System: SalesForce

There is an increasingly large number of integrations your team has to build, each with custom code to satisfy the unique workflow / business need… yet none of these integrations are what make the core of your product!

In the past these “integrations” were nothing more than a database query to read or insert new information in a monolithic application architecture. Now they are more complex and require custom integration logic due to the cloudification of Infrastructure and Transactional Systems.

Step 4. Building your product (FINALLY!)

Did you forget? You still have not actually built the features that make your Marketplace unique yet! All the time spent in integrating and operating the Transactional Systems, while a necessity, is not really the core focus of your business. It shouldn’t be the highest cost, nor the highest knowledge area your team needs to acquire!

Towards Abstracting the Integration Layer

The state of Modern Application development introduces a lot of advantages and acceleration through abstractions. You don’t need to build a CPU from scratch, nor do you need to build an entire network architecture to start developing.

Yet we have more to do in order to further optimize and abstract away the high cost areas of software development, and one I’m focused on in particular is the Integration Layer.

Cor, a new venture launching soon from myself, Dan Shaw and Jory Burson, will seek to optimize and enable abstractions for application system components that are essential, but not core, to your businesses. Cor will change how software is built by allowing your team to focus on their unique business value. Sign up to our waitlist to get early access and learn more.