Build vs Buy - PaaS to Increase Developer Productivity

What is a PaaS?

A PaaS is a complete development and deployment system with everything you need to create sophisticated, secure, performant, reliable software. Platforms like AWS, GCP and Azure give you the systems and resources you need to create just about anything. Companies like CircleCI add to these platforms by allowing you to run tests before deploying your code to your cloud provider. But, neither CirlceCI or AWS can actually decipher your environment from your repositories and create environments for you. They leave, what is often the hardest part to their users (usually large devops teams) to figure out. More holistic, all-in-one solutions exist like Heroku or Elastic Beanstalk, but these have major drawbacks, which this article will address in subsequent sections. A PaaS for CI/CD needs to include all this functionality while being simple and intuitive to use.

The major cloud platforms will offer you tools and services to help you construct parts and pieces of your PaaS, but none of them offer an easy to use, integrated solution to go from code to environment. Other companies will market themselves as a solution for CI/CD. What most of these companies like CircleCI, Travis, etc don’t do is the actual delivery. They can run your tests against a container and deliver the artifacts to your container registry of choice, but how any of that ends up in developers hands as staging environments or ephemeral environments or ultimately in production, is up to you. The step of taking your local development environment and creating a staging or production version from it is often the most difficult part, and this exercise “is left to the reader”.

Companies like Heroku or products like Elastic Beanstalk from Amazon are the closest to an integrated solution, but they lack extensibility that ultimately limits their usefulness as companies grow. Your company and application will be at the mercy of what they support; as new versions of software or as new software services are introduced in OpenSource you will get to use them when your PaaS decides to support them. An external PaaS is often useful during the early days of development, but as your company grows you will need to migrate off of it. A migration to an in-house solution consisting of custom code integrating off the shelf components and will take a long time and either result in a decrease in product velocity or an increase in expenditure, or both.

All companies that have an Internet presence have a functioning production system, but there comes a time when they need staging and development environments. The purpose of a PaaS is to effortlessly create all the different kinds of environments needed in your software development lifecycle. Many times the environments that aren’t production are classified as ‘staging environments’. If you are unfamiliar with staging environments please read through our article, “What is a staging environment?”. In the beginning a single staging environment may suffice, but eventually you need another, and another, and another, Sooner or later you realize creating these manually is time consuming, error prone and resource intensive and there must be a better way.

A PaaS is a complete development and deployment system with everything you need to create sophisticated, secure, performant, reliable software.”

What do all the biggest tech companies in the world do?

The largest technology companies in the world such as Facebook, Amazon, Netflix, Google, Apple, Microsoft, etc all have spent many millions of dollars and person-hours creating their own internal PaaS as they have (practically) unlimited resources and it’s a strategic advantage. They view being able to deploy thousands of times a day, safely and reliably as a foundational element of their business. The ability to move as fast as possible, even at their scale is one of their most important assets. But, for most companies the previous two statements are not true: you don’t have unlimited resources and spending months or years on something unrelated to your core business is not a strategic advantage. The problem for most companies has always been there are no holistic solutions that can grow with them.

Is building a PaaS your core competency?

Your only option before a product like Release was available was to hire specialists, devops engineers, to configure and develop software to integrate many disparate pieces of software (Spinnaker, Harness, Rancher, etc), using tools like Ansible with a lot of shell scripts. As you might imagine this is unreliable, hard to debug, and not something you want to be spending your limited budgets and time on.

In order to build a PaaS that works reliably, delivers all the features you need, can scale and evolve with your growing business you need a team of specialists. And more often than not their skills do not translate to your core business. Devops engineers carry a salary premium because of their specialized skill sets besides the added disadvantage of not being able to be treated like programming generalists. You don’t move your highly paid, non-general devops engineers to work on the frontend of your shopping cart experience. Once you have a devops team you will constantly be looking for things for them to work on and your internal Paas will be one of their favorite things. This situation is not in the best interest of your company. You are NOT in business to make an internal PaaS.

For example, if you sell shoes on the Internet, you want to spend money and time on developing easier checkout, faster delivery, removing barriers to purchase, running a/b tests on marketing/language/features, etc. Building a highly scalable, fast, reliable CI/CD PaaS doesn’t directly help you do any of these things, but it does help you do whatever it is you do faster, safer and more reliably. All things being equal “Speed Wins”. Being faster than your competitors is a strategic advantage. So what’s a company to do?

Maintenance and evolving the platform

When deciding on Build vs Buy the upfront costs and time of these projects, while difficult to calculate exactly, are understood as important in the decision making process. One can expect to spend 25-50% of the cost of development of your internal platform in maintenance once is up and running. The other thing many people forget to include in their decision making process is evolving the product. As new technology comes out, will you allow your internal tool to age and atrophy? You can't afford to let it wither on the vine, but funding it will always compete with your core business objectives.

These two factors (maintenance and updates) will become your largest expenditures on a project like this after 2-3 years if you are doing it well. So, if it costs $500k to build your internal PaaS, expect to spend at least that much over 2-3 years on maintenance and evolving the platform, or risk having it become obsolete. If your internal PaaS lives for 10 years all these expenditures add up and not just in money but opportunity cost as well.

Build vs. Buy Case Studies

Public Company with 200 engineers

GOAL: Release new functionality, features and products hundreds of times a week vs once a week.
‍
This company decided to build an internal PaaS with an eye towards 100% automated testing, automatic creation of production-like environments on every pull request, and fast and easy automated deployment to production. While ultimately successful it took 10 dedicated resources 18 months to get the basic functionality working. When it was all said and done it was a $4-$5 million dollar project that didn’t bear the fruit of faster more reliable deployments for 18 months. A dedicated team of 7 engineers maintains (> $1 Million) and evolves the platform, but even with that it’s fallen behind by not adapting to new technologies fast enough. They are now stuck with this thing from now until...well forever.

Fast growing startup with 40 engineers

GOAL: Leave Heroku and unify development/staging environment process and production
‍
‍This company was using Heroku for the “Review App” feature (ephemeral staging environments created when a pull request is opened, and removed when a pull request is closed) and the fact they needed little devops experience to get it working. But, as their app became more complicated, working around Heroku’s limitations became a large burden. Review apps were so important to them they bifurcated their developer experience and deployments by implementing production on ECS in AWS and not on Heroku. This isn’t ideal because of cost and the fact that staging/development can never look like production.

They were able to replace Heroku and much of their CircleCI pipeline with Release and in a couple of weeks while retaining “Review App” functionality. All of this was done with a decrease in overall spend, no weird work-arounds, and resulted in extra functionality that their internal PaaS would never possess. Release supports complicated apps and ease of auto-scaling, blue-green deploys, adding new services out-of-the box. They realized trying to keep pace with Release was not something they could afford to do.

Conclusion

While there are many great open source tools and toolkits to help you build an internal PaaS, it’s not for the faint of heart or people concerned with their pocketbook and time. When you factor in needing specialized resources, an internal PaaS is orthogonal to your core business, cost and time required, it rarely makes sense to build a tool like this. The total cost of ownership (TCO) on an internal product like this is massive when you take into account building, maintaining, and constantly evolving it to keep up with modern web, app, and AI applications. What is clear though, is a system that allows you to release as fast as your developers can code gives you a massive advantage over your competitors. The big guys put hundreds of engineers on their internal PaaS for a reason!

Release takes your already functioning standard development environments based on Docker and seamlessly and effortlessly runs them in the cloud. It allows you to have environments ready in seconds based on your developer workflows. It allows you to customize all aspects of your development, staging, performance testing, and any other kind of environments you can imagine. It can handle the simplest of applications to the most advanced, all the way to production. Stop futzing around with IAM roles, k8s, configuration of ALBs, dynamic url creation, adding new services for developers, and get back to building the best version of your company!