Waypoint

A couple of weeks ago, I saw a demo of Waypoint, the new tool Hashicorp announced today, that aims to provide an easy, intuitive and customizable “build, deploy and release” workflow.

This post isn’t a tutorial or a guide on how to use the product; the official documentation and tutorials are a good resource to understand how to configure and use the tool. In this post, I explain why I believe that philosophically, the design of Waypoint holds a lot of promise and potential.

What Waypoint Isn’t.

It probably helps to begin this post by spelling out what Waypoint isn’t.

Waypoint is not:

• a build system (like make, npm, maven etc.)
• a package manager (like helm, pip etc.)
• a CI system (like Jenkins, GitHub Actions etc.)
• an Artifact registry (like Artifactory, Docker Registry etc.)
• a runtime format (like OCI images, Docker containers, buildpacks, native binaries, archives etc.)
• a cluster orchestrator or platform (like Kubernetes, EC2, EKS etc.)

This is crucial since Waypoint does actually interface with all of these aforementioned entities, without really being or replacing any one of them.

So What Is Waypoint?

The marketing pitch is that Waypoint is a tool that aims to vastly simplify build and deployment of services. With a series of CLI commands and a declarative means of specifying deployment configurations, users can get their applications up and running on any cloud platform or OSS platform like Kubernetes with minimal effort, without requiring the developer to write custom configuration. As the docs state:

Waypoint was built provide ease of use. You don’t need to write Dockerfiles, YAML, etc. anymore. We have plugins to automatically detect your language, build an image, and deploy. Yes, you have to write some minimal configuration but we’re talking around 15 lines of text for one tool versus hundreds of lines across different tools using different languages.

I’d personally describe Waypoint as a minimalistic, non-prescriptive and heavily customizable build and deployment tool, that offloads most of the heavy-lifting of the complexity inherent in the “build — test — deploy — release” cycle to existing underlying build/test/deployment tools and services, while offering end-users a uniform and declarative interface to describe the “build — test — deploy — release” workflows in a manner that’s “native” to said underlying tools. Put simply, it’s the best kind of “glue infrastructure”.

That’s quite a mouthful and quite a lot to unpack. The rest of this post aims to unpack what this means.

Reimagining the “Build, Test, Deploy and Release” Workflow

I use the term “build and release workflow” and not CI/CD, because not every organization has embraced CI/CD wholesale, even if all organizations do “build and release” software at varying cadences.

Continuous or not, all “build and release” workflows typically comprise of multiple common components:

• Build: This involves producing the binaries that contain the new code changes. Most programming languages have their own suite of build tools. Most compute runtimes have their own supported artifact formats like Docker/OCI images, ELF binaries, ZIP archives, platforms specific artifacts (like AWS Lambda Layers) and more. Any “workflow” tooling that aims to cater to the largest possible developer community needs to accommodate building code written in different languages and producing the preferred format of artifact, preferably using existing language specific build tools.
• Artifact Storage: The artifacts built need to be persisted somewhere. While theoretically any object store can be repurposed for artifact storage, custom artifact registries offer myriad features like encryption at rest, vulnerability scanning, high availability, garbage collection of older artifacts, artifact caching and more. It’s pointless for any new tool to try to reinvent the wheel here, since there already exist well-established products that address this particular problem.
• Deploy: The new artifacts need to be deployed to the production (or test/staging) environment to begin serving new traffic. This often involves “installing” the artifacts on the hosts and starting the appropriate binaries. In my career, I’ve worked with deployment tooling that has run the gamut from simple git based deployments (Heroku) to labyrinthine shell scripts to “agentless”, ssh-based remote execution tools like Fabric/Capistrano/Ansible, to systems that run agents on hosts that “watch” for new deployments to (post-2015) features of cluster orchestrators like Nomad or Kubernetes. It’d be futile for a new tool to reinvent platform/runtime/cloud provider specific deployment mechanisms, since different platforms and cloud providers already provide their own opinionated APIs, custom tooling and workflows to enable deployment of code.
• Release: I’ve always believed that deploys shouldn’t quite be conflated with the “release” of code. Releasing code encompasses traffic management, updating DNS or load balancers, gradual draining of existing services, canary traffic management, blue/green deploys and more. Again, most platforms have opinionated APIs, control planes and paradigms to achieve this.

Since its inception, Hashicorp (no affiliation), has been a company that’s had an uncanny knack of releasing infrastructure products that have (barring a couple of exceptions) found product-market fit. This has allowed the company to largely flourish, without getting into the kind of relicensing shenanigans that’s plagued other open-source (and especially open-core) infrastructure companies.

People who’ve kept abreast of and used Hashicorp products as long as I have will probably realize that Waypoint isn’t Hashicorp’s first foray into the deployment or “workflow” space. Otto was a previous effort which aimed to be a “high-level abstraction to both develop and deploy applications”, and Atlas was another attempt at “providing integrated tooling and a collaborative workflow for both developers and operators; from application code push through to deployment artifact creation, infrastructure provisioning and ultimately application lifecycle management during runtime”.

In retrospect, it’s clear why neither tool quite cut it, and were both subsequently decommissioned or folded into other offerings. In my opinion, both Otto and Atlas tried to solve many disparate problems in one fell swoop. While this approach might’ve worked for Consul, which can be used as a distributed key-value store, a distributed lock service, service mesh, service discovery system and more (and I’ve indeed used Consul for many of these different purposes myself over the years), a “developer workflow” or even a “deployment workflow” that requires entirely opting into the Hashicorp suite of tools wasn’t quite the right product to address the “workflow” conundrum, especially not at a time (2015 onward) when the industry saw a mind-boggling explosion in the number of dedicated infrastructure tools, systems, platforms, APIs, paradigms and ecosystems. When users have a smorgasbord of choices to pick and choose from, it’s arguably harder for omakase products aiming to “unify” disparate concerns to find product-market fit without integrating with a vast array of available solutions.

Which begs the question — how is the third attempt at cracking this problem space different?

Well, for one, Waypoint has a much narrower focus than Otto or Atlas ever did. Waypoint tries to unify the developer experience without being prescriptive or requiring users to opt into using other Hashicorp products like Vagrant or Terraform or Packer.

Secondly, Waypoint seems to have taken a leaf out of Terraform’s book. Much like how Terraform has succeeded in managing to abstract away different cloud providers, SaaS APIs, platforms, open source systems and tools by providing users with a lingua franca (HCL) for defining, configuring, updating, maintaining and evolving raw infrastructure configurations and integrations, Waypoint aims to provide users with an HCL-centric declarative specification and a simple CLI interface to manage and evolve the “build and release” workflow. Like Terraform “providers”, Waypoint “plugins” allow integration with different external APIs and systems.

Third, the architecture of Waypoint is simple enough that a developer prototyping a service can get it up and running in no time. When I was kicking the tires, I used the “local execution” mode to bring up an instance of a service up locally on my laptop. The notion of “workspaces” make it trivial to deploy the same code to different environments. Features like this greatly expedite the “build locally, test locally, deploy to staging, deploy to prod” loop for developers.

Finally, albeit the proof of the scalability truly only lies in the pudding, the architecture of Waypoint is, in theory, flexible enough to be scaled up to serve an entire team or organization of engineers (by productionizing a remote Waypoint server and remote “runners”).

Conclusion

What currently makes deploying code non-trivial isn’t the lack of APIs or systems that address individual problems particularly well. It’s just how much effort it takes to glue all these components together into something workable.

Some of the most successful infrastructure companies of recent times have been “glue infrastructure” companies, whose core product isn’t groundbreaking new technology but “glue technology” offering groundbreaking user interfaces and a phenomenal developer experience.

None of the existing tools in the deployment space are exactly what I’d call “commodity CD” tools. As a matter of fact, tools like Spinnaker are complex enough to warrant their own learning curve. Furthermore, one can argue that a vast number of organizations simply don’t need a tool as complex as Spinnaker for deployments.

Waypoint, to me, is a step in the right direction. It’s still early days, but I’m excited to see how the project will evolve over the course of the next 12–18 months.