A couple of weeks ago, I saw a demo of Waypoint, the new tool Hashicorp announced today, that aims to provide an easy, intuitive and customizable “build, deploy and release” workflow.
This post isn’t a tutorial or a guide on how to use the product; the official documentation and tutorials are a good resource to understand how to configure and use the tool. In this post, I explain why I believe that philosophically, the design of Waypoint holds a lot of promise and potential.
It probably helps to begin this post by spelling out what Waypoint isn’t.
Waypoint is not:
A bash script is commonly a set of commands. There are three standard file descriptors of any command:
There are two commonly used redirection operators:
The most basic example of redirecting the output of a command to a file is:
echo "hello world" 1> foo.txt
This redirects the output of the echo command to a file called
foo.txt. Since 1 is the default file descriptor for the
> operator, it can be omitted. …
Yesterday, I read a phenomenal paper on how disruption free release of services that speak different protocols and serve different types of requests (long lived TCP/UDP sessions, requests involving huge chunks of data etc.) works at Facebook.
One of the techniques used by Facebook is what they call “Socket Takeover”.
Socket Takeover enables Zero Downtime Restarts for Proxygen by spinning up an updated instance in parallel that takes over the listening sockets, whereas the old instance goes into graceful draining phase. The new instance assumes the responsibility of serving the new connections and responding to health-check probes from the L4LB Katran. Old connections are served by the older instance until the end of draining period, after which other mechanism (e.g.,Downstream …
All too often, I see comments or opinion pieces that read like platitudes about how every team should be hiring junior engineers. Let me start off by saying that I’m all for hiring more junior engineers. There seems to be a perennial shortage of competent “senior” talent that seemingly every team is chasing, which often explains the “talent shortage” trope in tech.
Hiring and mentoring junior talent can be a huge competitive advantage. Junior engineers often bring new energy and enthusiasm to a team, are eager to learn and grow, and can be great ambassadors of the team. But there are many tactical challenges involved in successfully onboarding and growing a junior engineer that are hardly ever discussed. …
All too often I see a lot of time and energy expended by people on social media on topics they obviously care a lot about. If the purpose of this energy is advocacy and bringing about real change, then it’s hard to think of a less effective way toward reaching the widest audience one can aim to reach.
By some estimates, only about 20% of Americans are active on Twitter. Of this 20%, most users rarely tweet, but the most prolific 10% create 80% of tweets from adult U.S. users. This means about 80% of tweets are generated by 2% of the U.S. …
I recently read an excellent article in the Amazon Builder’s Library by Clare Liguori which goes into great detail about AWS’s CI/CD architecture. It’s a truly brilliant post and I recommend everyone interested in CI/CD infrastructures read the article. The article covers the gamut from unit testing to integration testing (in production) to staged rollouts in “waves” to automated rollbacks. I guarantee there’s something valuable you’ll learn from the article even if you’re not building the largest distributed systems in the world like AWS does.
The only surprising thing to me about the article was that it stated code review was mandatory before merging a change to the main branch. …
For the past few years, I’ve been publishing a list of my favorite tech talks from the previous year. As always, the usual caveats apply here, viz., this list isn’t comprehensive and excludes talks from many fields (blockchain, AI/ML, web and mobile development) that I do not have much experience in. However, if you do like systems engineering, cloud computing, performance, Linux and low level programming, this list might be right up your street.
In 2019 I was invited to be the track host at QCon New York. QCon is an industry conference which eschews a CFP process in favor of inviting specific individuals to be “track hosts” to curate a track. I was asked to curate the “Modern CS in the Real World” track, which is about applied CS research in the real world. My interpretation was slightly broader, in that I wanted the track to be about the application of computer science concepts, and not just purely CS research, in industry. …
As 2019 draws to a close, I wanted to jot down some thoughts on some of the most important technological adoptions and innovations in tech this past decade. I also look a bit into the future, enumerate a list of pain points and opportunities that can be addressed in the coming decade.
I must add the caveat here that this post doesn’t cover developments in fields like data science, artificial intelligence, frontend engineering and more, since I don’t personally have much experience in these areas.
One of the more welcome trends of the 2010s has been the resurgence of typed languages. True, typed languages had never quite faded into oblivion (C++ and Java were still dominant in 2010 as they are now), but dynamic languages saw a sharp uptick in usage since the Ruby on Rails movement emerged in 2005. The momentum seemed to have a hit crescendo with the open sourcing of Node.js …
Author’s Note: Thanks, as ever, to Fred Hebert, for reading a draft of this post and making some sterling suggestions. This is the third installment in my series on testing distributed systems. The posts in this series are the following:
Testing Microservices, the sane way (published December 2017)
Testing in Production, the safe way (published March 2018)
Testing in Production: the hard parts (published in September 2019)
Testing in Production: the fate of state (published December 2020)
There’s a fair bit of chatter about the virtues of testing in production these days. I’ve myself written about this topic over a year ago. This post isn’t so much of an argument as to why one should be testing in production (the previous posts linked above must’ve made a compelling enough case as to why testing in production is indispensable for certain kinds of systems) than an honest analysis of the challenges inherent in conducting these forms of tests. …
Distributed Tracing is often considered hard to deploy and its value proposition questionable at best. A variety of reasons are attributed to why tracing is “difficult”, an apocryphal concern being the difficulty in instrumenting every component of a distributed system to forward the appropriate headers along with every request. While this might be a valid concern, ultimately it’s not an insurmountable problem. This, incidentally, doesn’t explain why tracing isn’t typically used much by developers, even once it is deployed.
The hard part about distributed tracing isn’t collecting the traces, standardizing on trace propagation and exposition formats, or deciding when, where and how to sample. I’m by no means trivializing these “ingest problems” — indeed, there exist significant technical and (if we’re looking at truly open source standards and protocols) political challenges to overcome to get to the ideal when these can be considered “solved problems”. …