A while ago, I wrote about how file descriptors can be transferred over Unix Domain Sockets between processes, when a parent child relationship doesn’t exist between the two processes. One of the use cases for file descriptor transfer between processes is during deployment of network proxies that handle ingress traffic. However, the APIs offered by the kernel for file descriptor transfer between processes have been awkward to use and riddled with a number of gotchas.
On newer versions of Linux (5.6 and above), a far better API exists to achieve the aforementioned goal.
A running instance of a program is…
A couple of weeks ago, I saw a demo of Waypoint, the new tool Hashicorp announced today, that aims to provide an easy, intuitive and customizable “build, deploy and release” workflow.
This post isn’t a tutorial or a guide on how to use the product; the official documentation and tutorials are a good resource to understand how to configure and use the tool. In this post, I explain why I believe that philosophically, the design of Waypoint holds a lot of promise and potential.
It probably helps to begin this post by spelling out what Waypoint isn’t.
Waypoint is not:
A bash script is commonly a set of commands. There are three standard file descriptors of any command:
There are two commonly used redirection operators:
The most basic example of redirecting the output of a command to a file is:
echo "hello world" 1> foo.txt
This redirects the output of the echo command to a file called
foo.txt. Since 1 is the default file descriptor for the
> operator, it can…
Update 12/31/2020: If you’re on a newer kernel (Linux 5.6+), much of this complexity has been obviated with the introduction of a new system call
pidfd_getfd. Please see the post Seamless File Descriptor Transfer Between Processes with
pidfd_getfdpublished on 12/31/2020 for more details.
Yesterday, I read a phenomenal paper on how disruption free release of services that speak different protocols and serve different types of requests (long lived TCP/UDP sessions, requests involving huge chunks of data etc.) works at Facebook.
One of the techniques used by Facebook is what they call “Socket Takeover”.
Socket Takeover enables Zero…
All too often, I see comments or opinion pieces that read like platitudes about how every team should be hiring junior engineers. Let me start by saying that I’m all for hiring more junior engineers. There seems to be a perennial shortage of competent “senior” talent that seemingly every team is chasing, which often explains the “talent shortage” trope in tech.
Hiring and mentoring junior talent can be a huge competitive advantage. Junior engineers often bring new energy and enthusiasm to a team, are eager to learn and grow, and can be great ambassadors of the team.
But there are…
All too often I see a lot of time and energy expended by people on social media on topics they obviously care a lot about. If the purpose of this energy is advocacy and bringing about real change, then it’s hard to think of a less effective way toward reaching the widest audience one can aim to reach.
By some estimates, only about 20% of Americans are active on Twitter. Of this 20%, most users rarely tweet, but the most prolific 10% create 80% of tweets from adult U.S. users. This means about 80% of tweets are generated by 2%…
I recently read an excellent article in the Amazon Builder’s Library by Clare Liguori which goes into great detail about AWS’s CI/CD architecture. It’s a truly brilliant post and I recommend everyone interested in CI/CD infrastructures read the article. The article covers the gamut from unit testing to integration testing (in production) to staged rollouts in “waves” to automated rollbacks. I guarantee there’s something valuable you’ll learn from the article even if you’re not building the largest distributed systems in the world like AWS does.
The only surprising thing to me about the article was that it stated code review…
For the past few years, I’ve been publishing a list of my favorite tech talks from the previous year. As always, the usual caveats apply here, viz., this list isn’t comprehensive and excludes talks from many fields (blockchain, AI/ML, web and mobile development) that I do not have much experience in. However, if you do like systems engineering, cloud computing, performance, Linux and low level programming, this list might be right up your street.
In 2019 I was invited to be the track host at QCon New York. QCon is an industry conference which eschews a CFP process in favor…
As 2019 draws to a close, I wanted to jot down some thoughts on some of the most important technological adoptions and innovations in tech this past decade. I also look a bit into the future, enumerate a list of pain points and opportunities that can be addressed in the coming decade.
I must add the caveat here that this post doesn’t cover developments in fields like data science, artificial intelligence, frontend engineering and more, since I don’t personally have much experience in these areas.
One of the more welcome trends of the 2010s has been the resurgence of typed…
Author’s Note: Thanks, as ever, to Fred Hebert, for reading a draft of this post and making some sterling suggestions. This is the third installment in my series on testing distributed systems. The posts in this series are the following:
Testing Microservices, the sane way (published December 2017)
Testing in Production, the safe way (published March 2018)
Testing in Production: the hard parts (published in September 2019)
Testing in Production: the fate of state (published 2021)