At Yext, we’ve been using Go for server-side applications for some time, and over the past year we’ve been exploring using Go for building command-line tools as well. In this post, we’ll be exploring what made us adopt go for command line applications, some of the packages that we’ve found useful along the way, and how we handle distribution to a 90-strong team.

This is a companion post to a talk given at Go Language NYC on January 19th 2017, the slides for which are below.

Going All-In With Go For CLI Apps from Tom Elliott

Why Use Go for CLI?

There are four main factors that have encouraged us to write more and more command-line applications in Go at Yext.

Familiarity

Go is a familiar language for many of Yext’s developers, and as such, it doesn’t require much of a context switch to move from working on a service in Go, to working on a command-line tool in Go.

Code Reuse

With a number of services written in Go, we can make use of some of that code in our command line tools. RPC clients, build targets for protobufs and handy packages we’ve written along the way can all be carried over from the server to the command-line. This has allowed us to create tools that interact with and support our existing services very quickly.

Cross-Platform Support

While most of our developers use Macs, we have a number of Linux users on the team. Go’s cross-platform compilation gives us the benefits of a compiled language (reduced dependencies, etc), with the portability of a scripting language.

Distribution Flexibility

We’ll talk a bit more about this later, but since Go is a compiled language, we have a number of options for distributing our applications.

What We’ve Built

Now that you know a little about why we’re using Go to build command-line tools, let’s have a look at a few that we’ve developed.

srv

srv is an internal tool for building, testing and deploying Yext services. At its core, srv is a wrapper around our existing build/test/deployment tools which greatly simplifies configuration of our Continuous Delivery pipeline. Instead of including long setup scripts in our CD configuration, we can just use individual srv commands like:

$ srv build Pages
$ srv test Pages unit
$ srv publish Pages all release

This also makes it easy to reproduce build steps on developer machines, to help quickly debug issues uncovered in the CD process.

sites-cfg

Pages builds store directory sites based on templates and configuration stored in a Git repo. While working on this repo, sites-cfg provides a quick and easy way for developers to validate their changes without having to push them to the server. It also allows simple querying of existing sites.

$ sites-cfg listsites
$ sites-cfg validate stores.enterpriseclient.com

As we were able to take advantage of our existing server code, sites-cfg was extremely quick and straightforward to build.

Edward

Edward is an open-source tool we developed to simplify our development workflow. Working on a single feature at Yext can involve working with 10-20 different services, and in the past it was necessary to build and launch each of these separately to get an operating local environment.

Edward allows us to build and launch groups of services with a single command, as well as restarting them following changes, and following logs.

$ edward start pages
$ edward stop pages
$ edward tail sites-admin

Since we had to get up and running with our 200+ existing services, there are also built-in tools for auto-generating configuration to launch Go services, Java services built with ICBM, or Docker containers.

Standard Library

The Go Standard Library provides a number of packages that are extremely useful for command-line applications. The following are packages we found particularly useful.

Flags

import "flag"

The flag package allows command-line flags to be declared and parsed with very little code.

var port = flag.Int("port", 8080, "Port number for service")
flag.Parse()

The above defines an integer flag (-port) which defaults to 8080. The variable port is an int pointer that will be populated with the appropriate value when flag.Parse() has been called.

The description text you give your flags is displayed as help text when the user passe the -help flag, giving you some quick instructions right out of the box!

Directory tree walking

import "path/filepath"

Performing an action on every file in a directory tree is a surprisingly common action for command-line tools (just think of all the -r flags you’ve ever used). The filepath package makes this very straightforward. All you need to do is call filepath.Walk with a starting dir and a visitor function.

For example, to find all .c files:

func main() {
  _ = filepath.Walk(os.Args[1], visit)
}

func visit(path string, f os.FileInfo, err error) error {
  if filepath.Ext(path) == ".c" {
    fmt.Println(path)
  }
  return nil
}

Process Execution

import "os/exec"

The exec package provides a simple interface to execute other processes, by creating an instance of Cmd and then calling the appropriate function to run it.

cmd := exec.Command("echo", "hello")
err := cmd.Run()

Above, we run echo hello synchronously and receive an error to check if anything went wrong. Cmd also supports asyncronous execution as well as redirecting stdin, stdout and stderr.

Environment Variables

import "os"

The os package allows getting and setting of environment variables as you would expect:

os.Setenv("MYKEY", "VALUE")
value := os.Getenv("MYKEY")

But it also provides ExpandEnv, which will expand environment variables within a string, very useful when prepping commands for os.exec.

expanded := os.ExpandEnv("$GOPATH/github.com/user/repo")

The above example will expand the string given to the full path of the Go package in our GOPATH.

Platform-Specific Code

Finally, with Go’s cross-platform support, it is often necessary to have separate code to support different platforms. This can be achieved either through build tags, or via a file name. A source file with a name ending in an underscore and a platform name will only compile for that platform, such as dns_windows.go, which will only be compiled for Windows.

As for build tags, in the below example, this file will only be compiled for a platform that is not Linux or MacOS (let’s say these are the only supported platforms for this application). The file contains a call to a function that does not exist, which is given a name that will provide a useful error when trying to build for any other platform.

    // +build !linux,!darwin

    package main
    func init() { macOS_or_Linux_only() }

3rd Party Packages

In addition to the extensive standard library, there are a number of handy 3rd party packages out there that have been contributed by the Go community. Here are a few that we found useful.

CLI

cli is a framework for building command-line applications. It goes beyond the flags and arguments format provided by the flag package, providing a command/subcommand interface that would be familiar to any regular user of git.

The package also provides bash autocompletion support and hidden commands, which don’t appear in the generated help text. We’ve found the hidden command features useful for applications that need to call themselves to modify their environment, or keep a process running in the background.

Notable alternatives to cli include Cobra and Kingpin.

gopsutil

gopsutil is a go port of Python’s psutil, which provides functions to query system resources, send interrupts to running processes and retrieve network state information. We use this package in Edward to monitor forked processes and obtain a list of open ports on the local machine.

Note that gopsutil is still a work-in-progress at the time of writing, and some features are not supported on all platforms.

Distribution

Distribution of a command-line application differs from distribution of server applications in a number of ways. If you’re distributing a SaaS application, in particular, you will have control over the server (or VM, container, etc) where your service will reside, and will push directly to it as needed.

For command line applications, your users will pull a copy of your application when they wish to install, and you will not have control over the environment to which they deploy.

There are three major distribution methods for command-line applications written in Go:

  • go get
  • Build from source
  • Pre-built binary

These each have their own advantages and disadvantages, as we will see.

go get

This is the distribution method that will be most familiar to a Gopher. Go packages (and thus applications) in a public repo can be downloaded and installed with the command:

$ go get <package>

Which will automatically handle installation of required Go dependencies and for main packages, will install to $GOPATH/bin. You can also update existing installed packages using the -u flag.

This is the main distribution method we use for Edward.

The main advantage of this method is that there is zero overhead, you just push your package to a public repo and it’s ready to install. go get also handles dependencies for you, and is cross-platform by default.

However, as go get will always pull the latest commit from your repo, versioning tools can require careful management of your branches and commits. It also limits the complexity of your builds to the equivalent of a single call to go install, which ensures greater predictability in builds, but limits the use of generated code and other techniques. Finally, go get is difficult to use for closed-source projects, requiring some roundabout configuration for private repositories.

Build from source

This is a distribution method that will be familiar to many in the open source community. You include instructions in your repo, and users can download and build manually following those instructions.

This is the method used for sites-cfg. We distribute the source for this tool in our monorepo, and provide build instructions in our internal wiki for developers who need to use it.

Building from source allows a more complicated build process than go get, permitting use of go generate and 3rd party code generation tools. Since the reponsibility to download is on the user, supporting private repositories is straightforward. You can also tailor the build process to a process that users will be familiar with, such as ./configure && make.

This method does require more detailed instruction and knowledge than go get, and the addition of extra build tools can complicate cross-platform distribution and increase the risk of user error.

Pre-built binary

Finally, as Go is a compiled language, you have the option to build your binaries yourself and distribute them directly. With Go’s cross-compiling support, this can be easily automated, and the binaries can be distributed in package managers like homebrew for rapid installation if desired

srv is distributed to Yext developers as a binary stored in our monorepo, which is built as needed when changes are available.

Distributing binaries allows you to do away with the dependency on Go itself, which greatly expands your options for distribution channels (given that you only need to deliver a single file). This also gives you more control over versioning of your application, since you only build a new set of binaries when you’re ready to distribute.

However, this comes with the overhead of building the binaries, whether manually or using CI tools. You also need to configure your distribution channels. There is also the need to make an explicit decision on which platforms you wish to support, which may be a blessing from a support point of view, but is still something you need to take into account.

Update Notification

With (hopefully) many users of your command-line application, there is the problem of letting these users know when a new version of your application is available.

With Edward, we implemented a simple notification mechanism that alerted users of any new releases each time they used the tool. In keeping with GitHub’s use of Git tags for handling releases, we mark each new release of Edward with a tag of the form *..*. Whenever Edward is run, it will pull the latest version tag from the git repo, and provide the user with upgrade instructions if the tag indicates a newer version than the one installed.

A simplified version of this is illustrated below (error handling removed for brevity).

import "github.com/hashicorp/go-version"

func UpdateAvailable(repo, currentVersion) (bool, string, error) {
  output, _ := exec.Command(
                            "git",
                            "ls-remote",
                            "-t",
                            "git://"+repo
                           ).CombinedOutput()
  // Parse tag from output in the form [0-9]+\.[0-9]+\.[0-9]+
  latestVersion, _ = findLatestVersionTag(output)
  remote, _ := version.NewVersion(latestVersion)
  local, _ := version.NewVersion(currentVersion)
  return local.LessThan(remote), remote, nil
}

What Could You Do?

Now you’ve seen the sorts of tools and applications we’ve been developing in Go at Yext, perhaps you’re getting some ideas of your own. You could simplify your development workflow, add a command line interface to an existing open-source package, or just automate a repetitive task.