Introduction

Welcome to the Gofer documentation! This documentation is a reference for all available features and options of Gofer.

• To kick the tires visit: Getting Started
• To read more about Gofer's Feature set visit: Features
• To understand the why of Gofer visit: Philosophy

Gofer: Run short-lived jobs easily.

Summary

Gofer is an opinionated, streamlined automation engine designed for the cloud-native era. It's basically remote code execution as a platform.

Gofer focuses on the "what" and "when" of your workloads, leaving the "how" and "where" to pluggable, more sophisticated container orchestrators (such as K8s or Nomad or even local Docker).

It specializes in executing your custom scripts in a containerized environment, making it versatile for both developers and operations teams. Deploy Gofer effortlessly as a single static binary, and manage it using expressive, declarative configurations written in real programming languages.

Its primary function is to execute short-term jobs like code linting, build automation, testing, port scanning, ETL operations, or any task you can containerize and trigger based on events.

Low Priority

Previously I had discontinued Gofer, but I've been using it personally so much that I figured it would be good to keep it un-archived to push bug fixes and with LLMs being so good now maybe start rolling new features/refactors. But it's pretty low priority for me.

Why?:

• This is my idea of fun.
• Modern solutions...
  • are too complicated to setup and/or manage.
  • lack tight feedback loops while developing pipelines.
  • require you to marry your business logic code to pipeline logic code.
  • use configuration languages (or sometimes worse...their own DSL) as the interface to express what you want.
  • lack extensibility
• It is an experiment to see if theses are all solveable problems in the effort to create a simpler, faster solution.

Features:

• Simple Deployment: Install Gofer effortlessly with a single static binary and manage it through its intuitive command-line interface.
• Language Flexibility: Craft your pipelines in programming languages you're already comfortable with, such as Go or Rust—no more wrestling with unfamiliar YAML.
• Local Testing: Validate and run your pipelines locally, eliminating the guesswork of "commit and see" testing.
• Extensible Architecture: Easily extend Gofer's capabilities by writing your own plugins, backends, and more, in any language via OpenAPI.
• Built-In Storage: Comes with an integrated Object and Secret store for your convenience.
• DAG Support: Harness the power of Directed Acyclic Graphs (DAGs) for complex workflow automation.
• Versioned Pipelines: Every update to a pipeline is kept as a new version, so you can see exactly what each run ran with and roll back to an earlier one.

Demo:

Documentation & Getting Started

If you want to fully dive into Gofer, check out the documentation site!

Install

Extended installation information is available through the documentation site.

Download a specific release:

You can view and download releases by version here.

Download the latest release:

• Linux: wget -O gofer https://github.com/clintjedwards/gofer/releases/latest/download/gofer_amd64_linux_gnu

Build from source:

1. git clone https://github.com/clintjedwards/gofer && cd gofer
2. make build
3. ls ./target/release/gofer

You'll need Rust, plus mdbook and mdbook-linkcheck (cargo install mdbook mdbook-linkcheck) to build the documentation site that Gofer serves at /docs.

The Gofer binary comes with a CLI to manage the server as well as act as a client.

Development

Working on Gofer itself? DEVELOPMENT.md covers running it locally, regenerating the OpenAPI spec and SDKs, editing the docs and how releases are cut.

Authors

• Clint Edwards - Github

This software is provided as-is. It's a hobby project, done in my free time, and I don't get paid for doing it.

If you're looking for the previous Golang version you can find it here.

How does Gofer work?

Gofer works in a very simple client-server model. You deploy Gofer as a single binary to your favorite VPS and you can configure it to connect to all the tooling you currently use to run containers.

Gofer acts as a scheduling middle man between a user's intent to run a container at the behest of an event and your already established container orchestration system.

Workflow

Interaction with Gofer is mostly done through its command line interface which is included in the same binary as the master service.

General Workflow

1. Gofer is connected to a container orchestrator of some sort. This can be just your local docker service or something like K8s or Nomad.
2. It launches its configured extensions (extensions are just containers) and these extensions wait for events to happen or perform some service on behalf of your pipeline.
3. Users create pipelines (by configuration file) that define exactly in which order and what containers they would like to run.
4. Pipelines can be subscribed to extensions so they run automatically.
5. Either by extension or manual intervention a pipeline run will start and schedule the containers defined in the configuration file.
6. Gofer will collect the logs, exit code, and other essentials from each container run and provide them back to the user along with summaries of how that particular run performed.

Extension Implementation

1. When Gofer launches the first thing it does is create the extension containers the same way it schedules any other container.
2. The extension containers are all small REST API web services that are implemented using a specific interface provided by the sdk.
3. Gofer passes the extension a secret value that only it knows so that the extension doesn't respond to any requests that might come from other sources.
4. After the extension is initialized Gofer sends it every pipeline subscription it has (created with gofer pipeline subscribe).
5. The extension then takes note of this subscription and waits for the relevant event to happen.
6. When the event happens it figures out which pipeline should be alerted and sends an event to the main Gofer process.
7. The main gofer process then starts a pipeline run on behalf of the extension.

Glossary

• Namespace: A group of pipelines, usually one per team or project. Permissions and global secrets are scoped by namespace. Everything goes in the default namespace unless you say otherwise. See Namespaces.

• Pipeline: A pipeline is a collection of tasks that can be run at once. Pipelines can be defined via a pipeline configuration file. Once you have a pipeline config file you can create a new pipeline via the CLI (recommended) or API.

• Pipeline config (version): One saved copy of a pipeline's configuration. Each gofer up stores a new version (v1, v2, ...) instead of replacing the old one. See Updating Pipelines.

• Deployment: Making a pipeline config version the live one, which is the version new runs use. gofer up deploys by default.

• Run: A run is a single execution of a pipeline. A run can be started automatically via extensions or manually via the API or CLI

• Task: A task is the lowest unit in Gofer. It is a small abstraction over running a single container. Through tasks you can define what container you want to run, when to run it in relation to other containers, and what variables/secrets those containers should use.

• Task Execution: A task execution is the programmatic running of a single task container. Referencing a specific task execution is how you can examine the results, logs, and details of one of your tasks on any given run.

• Extension: An extension adds functionality to pipelines. Extensions start up with Gofer as long running containers, and pipelines subscribe to them to use them; most commonly to run automatically on a schedule or when something happens elsewhere.

• Subscription: A pipeline's link to an extension, along with the settings the extension needs for that pipeline (like how often to run). Created with gofer pipeline subscribe. See Running Pipelines Automatically.

• Pipeline secret / Global secret: Values kept in Gofer's secret store for tasks to use. Pipeline secrets belong to one pipeline. Global secrets are managed by admins and can be shared with any namespaces they allow.

• Pipeline object / Run object: Values kept in Gofer's object store. Pipeline objects stay until the pipeline hits its object limit; run objects belong to one run and are deleted after a few more runs. Run objects are how tasks pass values to each other.

Feature Guide

Write your pipelines in a real programming language.

Other infrastructure tooling tried configuration languages(yaml, hcl).... and they kinda suck¹. The Gofer CLI allows you to create your pipelines in a fully featured programming language. Pipelines can currently be written in Go or Rust².

DAG(Directed Acyclic Graph) Support.

Gofer provides the ability to run your containers in reference to your other containers.

With DAG support you can run containers:

• In parallel.
• After other containers.
• When particular containers fail.
• When particular containers succeed.

OpenAPI compatible

Gofer uses OpenAPI to construct its API surface. This means that Gofer's API is easy to use, well defined, and can easily be developed for in any language.

The use of OpenAPI gives us three main advantages:

1. The most up-to-date API contract can always be found by reading the openapi spec files included in the source.
2. Developing against the API for developers working within Golang/Rust simply means importing the provided sdk.
3. Developing against the API for developers not working within the Go/Rust language means simply generating an sdk using your language specific tooling

Namespaces

Gofer allows you to separate out your pipelines into different namespaces, allowing you to organize your teams and set permissions based on those namespaces.

Extensions

Extensions are the way users can add extra functionality to their pipelines. For instance the ability to automate their pipelines by waiting on bespoke events (like the passage of time).

Extensions are nothing more than containers themselves that talk to the main process when they require activity.

Gofer out of the box provides some default extensions like cron and interval. But even more powerful than that, it accepts any type of extension you can think up and code using the included extension sdk.

You can view how extensions work by visiting the sample extension 'interval'

Extensions are brought up alongside Gofer as long-running containers that it launches and manages.

Object Store

Gofer provides a built in object store you can access with the Gofer CLI. This object store provides a caching and data transfer mechanism so you can pass values from one container to the next, but also store objects that you might need for all containers.

Secret Store

Gofer provides a built in secret store you can access with the Gofer CLI. This secret store provides a way to pass secret values needed by your pipeline configuration into Gofer.

Events

Gofer provides a list of events for the most common actions performed. You can view this event stream via the Gofer API, allowing you to build on top of Gofer's actions and even using Gofer as a trigger for other events.

External Events

Gofer allows extensions to consume external events. This allows for extensions to respond to webhooks from favorite sites like Github and more.

Pluggable Everything

Gofer plugs into all your favorite backends your team is already using. This means that you never have to maintain things outside of your wheelhouse.

Whether you want to schedule your containers on K8s or AWS Lambda, or maybe you'd like to use an object store that you're more familiar with in minio or AWS S3, Gofer provides either an already created plugin or an interface to write your own.

Troubleshooting Gofer

This page provides various tips on how to troubleshoot and find issues/errors within Gofer.

Debugging runs

When a run fails, start with:

gofer run debug <pipeline_id> <run_id>

It prints a timeline of every task in the run, then details on each task that failed, was cancelled, or was skipped, including the last few lines of output from the failed ones. Skipped tasks show which dependency wasn't met, and tasks that failed before starting show why (like a secret or object that doesn't exist).

From there:

• gofer task logs <pipeline_id> <run_id> <task_id> shows a task's full output. Add --tail 50 to only see the end of it, which is much faster for long running tasks. Reading task logs covers searching and filtering them with less.
• gofer task get <pipeline_id> <run_id> <task_id> shows the exact variables the task's container was given, and how long the task spent in each phase.
• gofer task attach <pipeline_id> <run_id> <task_id> opens a shell in a task's container while it's still running, if the server allows it (api.allow_task_attach).

Slow runs

When a run takes longer than it should, the timeline in gofer run debug (and on the run's page in the web UI) splits each task into the time it spent waiting on its dependencies, setting up, and running. Setting up is everything between a task being ready to go and its container running: resolving secrets and objects, pulling the image, and starting the container. gofer task get breaks that down further, so a slow image pull stands out.

If the run itself took a while to begin, it was probably queued behind the pipeline's parallelism limit; the run's start time shows how long it waited.

Debugging the CLI

If the CLI is acting strangely, gofer context shows which server it's talking to and that server's version, who your token belongs to, which config file was read and where each setting came from. It still prints the local half when the server can't be reached.

Debugging extensions

Extensions are simply long running containers that internally wait for an event to happen and then communicate with Gofer through its API.

There are a few avenues to debug extensions:

• gofer extension list shows each extension's state. When one failed to start, the reason column says why; a missing global secret or a setting the extension doesn't take shows up there.
• gofer extension reload --dry-run rereads Gofer's config and shows which extensions differ from it, and which entries have problems, without changing anything.
• gofer extension logs <id> will stream an extension's logs.
• Each extension has a /api/debug endpoint that dumps debug information about that extension.

Debugging Tasks

When tasks aren't working quite right, it helps to have some simple tasks that you can use to debug. Gofer provides a few of these to aid in debugging.

Gofer's Philosophy

Short version

Gofer focuses on the usage of containers to run workloads that don't belong as long-running applications. At its core, it is a remote code execution platform. The ability to run containers easily is powerful tool for users who need to run various short-term workloads and don't want to learn a whole new paradigm to do it. Gofer respects the user by inviting them to follow a golden path for how to do things, but also giving them the power to leave it when necessary.

Gofer's core values are:

• Simple as possible, but no simpler.
• Easy to use. Don't get in the way of the user.
• Fast. Computers wait on people not the other way around.

Long version

If you'd like to read more about why Gofer exists and the journey in creating it, you can do so here

Getting Started

Let's start by setting up our first Gofer pipeline!

This guide runs everything on your own machine, so all you need is Linux, Docker, and Go (or Rust, if you'd rather write your pipeline in that). By the end you'll have:

1. Installed Gofer.
2. Started the Gofer server locally in development mode.
3. Written a pipeline configuration.
4. Registered that pipeline with Gofer.
5. Started a run and looked at its results.

Installing Gofer

Gofer comes as an easy to distribute pre-compiled binary that you can run on your machine locally, but you can always build Gofer from source if need be.

Pre-compiled (Recommended)

You can download the latest version for linux here:

wget -O gofer https://github.com/clintjedwards/gofer/releases/latest/download/gofer_amd64_linux_gnu
chmod +x gofer

From Source

You'll need Rust, plus mdbook and mdbook-linkcheck (cargo install mdbook mdbook-linkcheck) to build the documentation site that Gofer serves at /docs.

git clone https://github.com/clintjedwards/gofer && cd gofer
make build
ls ./target/release/gofer

Running the Server Locally

Gofer is deployed as a single static binary allowing you to run the full service locally so you can play with the internals before committing resources to it. Spinning Gofer up locally is also a great way to debug "what would happen if?" questions that might come up during the creation of pipeline config files.

Install Gofer

Install Docker

The way in which Gofer runs containers is called a Scheduler. When deploying Gofer at scale we can deploy it with a more serious container scheduler (Nomad, Kubernetes) but for now we're just going to use the default local docker scheduler included. This simply uses your local instance of docker instance to run containers.

But before we use your local docker service... you have to have one in the first place. If you don't have docker installed, the installation is quick. Rather than covering the specifics here you can instead find a guide on how to install docker for your operating system on its documentation site.

Gofer talks to Docker as the user you run it as, so make sure that user can run containers. docker ps working without sudo is a good sign; on Linux that usually means being in the docker group.

Start the server

By default the Gofer binary is able to run the server in development mode. Simply start the service by:

gofer service start

The server runs in the foreground and prints its logs, so leave it running and use a second terminal for the rest of this guide.

To check that it's up, open http://localhost:8080 in your browser to see Gofer's web UI. The CLI is already set up to talk to a local server, so gofer pipeline list should work too (it'll be empty for now).

What development mode means

With no configuration, Gofer starts with settings meant for trying it out on your own machine:

• Authentication is off. Anyone who can reach port 8080 can do anything, so don't run it like this on a shared machine or network.
• Everything is stored in /tmp. Your pipelines, runs, secrets, and objects are lost when your machine restarts.
• Secrets use a placeholder encryption key that's the same for every development install.

That's all fine for following this guide. When you're ready to run Gofer for real, the server configuration page covers turning these off.

Create Your First Pipeline Configuration

Before you can start running containers you must tell Gofer what you want to run. To do this we create what is called a pipeline configuration.

The creation of this pipeline configuration is very easy and can be done in either Golang or Rust. This allows you to use a fully-featured programming language to organize your pipelines, instead of dealing with YAML mess.

Let's Go!

As an example, let's just copy a pipeline that has been given to us already. We'll use Go as our language, which means you'll need to install it if you don't have it. The Gofer repository gives us a simple pipeline that we can copy and use.

Let's first create a folder where we'll put our pipeline:

mkdir /tmp/simple_pipeline

Then let's copy the Gofer provided pipeline's main file into the correct place:

cd /tmp/simple_pipeline
wget https://raw.githubusercontent.com/clintjedwards/gofer/main/examplePipelines/go/simple/main.go

This should create a main.go file inside our /tmp/simple_pipeline directory.

Lastly, let's initialize the new Golang program:

To complete our Go program we simply have to initialize it with the go mod command.

go mod init test/simple_pipeline
go mod tidy

What is in it?

Here is main.go with its long description shortened:

package main

import (
	"log"

	sdk "github.com/clintjedwards/gofer/sdk/go/config"
)

func main() {
	err := sdk.NewPipeline("simple", "Simple Pipeline").
		Description("This pipeline shows off a very simple Gofer pipeline...").
		Tasks(
			sdk.NewTask("simple-task", "ubuntu:latest").
				Description("This task simply prints our hello-world message and exits!").
				Command("echo", "Hello from Gofer!").Variables(map[string]string{"test": "sample"}),
		).Finish()
	if err != nil {
		log.Fatal(err)
	}
}

A pipeline consists of a few parts:

• An id and a name. simple is the id you'll use in commands; "Simple Pipeline" is the name people see.
• A description so others know what it's for.
• One or more tasks. Each task is a container to run. This one runs ubuntu:latest, prints a message, and gets an environment variable TEST=sample.
• Finish(), which checks the pipeline for mistakes and prints it out.

Try it

A pipeline config is an ordinary program, so you can run it yourself:

go run .

It prints your pipeline as JSON. That JSON is what Gofer actually receives; in the next step, gofer up runs this program for you and uploads what it prints. If something is wrong with the pipeline, like a task that depends on one that doesn't exist, go run . tells you here before anything reaches Gofer.

Register your pipeline

Now we will register your newly created pipeline configuration with Gofer!

More CLI to the rescue

From your terminal, let's use the Gofer binary to run the following command, pointing Gofer at your newly created pipeline folder:

gofer up /tmp/simple_pipeline

Examine created pipeline

It's that easy!

The Gofer command line application uses your local Golang compiler to compile and run your pipeline configuration, then uploads the result to Gofer. (For a Rust pipeline it uses cargo instead.)

You should have received a success message and some suggested commands:

 ✓ Registered pipeline: [simple] 'Simple Pipeline' v1

  View details of your pipeline: gofer fetch simple
  Start a new run: gofer pipeline run simple

We can view the details of our new pipeline by running:

gofer fetch simple

If you ever forget your pipeline ID you can list all pipelines in your namespace by using:

gofer pipeline list

Start a Run

Now that we've set up Gofer, defined our pipeline, and registered it we're ready to actually run our containers.

Press start

gofer pipeline run simple

What happens now?

When you start a run Gofer will attempt to schedule all your tasks according to their dependencies onto your chosen scheduler. In this case that scheduler is your local instance of Docker.

Your run should be chugging along now!

Watch it in the browser

The easiest way to follow a run is the web UI:

gofer web simple

This opens the pipeline's page. Click the newest run to see each task's state and logs as they happen, or use gofer web simple <run_id> to go straight to a run.

Or from the terminal

gofer fetch is a shortcut for looking things up. Each argument goes one level deeper (pipeline, then run, then task), and a + on the end lists everything at the next level instead.

View details about the pipeline:

gofer fetch simple

List the pipeline's runs:

gofer fetch simple +

View details about run 1:

gofer fetch simple 1

List the tasks that ran in run 1:

gofer fetch simple 1 +

View the details of one task, including the environment variables it was given:

gofer fetch simple 1 simple-task

Show a task's output:

gofer task logs simple 1 simple-task

Add --tail 50 to see only the end of a long log. Reading task logs has more, like searching through them.

When something goes wrong

If a run fails, gofer run debug simple 1 shows what happened: every task's result, why any were skipped, and the last lines of output from the ones that failed. The troubleshooting page has more.

What's Next?

Anything!

1. Make your pipeline run on its own, on a schedule or when something happens on Github.
2. Spruce up your pipeline definition with task dependencies, secrets, and objects.
3. Learn what happens when you update a pipeline.
4. Read the common patterns and best practices for passing files between tasks, cloning your repository, handling secrets, and more.
5. Keep playing with Gofer locally; gofer --help lists every command.
6. Learn more about Gofer terminology.
7. Deploy Gofer for real. Pair it with your favorite scheduler and start using it to automate your jobs.

Common Patterns and Best Practices

These are the habits that make Gofer pipelines easy to run and easy to trust. Most of them come from the same facts: every task is its own container, possibly on a different machine, and Gofer only knows what you tell it. It also doesn't tie pipelines to a git repository.

Here's everything at a glance:

Writing Tasks

• Make tasks safe to run twice.
• Take configuration from environment variables.
• Pin your image tags.
• Keep your images lean.
• Fail loudly.

Designing Pipelines

• Keep things simple.
• Use Any for cleanup and notifications.
• Limit pipelines that can't overlap.
• Keep your pipeline config next to your code.

Passing Data Between Tasks

• Small values
• Files and directories
• Things that trip people up

Working with Git Repositories

• Getting access
• Getting the code

Secrets and Access

• Prefer pipeline secrets over global ones.
• Only give API tokens to tasks that need them.

Writing Tasks

Make tasks safe to run twice.

Write your code in whatever language you want, but make it idempotent. Runs get retried by hand, started twice by accident, or cancelled halfway through, and Gofer doesn't guarantee a task only ever runs once. A deploy task that checks what's already there before changing anything is much easier to live with than one that assumes a clean slate.

Cancelling is part of this. When a task is cancelled, Gofer asks its container to stop (Docker sends SIGTERM) and then kills it if it's still running after task_execution_stop_timeout_seconds, 5 minutes by default. If your task holds a lock or is halfway through a write when that happens, catch the signal and clean up, or make sure the next run can pick up from a half finished state.

Give long tasks a timeout.

A task that hangs holds its run open, and a slot in the pipeline's parallelism, until something stops it. Give tasks a timeout close to how long they should actually take, with Timeout(10 * time.Minute) in Go or .timeout(Duration::from_secs(600)) in Rust. The clock starts when the container starts, so pulling the image doesn't count.

When a task runs past its timeout, Gofer stops it the same way it does for a cancel, then marks the task and the run as failed with the reason timeout. The description says which limit was hit: the task's own, or the server's task_execution_max_timeout_seconds (24 hours by default). The server's limit also applies to tasks that don't set a timeout and caps any that ask for longer.

Take configuration from environment variables.

Gofer passes everything into a task as environment variables, so your code should read its input and configuration from them, the way 12-factor apps do. That keeps one image usable across pipelines and lets you override a value for a single run with gofer pipeline run my-pipeline -v KEY=VALUE. See task environment variables for where variables come from and which one wins.

Pin your image tags.

Gofer keeps every version of your pipeline config and records which version each run used, so you can always look back at what ran. That only holds if the images are pinned too. With :latest, the config says the same thing on Monday and Friday while the code inside the image changed in between.

The Docker scheduler also only pulls an image when the host doesn't have it yet. So without AlwaysPullNewestImage(true), a :latest task runs whatever version that host happened to pull last, and two hosts can run different code for the same task. Prefer a version tag (my-org/deployer:1.4.2) or a digest (my-org/deployer@sha256:...). Save AlwaysPullNewestImage for images where you really do want the newest build every time, like your own tooling images, and accept that each run pays for the pull.

Keep your images lean.

There's no guarantee a task lands on a machine that already has its image. Large images take a long time to download and a lot of disk space, and that time comes out of every run that has to pull them. Start from a small base image and only install what the task needs.

Fail loudly.

Gofer decides whether a task succeeded from its exit code, and task dependencies decide what runs next from that. A script that swallows an error and exits 0 lets the publish step run after tests that actually failed. Start shell scripts with set -e (or set -eu), and make sure your own programs exit non-zero when something goes wrong.

Designing Pipelines

Keep things simple.

You could build a huge graph of containers that all depend on each other. But the main theme of Gofer is simplicity. Before splitting work into separate tasks, think about whether it actually helps. Separate tasks are good for running things in parallel, using different images, or reacting differently to one step failing. They cost you a container start each and you have to pass files between them through the object store. When in doubt, pick whichever version is easier for the next person to understand.

Use Any for cleanup and notifications.

A task that depends on another with RequiredParentStatusSuccess is skipped when that parent fails, and so is everything below it. For tasks that should run no matter what, like tearing down a test environment or posting the result to chat, depend on the parent with RequiredParentStatusAny. Use RequiredParentStatusFailure for things that should only happen when something breaks. gofer run debug <pipeline> <run> shows why each task was skipped when the graph doesn't do what you expected.

Limit pipelines that can't overlap.

By default a pipeline can have many runs going at once. That's fine for tests, but not for a deploy, or anything that updates a shared pipeline object like a cache, where the last run to finish wins. Set Parallelism(1) on those pipelines so new runs wait for the current one to finish.

Keep your pipeline config next to your code.

Pipeline configs are code, so keep them in the repository they build and review them like any other change. Run gofer up from the commit you want live, and roll back with gofer pipeline config deploy or by running gofer up again from an older commit. If you want a new version stored in Gofer without it going live yet, use gofer up --deploy false. See updating pipelines for how versions and deployments work.

Passing Data Between Tasks

Tasks don't share a filesystem, so use the object store instead. It's Gofer's replacement for a shared workspace.

• Run objects live for a single run (plus a few after it, see run_object_expiry_runs). Use them for handing a value or a file from one task to the tasks after it.
• Pipeline objects last across runs until the pipeline hits pipeline_object_limit. Use them for caches.

A task stores objects through Gofer's API, so give it InjectAPIToken(true). Gofer sets GOFER_API_BASE_URL, GOFER_NAMESPACE, GOFER_PIPELINE_ID and GOFER_RUN_ID on every task, so the request needs no other setup. The Gofer CLI reads the same variables, along with GOFER_TOKEN, so if your image has the gofer binary it works with no setup either.

Small values

For something like a version number, have the first task store it as a run object and have the tasks after it read it as an environment variable:

const storeVersion = `
VERSION="1.4.2" # Stand in for something your task works out, like a build number.
curl -sf -X POST \
  -H "Authorization: Bearer $GOFER_TOKEN" \
  -H "gofer-api-version: v0" \
  --data-binary "$VERSION" \
  "$GOFER_API_BASE_URL/api/namespaces/$GOFER_NAMESPACE/pipelines/$GOFER_PIPELINE_ID/runs/$GOFER_RUN_ID/objects/version"
`

err := sdk.NewPipeline("release", "Release").
    Tasks(
        sdk.NewTask("work-out-version", "curlimages/curl:latest").
            InjectAPIToken(true). // Gives the task a GOFER_TOKEN that can write this pipeline's objects.
            Command("sh", "-c", storeVersion),

        sdk.NewTask("publish", "alpine:latest").
            DependsOn("work-out-version", sdk.RequiredParentStatusSuccess).
            Variables(map[string]string{"VERSION": sdk.RunObject("version")}).
            Command("sh", "-c", "echo publishing version $VERSION"),
    ).Finish()

With the gofer binary, echo "$VERSION" | gofer run object put version @ does the same thing as the curl call.

Files and directories

For anything bigger, or anything binary, upload it in one task and download it in the next:

# Upload a directory as a run object.
tar -czf - ./dist | curl -sf -X POST \
  -H "Authorization: Bearer $GOFER_TOKEN" -H "gofer-api-version: v0" --data-binary @- \
  "$GOFER_API_BASE_URL/api/namespaces/$GOFER_NAMESPACE/pipelines/$GOFER_PIPELINE_ID/runs/$GOFER_RUN_ID/objects/dist"

# Download it in a later task.
curl -sf -H "Authorization: Bearer $GOFER_TOKEN" -H "gofer-api-version: v0" \
  "$GOFER_API_BASE_URL/api/namespaces/$GOFER_NAMESPACE/pipelines/$GOFER_PIPELINE_ID/runs/$GOFER_RUN_ID/objects/dist" \
  | tar -xzf -

The gofer binary does the tar work for you. Inside a task it picks up the pipeline and run from the environment, so all you need is the key and the directory:

gofer run object put dist ./dist           # Uploads the directory as a tarball.
gofer run object get dist --extract ./dist # Downloads and unpacks it.

Things that trip people up

• The reading task has to depend on the writing one with DependsOn. Tasks run in parallel by default, so without it the object might not exist yet. For variables, Gofer reads the object right before the task starts, so the task fails before its container even runs.
• Download files through the API, not as variables. sdk.RunObject("key") works well for small text values, but it converts the object to text, so it'll mangle a tarball or any other binary file.
• Pipeline objects can disappear. The oldest ones get deleted once the pipeline hits pipeline_object_limit, so write tasks that read them to still work when they're missing. To store one, use the same requests with /pipelines/$GOFER_PIPELINE_ID/objects/<key> in place of the run path, and sdk.PipelineObject to read it.
• If tasks can't reach Gofer, check server.extension_address. That's where GOFER_API_BASE_URL comes from, since tasks and extensions sit on the same container network.

Working with Git Repositories

Gofer doesn't know about your repository, so your tasks clone it themselves. That's two problems: getting access to the repo, and getting the code into each task that needs it.

Getting access

Use a read-only deploy key per repository, stored as a pipeline secret. Deploy keys are scoped to a single repo, so a leaked key only exposes that repo.

ssh-keygen -t ed25519 -N "" -f ./deploy_key       # Add deploy_key.pub to the repo as a read-only deploy key.
gofer secret pipeline put my-pipeline deploy_key < ./deploy_key
rm ./deploy_key

Then hand it to the task as a variable and point git at it:

const setupGit = `
apk add --no-cache git openssh-client
mkdir -p ~/.ssh
printf '%s\n' "$DEPLOY_KEY" > ~/.ssh/deploy_key   # Secrets are stored without the trailing newline ssh requires.
chmod 600 ~/.ssh/deploy_key
printf '%s\n' "$KNOWN_HOSTS" > ~/.ssh/known_hosts
export GIT_SSH_COMMAND="ssh -i ~/.ssh/deploy_key -o UserKnownHostsFile=$HOME/.ssh/known_hosts"
`

sdk.NewTask("build", "alpine:latest").
    Variables(map[string]string{
        "DEPLOY_KEY":  sdk.PipelineSecret("deploy_key"),
        "KNOWN_HOSTS": sdk.PipelineObject("known_hosts"),
    }).
    Command("sh", "-c", setupGit+"git clone git@github.com:my-org/my-repo.git")

Pin the host's SSH keys instead of turning off host checking. It's tempting to add -o StrictHostKeyChecking=no, but then anyone who can intercept the connection can feed your task whatever code they like. Copy the host's published keys (Github lists theirs in its docs under "Github's SSH key fingerprints") into a pipeline object once, as above, or bake them into your image:

gofer pipeline object put my-pipeline known_hosts ./github_known_hosts

To share a key across pipelines, store it as a global secret limited to the namespaces that need it, as in prefer pipeline secrets over global ones, and use sdk.GlobalSecret("team_a_deploy_key") in place of sdk.PipelineSecret. Keep shared keys read-only, since every pipeline in those namespaces can read them.

HTTPS with a token works too, and skips the SSH setup: git clone https://x-access-token:$GITHUB_TOKEN@github.com/my-org/my-repo.git. The tradeoff is that personal access tokens are usually tied to someone's account, so they break when that person leaves and often reach more repos than the pipeline needs. Prefer fine-grained tokens limited to one repo.

Getting the code

Start with the first option and only move down the list when it's actually too slow.

1. Fetch just the commit you need. This is fast enough for most repositories and it's the most correct option: you build exactly the commit that triggered the run, not whatever the branch points to by the time the task starts. The Github extension passes that commit in as GOFER_EXTENSION_GITHUB_PULLREQUEST_HEAD_SHA for pull requests and GOFER_EXTENSION_GITHUB_HEAD_COMMIT_ID for pushes.

git init -q repo && cd repo
git remote add origin git@github.com:my-org/my-repo.git
git fetch -q --depth 1 origin "$GOFER_EXTENSION_GITHUB_HEAD_COMMIT_ID"
git checkout -q FETCH_HEAD

For large repos, add --filter=blob:none to download file contents only as they're needed, or use git sparse-checkout to check out only the directories the task uses. Fetching a commit by its hash works on Github and Gitlab; some self-hosted git servers need uploadpack.allowReachableSHA1InWant turned on first.

2. Clone once per run and share it. When several tasks need the code, have one task fetch it and upload it as a run object, then have the others depend on that task and download it, as shown in files and directories. Every task sees the same tree and your git host only gets hit once per run.

3. Cache the repository across runs. Keep a tarball of the repo as a pipeline object. Each run downloads it, runs git fetch to pick up only what changed, then uploads it again. A few things to know:

• Upload over the old one by adding ?force=true to the request URL, or --force with the CLI.
• If two runs update the cache at the same time, the last one wins. That's fine for a cache, but don't keep anything there you can't rebuild. Parallelism(1) avoids it entirely.
• Pipeline objects past pipeline_object_limit get deleted, oldest first, so write the clone task to fall back to a fresh clone when the cache is missing.

4. Bake it into the image. For repos that rarely change, like shared scripts or tooling, copying them into the task's image at build time skips the download entirely.

5. Run a git caching proxy. For very large repos, or many runs close together, a caching proxy near your container hosts (like git-cache-http-server) serves clones locally and only goes back to the real host for new commits.

Secrets and Access

Prefer pipeline secrets over global ones.

Pipeline secrets can only be used by the pipeline they belong to, so a mistake stays contained. Reach for a global secret only when several pipelines really need the same value, and limit it to the namespaces that need it:

gofer secret global put team_a_deploy_key -n "team-a-.*" < ./deploy_key

Either way, pass secrets in with sdk.PipelineSecret or sdk.GlobalSecret instead of writing them into the pipeline config. Configs are stored and shown as written, so anything typed into one directly is visible to anyone who can read the pipeline.

Only give API tokens to tasks that need them.

InjectAPIToken(true) gives a task a token for Gofer's API, which it needs to store objects or use the gofer CLI. Turn it on only for those tasks. A task that just runs tests doesn't need to be able to talk to Gofer, and every token you hand out is one more thing that can leak from a log or a compromised dependency.

Pipeline Configuration

A pipeline is a directed acyclic graph of tasks that run together. A single execution of a pipeline is called a run. Gofer allows users to configure their pipeline via a configuration file written in Golang or Rust.

The general hierarchy for a pipeline is:

namespace
   \_ pipeline
         \_ run
             \_ task

That is to say:

1. A namespace might contain multiple pipelines.
2. A pipeline is made up of tasks/containers to be run.
3. Every time a pipeline is executed it is done through the concept of a run which makes sure the containers are executed properly.

SDK

Creating a pipeline involves using the SDK currently written in Go or Rust. A pipeline config is just a small program: it describes the pipeline with the SDK and prints it as JSON, and gofer up builds it, runs it, and sends that JSON to Gofer.

• Go: github.com/clintjedwards/gofer/sdk/go/config. Every function is documented on pkg.go.dev.

• Rust: the gofer_sdk crate. It isn't published on crates.io, so depend on it from Git, using the tag that matches your Gofer version:

  [dependencies]
  gofer_sdk = { git = "https://github.com/clintjedwards/gofer", tag = "v<gofer version>" }
  

  cargo doc --open shows its documentation.

Both SDKs have the same functions; Go uses CamelCase names and Rust uses snake_case.

Small Walkthrough

To introduce some of the concepts slowly, lets build a pipeline step by step. We'll be using Go as our pipeline configuration language and this documentation assumes you've already set up a new Go project and are operating in a main.go file. If you haven't you can set up one following the guide instructions.

A Simple Pipeline

Every pipeline is initialized with a simple pipeline declaration. It's here that we will name our pipeline, giving it a machine referable ID and a human referable name.

err := sdk.NewPipeline("simple", "My Simple Pipeline")

It's important to note here that while your human readable name ("My Simple Pipeline" in this case) can contain a large array of characters the ID can only contain letters, numbers, and hyphens, and has to be 3 to 32 characters long. Any other characters will result in an error when attempting to register the pipeline, due to each id needing to be URL safe.

Add a Description

Next we'll add a simple description to remind us what this pipeline is used for.

err := sdk.NewPipeline("simple", "My Simple Pipeline").
        Description("This pipeline is purely for testing purposes.")

The SDK uses a builder pattern, which allows us to simply add another function onto our Pipeline object which we can type our description into.

Add a task

Lastly let's add a task(container) to our pipeline. We'll add a simple ubuntu container and change the command that gets run on container start to just say "Hello from Gofer!".

err := sdk.NewPipeline("simple", "My Simple Pipeline").
        Description("This pipeline is purely for testing purposes.").
        Tasks(sdk.NewTask("simple-task", "ubuntu:latest").
			Description("This task simply prints our hello-world message and exits!").
			Command("echo", "Hello from Gofer!"),
    )

We used the Tasks function to add multiple tasks and then we use the SDK's NewTask function to create a task. You can see we:

• Give the task an ID, much like our pipeline earlier.
• Specify which image we want to use.
• Tack on a description.
• And then finally specify the command.

To tie a bow on it, we add the .Finish() function to specify that our pipeline is in its final form.

err := sdk.NewPipeline("my-pipeline", "My Simple Pipeline").
    Description("This pipeline is purely for testing purposes.").
    Tasks(sdk.NewTask("simple-task", "ubuntu:latest").
			Description("This task simply prints our hello-world message and exits!").
			Command("echo", "Hello from Gofer!"),
    ).Finish()

That's it! This is a fully functioning pipeline.

You can run and test this pipeline much like you would any other code you write. Running it will produce a JSON output which Gofer uses to pass to the server.

You can find examples like this and more in example pipelines

The same pipeline in Rust

With gofer_sdk added to your Cargo.toml (see SDK above), the same pipeline in src/main.rs looks like this:

use gofer_sdk::config::{Pipeline, Task};

fn main() {
    Pipeline::new("my-pipeline", "My Simple Pipeline")
        .description("This pipeline is purely for testing purposes.")
        .tasks(vec![
            Task::new("simple-task", "ubuntu:latest")
                .description("This task simply prints our hello-world message and exits!")
                .command(vec!["echo".to_string(), "Hello from Gofer!".to_string()]),
        ])
        .finish()
        .unwrap();
}

gofer up works out which language a pipeline is written in by looking for a go.mod or Cargo.toml in the folder you point it at.

Extra Examples

Auto Inject API Tokens

Gofer has the ability to auto-create and inject a token into your tasks. This is helpful if you want to use the Gofer CLI or the Gofer API to communicate with Gofer at some point in your task.

You can tell Gofer to do this by using the InjectAPIToken function for a particular task. The token shows up in the task as the GOFER_TOKEN environment variable, which is the same variable the Gofer CLI reads. Gofer also sets GOFER_API_BASE_URL and GOFER_NAMESPACE on every task, so the CLI needs no other setup. See passing data between tasks for a full example.

The token is deleted when the run's objects expire (see run_object_expiry_runs in the configuration reference).

err := sdk.NewPipeline("my-pipeline", "My Simple Pipeline").
    Description("This pipeline is purely for testing purposes.").
    Tasks(
		sdk.NewTask("simple-task", "ubuntu:latest").
			Description("This task simply prints our hello-world message and exits!").
			Command("echo", "Hello from Gofer!").InjectAPIToken(true),
    ).Finish()

Limiting Concurrent Runs

By default a pipeline can have up to 200 runs going at once (the server's pipeline_run_concurrency_limit). Use Parallelism to set a lower limit for one pipeline. For example, a deploy pipeline that should never run twice at the same time:

err := sdk.NewPipeline("deploy", "Deploy").
    Parallelism(1).
    Tasks(...).Finish()

Runs started while the pipeline is at its limit wait until an earlier one finishes. A pipeline can't set a limit higher than the server's.

Next Steps

• Tasks covers everything a task can do: dependencies, variables, secrets, and objects.
• Updating Pipelines explains what happens when you gofer up a pipeline again.
• Running Pipelines Automatically shows how to make a pipeline run on a schedule or when something happens.

Tasks

Gofer's abstraction for running a container is called a Task. Specifically Tasks are containers you point Gofer to and configure to perform some workload.

A Task can be any container you want to run. In the Getting Started example we take a regular standard ubuntu:latest container and customize it to run a passed in bash script.

Tasks(
    sdk.NewTask("simple-task", "ubuntu:latest").
        Description("This task simply prints our hello-world message and exits!").
        Command("echo", "Hello from Gofer!"),
)

Task Dependencies (DAGs)

By default every task in a pipeline starts as soon as the run does, so tasks run in parallel. To make a task wait on another one, use DependsOn and say what state the parent has to end in:

If the parent doesn't end in the required state, the child is skipped. Skipped tasks count as neither success nor failure, so whole branches below a skipped task get skipped too, except for tasks that depend on it with Any.

Tasks(
    sdk.NewTask("run-tests", "alpine:latest").
        Command("sh", "-c", "./run_tests.sh"),

    // Only publish if the tests pass.
    sdk.NewTask("publish-release", "alpine:latest").
        DependsOn("run-tests", sdk.RequiredParentStatusSuccess).
        Command("echo", "publishing release"),

    // Tell someone the tests broke.
    sdk.NewTask("alert", "alpine:latest").
        DependsOn("run-tests", sdk.RequiredParentStatusFailure).
        Command("echo", "the tests failed"),
)

A task can depend on several parents with DependsOnMany; it runs once all of them have finished in their required states. The dag example pipeline shows a few of these together. gofer run debug <pipeline> <run> is handy here since it shows why each task was skipped.

Task Environment Variables and Configuration

Gofer handles container configuration the cloud native way. That is to say every configuration is passed in as an environment variable. This allows for many advantages, the greatest of which is standardization.

When a container is run by Gofer, its environment variables come from three places:

1. Run variables: Values passed in when a run is started. You can pass them yourself with gofer pipeline run my-pipeline -v KEY=VALUE (repeat -v for more than one), and extensions pass them in when they start a run; the github extension, for example, passes in details about the commit that triggered it.
2. Your pipeline configuration: Values you set with the Variables function on a task.
3. Gofer's system variables: Values Gofer sets on every task, listed below.

If the same name comes from more than one place, the one higher on this list wins. So a run variable can override a value from your pipeline configuration for a single run, and either one can override a system variable.

Be careful overriding the GOFER_ variables. Gofer lets you, but the task then trusts whatever you set. For example, a task with InjectAPIToken sends its token to whatever GOFER_API_BASE_URL points at, so anyone who can start runs of a pipeline can point that somewhere else and collect the token. Only give run permissions to people you'd trust with the pipeline's token.

Variable names are always uppercased, so Variables(map[string]string{"log_level": "debug"}) shows up in the container as LOG_LEVEL. Names can't be empty or contain =, since they become environment variable names.

You can see exactly what a task was given with gofer task get <pipeline_id> <run_id> <task_id>.

These are the system variables Gofer injects into every task:

Using Secrets and Objects in Variables

A variable's value can pull from Gofer's secret store or object store instead of being written into the pipeline config. That keeps secrets out of your code and lets one task hand a value to the next.

You do this with a special string as the value. The SDK has a helper function for each kind, or you can write the string yourself:

sdk.NewTask("deploy", "my-org/deployer:latest").
    Variables(map[string]string{
        "ENVIRONMENT":  "production",                    // Passed in as is.
        "DEPLOY_KEY":   sdk.PipelineSecret("deploy_key"), // Becomes the value of the pipeline secret "deploy_key".
        "BUILD_NUMBER": sdk.RunObject("build_number"),    // Becomes the value of the run object "build_number".
    })

Secrets and objects have to be stored before the task that uses them starts:

gofer secret pipeline put my-pipeline deploy_key        # Prompts for the value.
gofer secret global put slack_token -n "ops-.*"          # Admins only; usable from namespaces matching "ops-.*".
gofer pipeline object put logs_header ./header.txt --pipeline my-pipeline

Gofer swaps in the real values right before each task starts, after its parents have finished. So a task can store a run object and any task that depends on it can read it as a variable; see passing data between tasks for a full example. Objects are converted to UTF-8 text when they're inserted; if you need the raw bytes, fetch the object through the API inside your task instead.

If a secret or object doesn't exist, or a global secret doesn't allow your namespace, the task fails before its container starts and the reason says which key was missing.

Private Registries

If a task's image lives in a registry that needs a login, give the task credentials with RegistryAuth. The password has to be a pipeline or global secret so it never ends up stored in your pipeline config; Gofer rejects the config otherwise. The user can be written in as is, or be a secret too:

sdk.NewTask("build", "registry.example.com/my-org/builder:latest").
    RegistryAuth("my-user", sdk.PipelineSecret("registry_pass"))

#![allow(unused)]
fn main() {
Task::new("build", "registry.example.com/my-org/builder:latest")
    .registry_auth("my-user", &pipeline_secret("registry_pass"))
}

The secrets are looked up right before the image is pulled, just like variables. Objects aren't allowed here, and if a secret is missing the task fails before its container starts.

What happens when a task is run?

The high level flow is:

1. Gofer checks to make sure your task configuration is valid.
2. Gofer waits for the task's parents to finish and checks they ended in the states the task requires.
3. Gofer parses the task configuration's variables list and registry auth. It replaces any secret or object strings with their actual values from the object or secret store.
4. Gofer then passes the details of your task to the configured scheduler, variables are passed in as environment variables.
5. Usually this means the scheduler will take the configuration and attempt to pull the image mentioned in the configuration.
6. Once the image is successfully pulled the container is then run with the settings passed.

Updating Pipelines

Every time you run gofer up with changes, Gofer stores your pipeline config as a new version (v1, v2, and so on) instead of overwriting the old one. Then it deploys that version, which makes it the one new runs use.

Before it registers anything, gofer up shows what's different from the latest version and asks you to confirm:

$ gofer up ./my-pipeline
 Changes since v2:
   │ ~ tasks.build.command:
   │       sh -c 'exec "$(command -v bash || echo sh)" -c "$1"' gofer-script 'echo one
   │     - echo two
   │     + echo 2
   │       echo three'
   │ ~ tasks.build.image: alpine:latest -> alpine:3.20
   │ + tasks.notify: new task using alpine:latest
 ? Register and deploy these changes? [y/N]: y
 ✓ Registered pipeline: [my-pipeline] 'My Pipeline' v3

Pass --yes to skip the question. Gofer also skips it when it isn't run from a terminal, like in CI. If nothing changed, no new version is made; Gofer just deploys the latest version if it isn't live yet.

Keeping every version means you can always see exactly what a past run ran with. Each run records its version, and gofer run get <pipeline_id> <run_id> shows it next to the run number.

Config states

Each version is in one of three states:

gofer pipeline config list <pipeline_id> shows every version and its state, and gofer pipeline config get <pipeline_id> <version> shows what's in one.

Registering without deploying

gofer up --deploy false ./my-pipeline stores the new version without making it live. Runs keep using the current live version until a new one is deployed. This is handy when you want the new version in Gofer, to look over or deploy later, without it taking effect yet. When you're ready, gofer pipeline config deploy <pipeline_id> <version> makes it live.

A brand new pipeline registered this way has no live version, so it can't run until one is deployed.

Deployments

Each time a version goes live is a deployment. Deploying swaps which version is live: the new one becomes Live and the old one becomes Deprecated. Runs that are already going keep running the version they started with.

gofer pipeline deployment list <pipeline_id> shows a pipeline's deployment history, which is useful for working out when a change went out. Only one deployment can happen at a time per pipeline.

Rolling back

To roll back, deploy the earlier version again. Find it with gofer pipeline config list <pipeline_id>, then:

gofer pipeline config deploy my-pipeline 2

That makes version 2 live and deprecates the current one, without registering anything new.

You can also gofer up the old code again, for example from an earlier commit. That registers it as a new version and deploys it, which is the way to go if the version you want has been deleted.

Cleaning up old versions

gofer pipeline config delete <pipeline_id> <version> removes a version. You can't delete the live version or a pipeline's only version.

Running Pipelines Automatically

A pipeline config describes what to run, but not when. Pipelines only run when someone starts them, unless they're subscribed to an extension. Extensions watch for something to happen, like a time passing or a push to a Github repository, and start a run when it does.

Subscriptions aren't part of the pipeline config. You add them with the CLI after the pipeline exists, and they stay in place when you register new versions of the pipeline with gofer up.

Subscribing

gofer pipeline subscribe <pipeline_id> <extension_id> <label> -s key=value

• extension_id is the extension to subscribe to. gofer extension list shows the ones your Gofer has.
• label is a name you pick for this subscription. A pipeline can subscribe to the same extension more than once, for example to run every hour and also every Monday, and the label tells them apart.
• -s key=value sets the subscription's parameters. Each extension has its own, listed on its page; repeat -s for more than one.

For example, to run the simple pipeline every five minutes with the interval extension:

gofer pipeline subscribe simple interval every-five-mins -s every="5m"

Or every Christmas at 1am with cron:

gofer pipeline subscribe simple cron yearly-on-xmas -s expression="0 1 25 12 * *"

Parameters can be secrets too, using the same pipeline_secret{{key}} or global_secret{{key}} strings as task variables. Gofer swaps in the real value when it sends the subscription to the extension.

Seeing and removing subscriptions

gofer fetch <pipeline_id> lists a pipeline's subscriptions along with each one's status, and so does the pipeline's page in the web UI (gofer web <pipeline_id>). active means it can trigger runs. error means the extension behind it isn't running, so nothing will fire until it's back; the reason column says what's wrong with the extension. To remove one:

gofer pipeline unsubscribe simple interval every-five-mins

To stop one from triggering runs for a while without losing its settings, disable it, and enable it again when you want it back:

gofer pipeline disable-subscription simple interval every-five-mins
gofer pipeline enable-subscription simple interval every-five-mins

What the extension passes to your run

When an extension starts a run it can pass in run variables, which your tasks receive as environment variables. The github extension, for example, passes in details about the event that triggered it. Each extension's page lists what it passes, and gofer task get <pipeline_id> <run_id> <task_id> shows exactly what a task received.

If an extension is stopped, disabled, or has failed, its subscriptions stay in place but nothing triggers them. They show as error until the extension is running again. If a pipeline that should be running on its own isn't, start with gofer fetch <pipeline_id>.

Namespaces

Namespaces divide pipelines into groups, usually one per team or project. Every pipeline lives in exactly one namespace, and pipeline ids only have to be unique within their namespace.

Gofer starts with a single namespace called default, which is where everything goes unless you say otherwise. If you're the only one using Gofer, you may never need another.

Creating a namespace

Only admins can create, update, or delete namespaces:

gofer namespace create frontend "Frontend Team" "Pipelines for the website and design system"
gofer namespace list

Like pipeline ids, namespace ids can only contain letters, numbers, and hyphens, and have to be 3 to 32 characters long.

Working in a namespace

CLI commands work in your default namespace, which is default unless you change it. There are two ways to use a different one:

• Pass --namespace to a single command: gofer up --namespace frontend ./website.
• Change your default by setting namespace in your CLI configuration, or the GOFER_NAMESPACE environment variable.

What namespaces control

• Permissions. Roles grant access by namespace, using a regex like frontend or ops-.*. This is how you give a team full access to its own pipelines without access to anyone else's. See Authentication and Authorization. The built in user role only covers the default namespace, so tokens for other namespaces need a role of their own.
• Global secrets. Each global secret lists which namespaces can use it, so a team's secrets can be shared across its pipelines without being visible to other teams. See Secret Store.

Server Configuration

Gofer runs as a single static binary that you deploy onto your favorite VPS.

While Gofer will happily run in development mode without any additional configuration, this mode is NOT recommended for production workloads and not intended to be secure.

Instead Gofer allows you to edit its startup configuration allowing you to configure it to run on your favorite container orchestrator, object store, and/or secret backend.

Setup

There are a few steps to setting up the Gofer service for production:

1) Configuration

First you will need to properly configure the Gofer service.

Gofer accepts configuration through environment variables or a configuration file. If a configuration key is set both in an environment variable and in a configuration file, the value of the environment variable's value will be the final value.

Gofer reads its configuration file from /etc/gofer/gofer_web.toml, or from the path passed to gofer service start --config <path>. Extensions are configured in this file too, and gofer extension reload rereads it to pick up extension changes without a restart.

Every key in the configuration reference can also be set as an environment variable. Each one starts with a prefix of GOFER_WEB_ and uses a double underscore between the block and the key. So the api.log_level configuration can be set as:

export GOFER_WEB_API__LOG_LEVEL=debug

Configuration file

The Gofer service configuration file is written in TOML.

Load order

Gofer starts from its built in defaults, then layers on top of them, in order:

1. The file passed to gofer service start --config <path>, or /etc/gofer/gofer_web.toml if no path is given.
2. GOFER_WEB_ environment variables.

You only need to set the keys you want to change.

Bare minimum production file

These are the bare minimum values you should populate for a production ready Gofer configuration.

The values below should be changed depending on your environment. The defaults keep everything in /tmp, which will lead to loss of data on server restarts, and they turn off authentication.

[server]
bind_address = "0.0.0.0:8080"
# The address extension and task containers use to reach Gofer. With the docker scheduler that's the docker
# bridge's address on the host, but with TLS on, containers check Gofer's certificate against this address. So use
# a name your certificate covers that resolves to the host from inside containers.
extension_address = "https://gofer.internal:8080"
storage_path = "/var/lib/gofer/gofer.db"
use_tls = true
tls_cert_path = "/etc/gofer/tls/cert.pem"
tls_key_path = "/etc/gofer/tls/key.pem"

# The defaults are on to make local development easy. All of them are unsafe in production.
[development]
pretty_logging = false
bypass_auth = false
use_included_certs = false

[api]
task_execution_logs_dir = "/var/lib/gofer/logs"

[external_events]
use_tls = true
tls_cert_path = "/etc/gofer/tls/cert.pem"
tls_key_path = "/etc/gofer/tls/key.pem"

[object_store.filesystem]
path = "/var/lib/gofer/objects"

[secret_store.sqlite]
path = "/var/lib/gofer/secrets.db"
# Exactly 32 characters; `openssl rand -hex 16` makes one. Never change it after secrets are stored.
encryption_key = "<your key here>"

If you don't use external events (for example the Github extension), set external_events.enable = false instead of configuring TLS for it.

Gofer creates the object store directory itself, but not the others, so create /var/lib/gofer and /var/lib/gofer/logs and make sure the user Gofer runs as can write to them.

2) Running the binary

You can find the most recent releases of Gofer on the github releases page..

Simply use whatever configuration management system you're most familiar with to place the binary on your chosen VPS and manage it. You can find a quick and dirty wget command to pull the latest version in the getting started documentation.

3) First steps

You will notice upon service start that the Gofer CLI is unable to make any requests due to permissions.

You will first need to handle the problem of auth. Every request to Gofer must use an API key so Gofer can appropriately direct requests.

More information about auth in general terms can be found here.

To create your root/bootstrap token use the command: gofer token bootstrap

From here you can use your root token to provision extra, lower permissioned tokens for everyday use. For example, a token for a person who works in the default namespace:

export GOFER_TOKEN=<your bootstrap token>
gofer token create alice --roles user

The built in user role can do everything with pipelines in the default namespace. For other namespaces, create a role that covers them first; Authentication and Authorization explains how. Pass --expiry <seconds> to have a token stop working after a while.

When communicating with Gofer through the CLI you can set the token to be automatically passed per request in one of many ways.

Configuration Reference

Gofer reads its configuration from /etc/gofer/gofer_web.toml, or the file passed to gofer service start --config <path>. Any key can also be set through an environment variable that starts with GOFER_WEB_, using a double underscore between the block and the key. For example api.log_level becomes GOFER_WEB_API__LOG_LEVEL. Environment variables win over the file.

Every key has a default, so you only need to set the ones you want to change. The defaults are set up for running Gofer locally; see the bare minimum production file for what to change before running it for real.

The defaults come from default_api_config.toml and the keys themselves are defined in conf/api.rs. If this page and those files ever disagree, the files are right.

Keys that end in _seconds take a whole number of seconds, and keys that end in _runs take a number of runs.

API

[api]
pipeline_run_concurrency_limit = 200
global_run_concurrency_limit = 2000
event_log_retention_seconds = 15768000    # 6 months
event_prune_interval_seconds = 604800     # 1 week
log_level = "info"
task_execution_log_retention_runs = 50
task_execution_logs_dir = "/tmp"
task_execution_stop_timeout_seconds = 300 # 5 mins
task_execution_max_timeout_seconds = 86400 # 24 hours
allow_task_attach = true

Server

[server]
bind_address = "0.0.0.0:8080"
extension_address = "172.17.0.1:8080"
storage_path = "/tmp/gofer.db"
use_tls = false

Development

Feature flags that make running Gofer locally easier. The defaults are all on, which is what lets gofer service start work with no configuration. Turn them all off in production.

[development]
pretty_logging = true
bypass_auth = true
use_included_certs = true

External Events

The external events service is a separate HTTP server that takes webhooks and passes them on to extensions. It runs on its own port so you can expose it to services like Github without exposing the rest of Gofer. See External Events for how it works.

[external_events]
enable = true
bind_address = "0.0.0.0:8081"
use_tls = false

Object Store

The object store holds values that tasks share with each other or keep between runs. More about the object store.

Filesystem

[object_store]
engine = "filesystem"
pipeline_object_limit = 50
run_object_expiry_runs = 2

[object_store.filesystem]
path = "/tmp/gofer_objects"

Secret Store

The secret store holds secrets that pipelines and extensions pull in at runtime. More about the secret store.

Sqlite

[secret_store]
engine = "sqlite"

[secret_store.sqlite]
path = "/tmp/gofer_secrets.db"
encryption_key = "changemechangemechangemechangeme"

Scheduler

The scheduler is the container orchestrator Gofer runs tasks and extensions on. More about schedulers.

Docker

[scheduler]
engine = "docker"

[scheduler.docker]
prune = true
prune_interval_seconds = 604800
timeout_seconds = 300

Extensions

Controls Gofer's extension system. More about extensions.

gofer extension reload applies changes to the install entries without restarting Gofer. Everything else in this block needs a restart.

[extensions]
stop_timeout_seconds = 300    # 5 mins
use_tls = false
verify_certs = false
health_check_interval_seconds = 60
health_check_failures_before_action = 3
health_check_auto_restart = true
health_check_max_restarts = 1

[[extensions.install]]
id = "github"
manifest = "https://raw.githubusercontent.com/clintjedwards/gofer/v<gofer version>/containers/extensions/github/manifest.toml"
[extensions.install.settings]
app_id = "112348"
app_installation = "99560091"
app_key = "global_secret{{github-app-key}}"
app_webhook_secret = "global_secret{{github-app-webhook-secret}}"

[[extensions.install]]
id = "cron"
enabled = false

Authentication and Authorization

Gofer's authentication and authorization systems are designed to be lightweight and unobtrusive, allowing you to focus on your tasks with minimal interference. The permissioning system is based on Role-Based Access Control (RBAC) and is primarily used to prevent certain token holders, such as extensions, from accessing excessive parts of the system.

For most users, this means the ability to categorize users into distinct groups, typically at the namespace or pipeline level.

Authorization in Gofer is token-based, a principle that extends seamlessly to the frontend as well.

Authorization

Gofer organizes grants into roles, which are then assigned to specific tokens. A token can do whatever its roles' grants allow.

Roles

A role is a collection of grants. The most significant role is the bootstrap role, which is a special role. The bootstrap role is equivalent to a root role and is assigned to the first token you receive within Gofer.

Gofer also includes 'system roles' which are special roles that cannot be modified or removed. These roles are generally used by other components of Gofer for specific purposes or are there for your convenience.

You can view a list of these roles by using the gofer role list command and identifying the roles where system_role is marked as true.

Grants

A role's grants are split into three groups, based on what they apply to:

{
  "id": "devops",
  "description": "Full access to devops namespaces and can follow events",
  "grants": {
    "namespaces": [
      {
        "namespace": "devops.*",
        "pipeline": ".*",
        "resources": ["pipelines", "configs", "deployments", "runs", "task_executions", "objects", "secrets", "subscriptions"],
        "actions": ["read", "write", "delete"]
      }
    ],
    "extensions": [
      { "extension": "github", "resources": ["objects"], "actions": ["read"] }
    ],
    "global": [
      { "resources": ["events"], "actions": ["read"] }
    ]
  }
}

Each grant stands on its own. A request is allowed only if a single grant matches what the request targets and includes both the resource and the action. Gofer never combines separate grants to satisfy a request. For example:

"namespaces": [
  { "namespace": "frontend", "pipeline": "website", "resources": ["runs"], "actions": ["read"] },
  { "namespace": "backend", "pipeline": "billing", "resources": ["secrets"], "actions": ["read"] }
]

This allows reading runs for frontend/website and secrets for backend/billing, but not runs for frontend/billing or secrets for frontend/website.

A token can have several roles. The grants from all of its roles are pooled together, and the request is allowed if any one of them covers it.

Namespace grants

Namespace grants cover everything that lives under a pipeline.

Any namespace grant also lets the token read the namespace itself.

Extension grants

Any extension grant also lets the token read the extension's details.

Global grants

Global grants cover things that don't belong to a namespace or extension. Only read can be granted; everything else here is admin only.

Gofer rejects global grants for secrets or system, or with write or delete, since admin only routes would ignore them.

Targets

Targets are regexes that always match the entire id. "namespace": "default" matches default but not not-default; use default.* for a prefix match. Writing ^ and $ yourself is fine but not needed. Use .* to match everything.

Gofer rejects roles with empty or invalid targets, grants without any resources or actions, and resources that don't belong to the grant's group.

When a route lists things, like listing namespaces, the token only needs a matching grant of the right kind. The results are then filtered down to the items the token is allowed to read.

Actions

There are three actions: read, write and delete. Each route belongs to one of them, which generally follows the HTTP method: GET is read, POST and PATCH are write, and DELETE is delete. Cancelling a run or task execution is a delete, and attaching to a task execution is a write. Attaching lets a token run any command in the task's container; operators can turn it off for everyone with the allow_task_attach server setting.

Admin only routes

Some routes can only be used by tokens with the admin or bootstrap role, regardless of what grants a token has. No grant can give access to them:

• Creating, updating, and deleting namespaces.
• Listing, creating, updating, and deleting tokens.
• Creating, updating, and deleting roles.
• Reloading extensions (planning, applying, and reverting), purging them, and reading an extension's debug info.
• Everything in the global secret store.
• Reading and updating system preferences.
• Deleting events.
• Using include_secret to see unredacted values.

Creating roles

gofer role create <id> <description> --file role.json creates a role from a JSON file containing the grants object. Without --file, the CLI prompts for each grant. gofer role update <id> --file role.json replaces a role's grants.

System roles

Gofer also creates a few roles automatically:

• extension_<extension_id>: given to each extension. It can use its own object store and read its own subscriptions, start runs in any pipeline, read most pipeline information, and follow events.
• inject_api_token_<namespace_id>_<pipeline_id>: given to tokens created by a pipeline's inject_api_token option. It can read and write its own pipeline's runs, objects, and configs, read pipelines in the default namespace and its own namespace, and follow events.

Errors

• 401 Unauthorized means the request didn't include a valid token: it's missing, unknown, disabled, or expired.
• 403 Forbidden means the token is valid but doesn't have permission for the route. The response explains what the route needs.

Authentication

Before you can start using Gofer, you need to obtain an API token. You can retrieve the bootstrap token using the gofer token bootstrap command or by making a request to the /api/tokens/bootstrap route.

Once a bootstrap token is collected it can no longer be collected.

How to auth via the API

Gofer requires two headers for successful authentication:

• Authorization: Bearer <token>
• gofer-api-version: v<version_number>

How to auth via the CLI

The Gofer CLI accepts multiple methods for setting a token once you have one.

External Events

Gofer has an alternate endpoint specifically for external events streams¹. This endpoint takes in http requests from the outside and passes them to the relevant extension.

You can find more about external event configuration in the configuration-values reference.

enable = true
bind_address = "0.0.0.0:8081"
use_tls = false

Authentication

This endpoint does not use Gofer API tokens. The services that send external events, like Github webhooks, have no way to include one. Instead, Gofer passes every request through and the extension decides whether to trust it. For example, the Github extension checks the webhook signature Github includes in the headers.

This means anyone who can reach this port can send requests to any installed extension. Any extension that accepts external events should verify them, and you should limit who can reach the port where possible.

It works like this:

1. When the Gofer service is started it starts the external events service on a separate port per the service configuration settings. It is also possible to just turn off this feature via the same configuration file.

2. External services can send Gofer http POST requests with payloads and headers specific to the extension they're trying to communicate with. You target a specific extension by putting its id at the end of the path.

   ex: https://mygofer.mydomain.com/api/external/github <- extension id

3. Gofer serializes and forwards the request, including all of its headers and body, to the relevant extension where it is validated for authenticity of sender and then processed.

4. An extension may then handle this external event in any way it pleases. For example, the Github extension takes in external events which are expected to be Github webhooks and starts a pipeline if the event type matches one the user wanted.

Scheduler

Gofer runs the containers you reference in the pipeline configuration via a container orchestrator referred to here as a "scheduler".

The vision of Gofer is for you to use whatever scheduler your team is most familiar with.

Supported Schedulers

The only currently supported scheduler is local docker. This scheduler is used for small deployments and development work.

How to add new Schedulers?

Schedulers are pluggable, but for them to maintain good performance and simplicity the code that orchestrates them must be added to the schedulers folder within Gofer(which means they have to be written in Rust).

Docker scheduler

The docker scheduler uses the machine's local docker engine to run containers. This is great for small or development workloads and very simple to implement. Simply download docker and go!

[scheduler]
engine = "docker"

[scheduler.docker]
prune = true
prune_interval_seconds = 604800
timeout_seconds = 300

Configuration

Docker needs to be installed and the Gofer process needs to have the required permissions to run containers upon it.

Other than that the docker scheduler just needs to know how to clean up after itself.

Object Store

Gofer provides an object store as a way to share values and objects between containers. It can also be used as a cache. It is common for one container to run, generate an artifact or values, and then store that object in the object store for the next container or next run. The object store can be accessed through the Gofer CLI or through the normal Gofer API. Tasks can also receive objects directly as environment variables; see Using Secrets and Objects in Variables. For examples of handing values and files from one task to the next, see passing data between tasks.

Gofer divides the objects stored into two different lifetime groups:

Pipeline-level objects

Gofer can store objects permanently for each pipeline. You can store objects at the pipeline-level by using the gofer pipeline object store command:

gofer pipeline object put my_key1 ./some_file --pipeline my-pipeline
echo "my_value" | gofer pipeline object put my_key2 @ --pipeline my-pipeline
gofer pipeline object get my_key1 --pipeline my-pipeline

The limitation to pipeline level objects is that they have a limit of the number of objects that can be stored per-pipeline (pipeline_object_limit). Once that limit is reached the oldest object in the store will be removed for the newest object. Overwriting an existing object with --force counts it as the newest.

Run-level objects

Gofer can also store objects on a per-run basis. Unlike the pipeline-level objects run-level do not have a limit to how many can be stored, but instead have a limit of how long they last. Only the most recent runs keep their objects (run_object_expiry_runs, 2 by default); once a pipeline moves past that, the oldest run's objects are deleted.

You can access the run-level store using the run level store CLI commands. Here is an example:

gofer run object put my_key ./some_file --pipeline my-pipeline --run 1
gofer run object get my_key --pipeline my-pipeline --run 1

Inside a task you can leave off --pipeline and --run. The CLI falls back to GOFER_PIPELINE_ID and GOFER_RUN_ID, which Gofer sets on every task.

Keys

Object keys can be up to 128 characters of letters, numbers, _, ., and -, so file names like build.tar.gz work as keys. Keys only need to be unique within their pipeline or run.

Storing directories

If you put a directory, the CLI uploads it as a gzipped tarball. get --extract <dir> downloads and unpacks it, so you don't have to write your own tar commands:

gofer run object put dist ./dist           # In the task that builds it.
gofer run object get dist --extract ./dist # In a task that depends on it.

Supported Object Stores

The only currently supported object store is the filesystem object store. Reference the configuration reference for a full list of configuration settings and options.

How to add new Object Stores?

Object stores are pluggable, but for them to maintain good performance and simplicity the code that orchestrates them must be added to the object_store folder within Gofer(which means they have to be written in Rust).

Filesystem object store

The filesystem object store keeps objects as files in a directory on the local machine. It is great for development and small deployments.

[object_store]
engine = "filesystem"
pipeline_object_limit = 50
run_object_expiry_runs = 2

[object_store.filesystem]
path = "/tmp/gofer_objects"

Configuration

The filesystem store needs a directory on the local machine, making the only parameter it accepts a path to that directory.

Secret Store

Gofer provides a secret store as a way to enable users to pass secrets into pipeline configuration files.

The secrets included in the pipeline file use a special syntax so that Gofer understands when it is given a secret value instead of a normal variable:

Variables(map[string]string{
    "DEPLOY_KEY":  sdk.PipelineSecret("deploy_key"),  // or "pipeline_secret{{deploy_key}}"
    "SLACK_TOKEN": sdk.GlobalSecret("slack_token"),   // or "global_secret{{slack_token}}"
})

Gofer swaps in the real value right before the task's container starts, so the secret never appears in your pipeline config. There are two kinds of secrets:

• Pipeline secrets belong to a single pipeline and are set with gofer secret pipeline put <pipeline> <key>.
• Global secrets are shared across pipelines and only admins can manage them. Each one has a list of namespaces (regexes) that are allowed to use it, set with gofer secret global put <key> -n <namespace regex>.

Secret keys can be up to 128 characters of letters, numbers, _, ., and -.

See Using Secrets and Objects in Variables for more detail.

[secret_store]
engine = "sqlite"

Supported Secret Stores

The only currently supported secret store is the sqlite secret store. Reference the configuration reference for a full list of configuration settings and options.

How to add new Secret Stores?

Secret stores are pluggable, but for them to maintain good performance and simplicity the code that orchestrates them must be added to the secret_store folder within Gofer(which means they have to be written in Rust).

Sqlite secret store

The sqlite secret store is great for development and small deployments.

[secret_store]
engine = "sqlite"

[secret_store.sqlite]
path = "/tmp/gofer_secrets.db"
encryption_key = "changemechangemechangemechangeme"

Configuration

Sqlite keeps secrets in a database file on the local machine, encrypted with the key you give it.

Extensions

Extensions are Gofer's way of adding additional functionality to pipelines. You can subscribe your pipeline to an extension, allowing that extension to give your pipeline extra powers.

The most straight-forward example of this is the interval extension. It lets your pipeline run every time some amount of time has passed. Say you have a pipeline that needs to run every 5 minutes; you'd subscribe it to the interval extension with an interval of 5m:

gofer pipeline subscribe simple interval every-five-mins -s every="5m"

On startup, Gofer launches the interval extension as a long-running container. When your pipeline subscribes to it, the interval extension starts a timer, and when 5 minutes have passed it sends an API request to Gofer, causing Gofer to run your pipeline.

Gofer's extensions

Gofer comes with its own set of extensions, a bit like a standard library. They're all maintained and released alongside Gofer, and they come in two kinds:

• Default extensions, cron and interval, are basic enough that everyone gets them. They're installed unless you turn them off, and their manifests are built into Gofer.
• Optional extensions, like github, are for things not everyone needs. You add them to your config when you want them, the same way you'd install any other extension.

You can also write your own.

Installing and configuring extensions

Extensions are installed and configured in one place: Gofer's server config. There's no separate install command; you describe the extensions you want and Gofer makes it so.

Installing an extension takes four steps:

1. Put the extension's secrets in Gofer's global secret store. Extensions that talk to other services usually need credentials. Those never go in the config file itself.

   # Prompts for the value so it stays out of your shell history. You can also pipe in a file.
   gofer secret global put github-app-key < my-github-app.private-key.pem
   

2. Add an [[extensions.install]] entry to Gofer's config that points at the extension's manifest. The manifest is a small file that ships with each extension and tells Gofer everything about it: which container image to run, which settings it takes, and which parameters pipelines pass when they subscribe.

3. Fill in the extension's settings. Secrets are written as a reference to the global secret store, like global_secret{{github-app-key}}, instead of the value itself.

   [[extensions.install]]
   id = "github"
   manifest = "https://raw.githubusercontent.com/clintjedwards/gofer/v<gofer version>/containers/extensions/github/manifest.toml"
   [extensions.install.settings]
   app_id = "112348"
   app_installation = "99560091"
   app_key = "global_secret{{github-app-key}}"
   app_webhook_secret = "global_secret{{github-app-webhook-secret}}"
   

   You don't have to write this by hand. gofer extension manifest <manifest URL or path> prints a ready-to-paste block listing every setting the extension takes, with its documentation, plus the commands to create any secrets it needs.

4. Reload. Either restart Gofer, or run gofer extension reload to apply the change to the running server. See Reloading below.

The [[extensions.install]] entry

Secrets

Any setting can be a global secret reference, and settings the extension marks as secret must be one. Gofer won't accept a raw value for them. Gofer swaps references for the real values only when it starts the extension's container, so the actual secret never appears in Gofer's config, its database, or gofer extension get output.

Global secrets normally have a list of namespaces that are allowed to use them. That list doesn't apply to extensions, since extensions don't belong to a namespace. Only admins can configure extensions or manage global secrets, so an extension can use any global secret.

Default extensions

Every Gofer install gets the cron and interval extensions without listing them. Their manifests are built into Gofer, so they always match the version of Gofer you're running and starting them never depends on downloading anything. Gofer serves them at /extensions/manifests/<name>.toml if you want to read one.

To change one of them, add an entry with the same id. Anything you set replaces the default:

# Use a different minimum interval.
[[extensions.install]]
id = "interval"
[extensions.install.settings]
min_interval = "5m"

# Don't run cron at all.
[[extensions.install]]
id = "cron"
enabled = false

Reloading

Restarting Gofer brings every extension in line with the config. Gofer starts what's new, restarts what changed, and stops what was disabled or removed. An extension that can't start (a typo in its settings, a missing secret, a bad image) doesn't stop Gofer from starting; it's marked as failed and gofer extension list shows why.

If you'd rather not restart Gofer, gofer extension reload does the same thing step by step:

1. Gofer rereads its config file and shows what would change for each extension: what's being installed, upgraded, or stopped, which settings changed, and which secrets were rotated. Nothing has changed yet at this point. gofer extension reload --dry-run stops here.
2. You confirm each extension one at a time, or skip it. --yes applies everything without asking.
3. If an extension's new version won't start, you're offered the version that was running before. With --yes that happens automatically. The extension then shows up in the next reload again until its config is fixed.
4. You get a summary of what happened to each extension.

$ gofer extension reload
  slack (update)
  │ manifest: https://example.com/slack/1.3.0/manifest.toml
  │ ~ manifest: https://example.com/slack/1.2.0/manifest.toml -> https://example.com/slack/1.3.0/manifest.toml
  │ ~ image: ghcr.io/example/slack:1.2.0 -> ghcr.io/example/slack:1.3.0
  │ ~ settings.token: global_secret{{slack-token}} -> global_secret{{slack-token}} (secret value changed)

1 change(s) to apply, 2 extension(s) unchanged.
? update 'slack'? [y]es / [s]kip / [q]uit:

A few things to know:

• Applying a change restarts the extension. Gofer stops the old container before starting the new one, so the extension is briefly unavailable. Events it would have handled in that window can be missed; for example, GitHub doesn't resend webhooks that fail. Running both at once isn't an option, since two copies would both act on the same pipeline subscriptions.
• Entries with problems are left alone. If an entry's manifest can't be fetched, a required setting is missing, or a secret doesn't exist, reload reports it and skips it. Whatever was running keeps running with its old config.
• Reload checks that nothing changed after you looked. If the config, a manifest, or a secret changes between the summary and your confirmation, Gofer refuses to apply it and asks you to run reload again.
• Only the extensions config is reloaded. Changes to other parts of Gofer's config still need a restart. Reload rereads the config file Gofer was started with (/etc/gofer/gofer_web.toml, or the path given to gofer service start --config). Environment variables can't change while Gofer is running, so settings that come from them stay as they were at startup.

Removing an extension

Removing an extension's entry from the config and reloading stops the extension, but keeps its pipeline subscriptions and stored objects. If you add the entry back, it picks up where it left off.

To delete an extension for good, purge it once it's out of the config:

gofer extension purge github

Purge shows how many pipelines will lose their subscription and asks before deleting anything.

Extension states

gofer extension list and gofer extension get <id> show two things about each extension.

The status says what the config wants:

The state says what's actually happening:

Every state change is published as an extension_state_changed event with the new state and reason, so gofer event list shows when an extension went up or down.

Health checks

Gofer checks each running extension's health endpoint once a minute. Any call Gofer makes to an extension that can't connect, like handing it a subscription or an external event, counts as a failed check too, so a dead extension is noticed as soon as something tries to use it.

1. The first failed check marks the extension degraded. A passing check puts it back to running.
2. After a few failed checks in a row, Gofer restarts it.
3. If it keeps failing after that, Gofer marks it failed, stops its container, and stops checking it. Fix whatever is wrong and run gofer extension reload to start it again.

The interval, how many failures to allow, and whether and how often to restart are all set in the extensions config.

Upgrading

See Upgrading Extensions.

How do extension settings reach the extension?

Gofer passes everything to the extension's container as environment variables:

• GOFER_EXTENSION_SYSTEM_* variables are set by Gofer itself: the extension's id, its API token, where to reach Gofer, TLS settings, and so on.
• GOFER_EXTENSION_CONFIG_<KEY> variables carry the settings from the config entry, with the key uppercased. The app_key setting arrives as GOFER_EXTENSION_CONFIG_APP_KEY. The SDK's GetConfigFromEnv("app_key") reads it for you.

Pipeline subscription parameters aren't environment variables; they're sent to the extension when a pipeline subscribes.

Extension Manifests

A manifest is a small TOML file that describes an extension to Gofer: which container image to run, which settings the extension takes, and which parameters pipelines pass when they subscribe to it. Every [[extensions.install]] entry in Gofer's config points at one.

The manifest is the only place Gofer learns about an extension; it never asks the running extension. That means:

• Gofer can check your settings before starting anything, catching a typo, a missing required setting, or a raw secret where a reference belongs.
• gofer extension reload can show exactly what will change before anything does.
• gofer extension get shows an extension's settings and parameters whether or not it's running. An extension that failed to start is documented just as well as one that's up.

Format

image = "ghcr.io/clintjedwards/gofer/extensions/github:0.11.0"
documentation = "You can find more information on this extension at the official Gofer docs site: ..."

[[config_params]]
key = "app_id"
required = true
secret = false
default = ""
documentation = "The Github app's ID, shown on the app's settings page."

[[config_params]]
key = "app_key"
required = true
secret = true
default = ""
documentation = "The Github app's private key in PEM format, exactly as Github gives it to you."

[[pipeline_subscription_params]]
key = "repository"
required = true
secret = false
default = ""
documentation = "The repository the pipeline will be alerted for. The format is <organization>/<repository>"

Each [[config_params]] and [[pipeline_subscription_params]] entry has:

Gofer checks config params itself. Pipeline subscription params are listed for documentation; the extension checks them when a pipeline subscribes.

Where manifests come from

The manifest field in an [[extensions.install]] entry can be:

• An https:// URL. Gofer downloads it every time it starts or reloads. Plain http:// is refused: the manifest decides which image Gofer runs with the extension's API token, so it shouldn't be possible to tamper with it in transit.
• A path on the Gofer server, like /etc/gofer/extensions/my_extension.toml. Relative paths are relative to the directory Gofer was started in.

Gofer's default extensions (cron and interval) don't need either: their manifests are built into Gofer, and Gofer serves them at /extensions/manifests/<name>.toml. An entry for one of them can leave manifest out. Gofer's optional extensions, like github, publish their manifests in the Gofer repository; see Gofer's extensions.

Generating a manifest

Manifests are generated by the extension's own code, so they can't drift away from the settings it actually reads. Extensions built with the Go or Rust SDK print theirs when run with the manifest argument:

./my_extension manifest --image ghcr.io/me/my_extension:1.2.0 > manifest.toml

See Writing Extensions for how that works, and publish the resulting file somewhere operators can reach it alongside your image.

When an extension doesn't publish a manifest

Any extension built with the Go or Rust SDK can print its own manifest, so you can make one yourself when the author didn't publish one. Always pass the exact image tag you're going to run; the manifest pins it, and a floating tag like latest can later point at a version whose settings don't match.

The quickest way is to ask the image itself. This works when the image's entrypoint is the extension binary, which is how the SDK examples and Gofer's own extensions are built:

docker run --rm ghcr.io/someone/their_extension:1.2.0 manifest --image ghcr.io/someone/their_extension:1.2.0 \
    > /etc/gofer/extensions/their_extension.toml

Otherwise, build the extension from its source at the matching version and run the binary:

git clone https://github.com/someone/their_extension && cd their_extension
git checkout v1.2.0
go run . manifest --image ghcr.io/someone/their_extension:1.2.0 > /etc/gofer/extensions/their_extension.toml
# or for a Rust extension: cargo run -- manifest --image ...

Then point the extension's [[extensions.install]] entry at the file on the Gofer server:

[[extensions.install]]
id = "their-extension"
manifest = "/etc/gofer/extensions/their_extension.toml"

Extensions that don't use either SDK won't have a manifest command. For those, write the file by hand using the format above.

Turning a manifest into config

gofer extension manifest reads a manifest and prints an [[extensions.install]] block to paste into Gofer's config, along with the commands to create the global secrets it refers to:

$ gofer extension manifest https://example.com/my_extension/manifest.toml --id my-extension
# Store the secret settings first; leave out '< <file>' to be prompted instead:
#   gofer secret global put my-extension-api-token < <file>

[[extensions.install]]
id = "my-extension"
manifest = "https://example.com/my_extension/manifest.toml"
[extensions.install.settings]

# The token used to call the example API.
# (required, secret; must be a global secret reference)
api_token = "global_secret{{my-extension-api-token}}"

Upgrading Extensions

An extension's version comes from its manifest, which pins an exact image. Upgrading means pointing the extension at a newer manifest and reloading.

Upgrading an extension

1. Change the manifest in the extension's [[extensions.install]] entry to the new version's manifest.
2. Check the new version's manifest or docs for settings that were added, removed, or renamed. If a new required setting is missing, the reload will tell you which one.
3. Run gofer extension reload. It shows the image change and any setting changes before anything happens, then restarts the extension once you confirm.
4. If the new version won't start, accept the offer to put the previous version back. The extension keeps running the old version until you fix the config and reload again.

The extension is briefly unavailable while it restarts. Events it would have handled during that window can be missed; GitHub, for example, doesn't resend webhooks that fail. Plan upgrades of webhook driven extensions for a quiet moment.

Default extensions

The manifests for Gofer's default extensions (cron and interval) are built into Gofer, so upgrading Gofer upgrades them: the next restart, or gofer extension reload, moves them to the versions released with the new Gofer. If you've pointed one at your own manifest in the config, Gofer leaves that alone.

Gofer's optional extensions, like github, upgrade like any other extension: change the version in their manifest URL to match your new Gofer and reload.

Compatibility

Gofer and its extensions follow one rule: within a major version of Gofer, extensions stay backwards compatible.

• An extension has to keep reading the data it stored with earlier versions (its objects in Gofer's object store, for example). Upgrading it should never require the operator to migrate anything by hand.
• Settings shouldn't disappear or change meaning. A new setting should either be optional or come with a default.
• Breaking changes wait for Gofer's next major version, and the extension's docs should spell out the upgrade path from each earlier version.

Before Gofer 1.0, minor versions count as major ones; Gofer 0.11 can make breaking changes from Gofer 0.10.

Notes for extension authors

The simplest way to stay compatible is to keep reading old formats rather than rewriting them. If you change how you store something, read both the old and new format and write the new one. An extension that converts old data the first time it starts works too, but then a revert to the previous version can't read the converted data, so prefer reading both whenever you can.

After Gofer 0.12: GOFER_EXTENSION_SYSTEM_GOFER_HOST is a full URL

Gofer used to pass a bare host:port, and extensions picked http or https from GOFER_EXTENSION_SYSTEM_USE_TLS. That setting is about whether the extension itself serves TLS, not Gofer, so extensions used the wrong scheme whenever the two differed. Gofer now passes the full URL, like https://gofer.internal:8080, and the Go SDK's NewClientWithHeaders and ListExtensionSubscriptions take it as is and no longer have a useTLS argument. Rebuild Go extensions against the new SDK. Rust extensions read gofer_host and should use it as is.

Writing Extensions

Just like tasks, extensions are simply containers, which makes them easy to test and move around. An extension is a small HTTP service that Gofer starts, sends pipeline subscriptions to, and that calls Gofer's API when something should happen (usually starting a run).

The easiest way to write one is with one of the SDKs. The Go SDK is github.com/clintjedwards/gofer/sdk/go/extensions. It handles the HTTP service, authentication with Gofer, and printing your extension's manifest. The interval extension is heavily commented and makes a good starting point.

The shape of an extension

You give the SDK two things:

1. Your extension's documentation, as plain data: the settings it takes (ConfigParams), the parameters pipelines pass when they subscribe (PipelineSubscriptionParams), and a description.
2. A function that builds your extension. It reads its settings, restores its subscriptions, and returns a value that implements extsdk.ExtensionServiceInterface (Health, Debug, Subscribe, Unsubscribe, Shutdown, and ExternalEvent).

var documentation = extsdk.Documentation{
	Body: "Runs pipelines when the example API says so.",
	ConfigParams: []extsdk.Parameter{
		{
			Key:           "api_token",
			Documentation: "The token used to call the example API.",
			Required:      true,
			Secret:        true,
		},
		{
			Key:           "poll_interval",
			Documentation: "How often to check the example API.",
			Default:       "1m",
		},
	},
	PipelineSubscriptionParams: []extsdk.Parameter{
		{
			Key:           "topic",
			Documentation: "Which topic to watch.",
			Required:      true,
		},
	},
}

func newExtension() *extension {
	token := extsdk.GetConfigFromEnv("api_token")
	// ...set up the extension, restore subscriptions with sdk.ListExtensionSubscriptions, and so on.
	return &extension{token: token}
}

func main() {
	extsdk.Run(documentation, func() extsdk.ExtensionServiceInterface { return newExtension() })
}

Keeping the documentation separate from the extension itself is what makes manifests work. When the binary is run as ./extension manifest --image <image>, Run prints the manifest and exits without ever calling newExtension, so it doesn't need any settings, a Gofer server, or credentials.

The manifest is the only way Gofer sees your documentation; it never asks the running extension. Changing a setting or parameter means regenerating and republishing the manifest along with the new image.

Settings

• Mark any setting that holds a credential with Secret: true. Gofer will only accept a global secret reference for it, which keeps the real value out of Gofer's config and database.
• Give optional settings a Default when there's a sensible one. Gofer fills it in when the operator doesn't set the key, and shows it in gofer extension manifest output.
• Read settings with extsdk.GetConfigFromEnv("key"). Gofer passes them as GOFER_EXTENSION_CONFIG_<KEY> environment variables.
• Settings are read once when the extension starts. Changing one means a restart, which gofer extension reload handles.

Publishing

Build and push your image, then generate a manifest that points at that exact image tag and publish it somewhere operators can download it over https:

docker build -t ghcr.io/me/my_extension:1.2.0 .
docker push ghcr.io/me/my_extension:1.2.0
go run . manifest --image ghcr.io/me/my_extension:1.2.0 > manifest.toml

Keeping manifest.toml in your repository and regenerating it on each release works well; that's how Gofer's own extensions do it (make generate-manifests writes them and make check-manifests fails if one is out of date).

Operators then install your extension by pointing an [[extensions.install]] entry at the manifest. See Installing and configuring extensions.

Compatibility

Operators upgrade extensions in place, so new versions have to keep working with the data and settings older versions left behind. See Upgrading Extensions.

Rust

The Rust SDK (gofer_sdk::extension) works the same way. You implement the Extension trait and hand run your documentation and an async function that builds the extension. Running the binary as <binary> manifest --image <image> prints the manifest without calling that function, and the output is identical to what the Go SDK would print for the same documentation.

use gofer_sdk::extension::{Documentation, Extension, Parameter, get_config_from_env, run};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let documentation = Documentation {
        body: "Runs pipelines when the example API says so.".into(),
        config_params: vec![Parameter {
            key: "api_token".into(),
            required: true,
            documentation: "The token used to call the example API.".into(),
            secret: true,
            default: String::new(),
        }],
        pipeline_subscription_params: vec![],
    };

    run(documentation, || async {
        let token = get_config_from_env("api_token").ok_or("api_token is required")?;
        // ...set up the extension and restore its subscriptions.
        Ok(Box::new(MyExtension { token }) as Box<dyn Extension>)
    })
    .await
}

The function can be async and return an error, so it can call Gofer's API to restore subscriptions and fail cleanly if a setting is wrong.

Gofer's Extensions

Gofer comes with a set of extensions, a bit like a standard library. They're all maintained and released alongside Gofer and share its version number. Some are basic enough that every install gets them; the rest are there when you need them.

Default extensions

These are always available: Gofer installs them unless the config turns them off with enabled = false.

Their manifests are built into Gofer, so they don't need a manifest in the config, always match the version of Gofer you're running, and never depend on downloading anything to start. Gofer serves them at /extensions/manifests/<name>.toml (for example http://localhost:8080/extensions/manifests/cron.toml) so you can read exactly what it runs. Upgrading Gofer upgrades them on the next restart or gofer extension reload.

If you need a different version of one, set manifest in its [[extensions.install]] entry to your own copy and Gofer uses that instead.

Optional extensions

These are just as supported as the default ones, but not everyone needs them, so they aren't installed until you add them to your config. You install them with a manifest URL, the same way as any other extension.

Each one keeps its manifest in the Gofer repository, next to its code, and every Gofer release tag has a copy pinned to the images built for that release:

https://raw.githubusercontent.com/clintjedwards/gofer/v<gofer version>/containers/extensions/<name>/manifest.toml

They're released alongside Gofer and share its version number, so use the manifest from the same release as your Gofer server, and change the version in the URL when you upgrade Gofer. See Upgrading Extensions.

Images are still published under floating tags (0.11, 0, and latest) as well as the full version, but manifests always pin the full version so that a reload only ever changes an extension when you change its manifest.

Cron Extension

Cron allows users to schedule pipeline runs on long term intervals and specific days.

Runs start at the top of each matching minute. Expressions are evaluated in the container's local time, which is UTC in the provided image.

It uses a stripped down version of the cron syntax to do so:

Field           Allowed values  Allowed special characters

Minutes         0-59            * , -
Hours           0-23            * , -
Day of month    1-31            * , -
Month           1-12            * , -
Day of week     0-6             * , -
Year            1970-2100       * , -

┌───────────── minute (0 - 59)
│ ┌───────────── hour (0 - 23)
│ │ ┌───────────── day of the month (1 - 31)
│ │ │ ┌───────────── month (1 - 12)
│ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday)
│ │ │ │ │ ┌───────────── Year (1970-2100)
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │
* * * * * *

Pipeline Configuration

• expression <string>: Specifies the cron expression of the interval desired.

Every year on Xmas

gofer pipeline subscribe simple cron yearly-on-xmas -s expression="0 1 25 12 * *"

Extension Configuration

Cron takes no settings. It's installed by default; to turn it off, add this to Gofer's config and run gofer extension reload:

[[extensions.install]]
id = "cron"
enabled = false

Interval Extension

Interval simply runs the subscribed pipeline at the given time interval continously.

Parameters/Pipeline Configuration

• every <string>: Specifies the time duration between events. Unless changed via the extension configuration, the minimum for this is 1 minute.

gofer pipeline subscribe simple interval every-five-mins -s every="5m"

Extension Configuration

Interval is installed by default. Its settings go under an [[extensions.install]] entry with id = "interval" in Gofer's config; run gofer extension reload after changing them.

Example

[[extensions.install]]
id = "interval"
[extensions.install.settings]
min_interval = "5m"

Github Extension

The Github extension allows Gofer pipelines to be run on Github webhook events. This makes it possible to write event driven workloads that depend on an action happening on Github.

See the events section below for all supported events and the environment variables they pass to each pipeline.

Pipeline Configuration

Example

gofer pipeline subscribe simple github run-tests \
    -s "repository=clintjedwards/experimental" \
    -s "event_filter=pull_request_with_check/opened,synchronize,reopened"

Extension Configuration

The Github extension isn't installed by default. It requires a Github app of your own; the setup walkthrough below covers creating one and installing the extension.

Secret settings must be global secret references like global_secret{{github-app-key}}.

Example

[[extensions.install]]
id = "github"
manifest = "https://raw.githubusercontent.com/clintjedwards/gofer/v<gofer version>/containers/extensions/github/manifest.toml"
[extensions.install.settings]
app_id = "112348"
app_installation = "99560091"
app_key = "global_secret{{github-app-key}}"
app_webhook_secret = "global_secret{{github-app-webhook-secret}}"
gofer_url = "https://gofer.example.com"

Setting it up

Due to the nature of Github's API and webhooks, you'll need to first set up a new Github app to use with Gofer's Github extension. Once this app has been set up, you'll have everything you need to configure the extension.

Here is a quick and dirty walkthrough on the important parts of setting up the Github application.

1. Create a new Github application:

Github's documentation will be the most up to date and relevant so please see their walkthrough.

On the configuration page for the new Github application the following should be noted:

• APP ID: Take note of the id; it will be used later for extension configuration.

• Webhook URL: Should be the address of your Gofer's external extension instance and pointing to the events/github endpoint:

  ex: https://mygoferinstance.yourdomain.com/external/github

• Webhook Secret: Make this a secure, long, random string of characters and note it for future extension configuration.

• Private Keys: Generate a private key and keep the .pem file Github gives you; you'll store it in Gofer's secret store in step 3.

2. Find the installation ID

Once the Github application has been created, install it. This will give you an opportunity to configure the permissions and scope of the Github application. It is recommended that you give read-only permissions to any permissions that might include webhooks and read-write for code-suite and code-runs.

You might also utilize this Github App to perform other actions within Github with either other extensions or your own pipeline jobs. Remember to allow the correct permissions for all use cases.

The installation ID is unfortunately hidden in an event that gets sent once the Github app has been created and installed. You can find it by navigating to the settings page for the Github application and then viewing it in the "Recent Deliveries" page.

🪧 These recent deliveries only last a short amount of time, so if you take a while to check on them, they might not exist anymore. If that has happened you should be able to create another event and that will create another recent delivery.

3. Store the secrets

Put the private key and webhook secret in Gofer's global secret store. Leaving out the value prompts for it, which keeps it out of your shell history.

gofer secret global put github-app-key < ~/Downloads/myorg-gofer.2022-01-24.private-key.pem
gofer secret global put github-app-webhook-secret

4. Add the extension to Gofer's config

Add an [[extensions.install]] entry like the example above, with your app ID and installation ID and references to the two secrets. Point manifest at the github manifest for your version of Gofer.

gofer extension manifest <manifest url> --id github prints this block for you, with every setting documented.

5. Reload

gofer extension reload

Gofer shows the extension it's about to install and its settings; confirm, and it starts. If it doesn't, gofer extension list shows why and gofer extension logs github shows what the extension itself reported.

Events

Gofer's extensions have the ability to pass along event specific information in the form of environment variables that get injected into each container's run. Most of these variables are pulled from the webhook request that comes in.

Below is a breakdown of the environment variables that are passed to a run based on the event that was generated. You can find more information about the format the variables will be in by referencing the payloads for the event.

Events below are the only events that are supported.

Command Line

Gofer's main way of providing interaction is through a command line application included in the Gofer binary.

This command line tool is how you upload pipelines, view runs, upload artifacts and many other common Gofer tasks.

To view the possible commands for the Gofer pipeline simply run gofer --help.

Reading task logs

gofer task logs <pipeline_id> <run_id> <task_id> prints a task's output. If the task is still running it keeps following the log until the task finishes, so you can watch it as it goes.

For long logs you usually only want one end of it:

gofer task logs my-pipeline 3 build --tail 50     # the last 50 lines, then keep following if it's still running
gofer task logs my-pipeline 3 build | head -n 50  # the first 50 lines

--tail is handled by the server, which skips straight to the end of the log instead of sending you the whole thing. That makes it much faster than piping into tail and it works on tasks that are still running, where tail would sit waiting for a log that hasn't ended yet. The start of the log needs no help from the server, so head works fine.

The output is plain lines, so the usual tools work on it:

gofer task logs my-pipeline 3 build | grep -i error

Searching with less

For anything more than a quick look, pipe the logs into less:

gofer task logs my-pipeline 3 build | less -R

-R keeps any colors the task printed. Once it's open:

Patterns are regular expressions, and searches are case sensitive unless you start less with -i.

If the task is still running, be careful with F. Pressing Ctrl-C to stop following also stops gofer, so the log ends wherever it was at that moment. You can keep scrolling what you have, and run the command again to pick up the rest.

Opening the web UI

gofer web opens Gofer's web UI in your browser, already signed in with the token your CLI uses.

gofer web                # the front page
gofer web my-pipeline    # that pipeline's page: recent runs, tasks, and subscriptions
gofer web my-pipeline 3  # run 3 of that pipeline

Your token never goes in the link. The CLI asks Gofer for a one time code that expires after 60 seconds, and the page trades it for your token when it loads. Use --print to get the link instead of opening a browser, which is handy over SSH. If the CLI doesn't have a working token, the page opens without signing in, and you can still browse the default namespace.

Configuration

The Gofer CLI accepts configuration through a configuration file or environment variables.

When both are used, environment variables win. So if your config file sets namespace = "default" and you export GOFER_NAMESPACE=ops, the CLI uses ops. Some commands also take flags like --namespace that override both for that one command.

Environment variables

Each configuration option can be set as an environment variable by upper casing it and adding a GOFER_ prefix.

For example, setting your API token:

export GOFER_TOKEN=mysupersecrettoken
gofer token whoami

Or pointing the CLI at a different server:

export GOFER_API_BASE_URL=http://localhost:8080

Configuration file

For convenience reasons Gofer can also use a standard configuration file. The language of this file is TOML. Each option is just key = value.

Configuration file locations

Gofer reads the first of these it finds:

1. $HOME/.gofer.toml
2. $HOME/.config/gofer.toml

Development builds of Gofer read $HOME/.gofer_dev.toml and $HOME/.config/gofer_dev.toml instead, so you can work on Gofer without touching the config you use for your real server.

gofer context shows which file was read and, for each setting, whether its value came from that file, an environment variable or the default.

Configuration file options

Example configuration file

# ~/.gofer.toml
api_base_url  = "https://gofer.example.com"
namespace     = "default"
output_format = "spinner"
token         = "mysupersecrettoken"

API Reference

Gofer exposes a REST API; every endpoint lives under /api and speaks JSON unless noted otherwise. These pages are generated from the OpenAPI spec (Gofer v0.13.1) by make generate-api-docs.

Authentication

Every request needs a bearer token. See Authentication and Authorization for how tokens are created and what they can do.

Versioning

Every request must carry a gofer-api-version header. The only version right now is v0.

curl http://localhost:8080/api/system/metadata \
  -H "Authorization: Bearer $GOFER_TOKEN" \
  -H "gofer-api-version: v0"

Websockets

A few endpoints (event streaming, log following, task attach) upgrade the connection to a websocket instead of returning a JSON body. They're marked with a WS badge.

Sections

Configs

Pipeline configs are versioned configurations for a particular pipeline.

List all pipeline configs

A pipeline's config is the small program you write to configure how you want your pipeline to run.

Path parameters

Query parameters

Responses

Register a new pipeline configuration

This creates both the pipeline metadata and the initial config object.

Path parameters

Request body (required): RegisterPipelineConfigRequest

Responses

Get a specific version of a pipeline configuration

A version of 0 indicates to return the latest pipeline config.

Path parameters

Query parameters

Responses

Deploy pipeline config

Path parameters

Responses

Delete pipeline config by version

Path parameters

Responses

Deployments

A deployment represents a transition between pipeline versions

List all deployments

Path parameters

Responses

Get api deployment by id

Path parameters

Responses

Events

Gofer emits events for actions that happen within it's purview. You can use the event api to get a list of all events or request specific events.

List all events

This endpoint upgrades the connection to a websocket: after the 101 response the server streams data over the socket instead of returning a body.

Query parameters

Responses

Get api event by id

Path parameters

Responses

Delete api event by id

This route is only accessible by admin tokens.

Path parameters

Responses

Extensions

An extension is a way to give pipelines more functionality. This might include automatically running your pipeline or printing the results of a run to Slack or more. Pipelines can subscribe to one or more extensions (usually with some individual configuration) and those extensions perform actions on behalf of the pipeline.

List all extensions currently registered

Responses

Compare Gofer's config with the extensions currently running

Rereads Gofer's config file and reports, for each extension, what applying the config would change. Nothing is changed. This backs gofer extension reload. This route is only accessible for admin tokens.

Responses

Returns details about a specific extension

Path parameters

Responses

Permanently delete an extension along with its pipeline subscriptions and stored objects

Only extensions that have already been removed from Gofer's config can be purged; Gofer would just install a configured one again. This route is only accessible for admin tokens.

Path parameters

Responses

Bring one extension in line with Gofer's config

Rereads Gofer's config file and installs, updates, starts, or stops the extension to match. The request carries the plan_hash from the plan the operator reviewed; if anything changed since then (the config, the manifest, or a secret) the request is refused so the operator can look at a fresh plan. This route is only accessible for admin tokens.

Path parameters

Request body (required): ApplyExtensionRequest

Responses

Dump extension debug information

Path parameters

Responses

Retrieves logs from the extension container

This endpoint upgrades the connection to a websocket: after the 101 response the server streams data over the socket instead of returning a body.

Path parameters

Responses

Go back to the version of an extension that was running before the last apply

Meant for when a new version won't start. The extension stays different from Gofer's config until the config is fixed, so it'll show up in the next plan again. This route is only accessible for admin tokens.

Path parameters

Responses

List an extension's subscriptions

Leaves out disabled subscriptions unless include_disabled is set.

Path parameters

Query parameters

Responses

Namespaces

A namespace represents a grouping of pipelines. Normally it is used to divide teams or logically different sections of workloads. It is the highest level unit as it sits above pipelines in the hierarchy of Gofer

List all namespaces

Responses

Create a new namespace

This route is only accessible for admin tokens.

Request body (required): CreateNamespaceRequest

Responses

Get api namespace by id

Path parameters

Responses

Update a namespace's details

This route is only accessible for admin tokens.

Path parameters

Request body (required): UpdateNamespaceRequest

Responses

Delete api namespace by id

This route is only accessible for admin tokens.

Path parameters

Responses

Objects

The object store is a temporary key-vale storage mechanism for pipelines and runs. It allows the user to cache objects for the lifetime of multiple runs or for the lifetime of a single run.There are two separate types of objects, each useful for its own use case. Visit the documentation for more details on the associated lifetimes of pipeline specific and run specific objects

List all extension objects

Path parameters

Responses

Get extension object by key

Path parameters

Responses

Insert a new object into the extension object store

Path parameters

Query parameters

Request body (required): raw bytes (application/octet-stream)

Responses

Delete extension object by key

Path parameters

Responses

List all pipeline objects

Path parameters

Responses

Get pipeline object by key

Path parameters

Responses

Insert a new object into the pipeline object store

Path parameters

Query parameters

Request body (required): raw bytes (application/octet-stream)

Responses

Delete pipeline object by key

Path parameters

Responses

List all run objects

Path parameters

Responses

Get run object by key

Path parameters

Responses

Insert a new object into the run object store

Overwrites can be performed by passing the force query param.

Path parameters

Query parameters

Request body (required): raw bytes (application/octet-stream)

Responses

Delete run object by key

Path parameters

Responses

Permissions

Gofer has an RBAC system which can be utilized to give different tokens/users permissions.

List all roles

Responses

Create a new role

This route is only accessible for admin tokens.

Request body (required): CreateRoleRequest

Responses

Get api role by id

Path parameters

Responses

Update a role's details

This route is only accessible for admin tokens.

Path parameters

Request body (required): UpdateRoleRequest

Responses

Delete api role by id

This route is only accessible for admin tokens.

Path parameters

Responses

Pipelines

A pipeline is a graph of containers that accomplish some goal. Pipelines are created via a Pipeline configuration file and can be set to be run automatically via attached extensions

List all pipelines

Returns the metadata for all pipelines. If you want a more complete picture of the pipeline details combine this endpoint with the configs endpoint to grab the metadata AND the user's pipeline configuration.

Path parameters

Responses

Get pipeline by id

Path parameters

Responses

Update a pipeline's state

Path parameters

Request body (required): UpdatePipelineRequest

Responses

Delete pipeline by id

IMPORTANT: Deleting a pipeline is set to cascade. All downstream objects to the pipeline (configs, secrets, runs, tasks) will be removed as well.

Path parameters

Responses

Runs

A run is a specific execution of a pipeline at a specific point in time. A run is made up of multiple tasks that all execute according to their dependency on each other.

List all runs

Returns a list of all runs by pipeline id.

Path parameters

Query parameters

Responses

Start a run of a particular pipeline

Path parameters

Request body (required): StartRunRequest

Responses

Get run by id

Path parameters

Responses

Cancel a run by id

Path parameters

Responses

Secrets

Gofer allows user to enter secrets on both a global and pipeline scope. This is useful for workloads that need access to secret values and want a quick, convenient way to access those secrets. Global secrets are managed by admins and can grant pipelines access to secrets shared amongst many namespaces. Pipeline secrets on the other hand are only accessible from within that specific pipeline

List all pipeline secrets

Path parameters

Responses

Insert a new secret into the pipeline secret store

Path parameters

Request body (required): PutPipelineSecretRequest

Responses

Get pipeline secret by key

Path parameters

Query parameters

Responses

Delete pipeline secret by key

Path parameters

Responses

List all global secrets

Admin tokens required.

Responses

Insert a new secret into the global secret store

This route is only accessible for admin tokens.

Request body (required): PutGlobalSecretRequest

Responses

Get global secret by key

Admin token required.

Path parameters

Query parameters

Responses

Delete global secret by key

This route is only accessible for admin tokens.

Path parameters

Responses

Subscriptions

A subscription represents a pipeline's subscription to a extension.

List all subscriptions

Path parameters

Responses

Create a new subscription

Path parameters

Request body (required): CreateSubscriptionRequest

Responses

Get subscription by id

Path parameters

Responses

Update a subscription's state

Path parameters

Request body (required): UpdateSubscriptionRequest

Responses

Delete subscription by id

Path parameters

Responses

System

Routes focused on meta-information for the Gofer service

Get system parameters

Responses

Update system parameters

Request body (required): UpdateSystemPreferencesRequest

Responses

Describe current system meta-information

Return a number of internal metadata about the Gofer service itself.

Responses

Tasks

A task is the lowest unit of execution for a pipeline. A task execution is the tracking of a task, which is to say a task execution is simply the tracking of the container that is in the act of being executed.

List all task executions

Returns a list of all task executions by run.

Path parameters

Query parameters

Responses

Get task execution by id

Path parameters

Query parameters

Responses

Cancel a task execution by id

Path parameters

Query parameters

Responses

Run command on a running task execution container

This allows you to run a command on a task execution container and connect to the stdin and stdout/err for said container.

Useful for debugging.

This endpoint upgrades the connection to a websocket: after the 101 response the server streams data over the socket instead of returning a body.

Path parameters