Running Pipelines Automatically

A pipeline config describes what to run, but not when. Pipelines only run when someone starts them, unless they're subscribed to an extension. Extensions watch for something to happen, like a time passing or a push to a Github repository, and start a run when it does.

Subscriptions aren't part of the pipeline config. You add them with the CLI after the pipeline exists, and they stay in place when you register new versions of the pipeline with gofer up.

Subscribing

gofer pipeline subscribe <pipeline_id> <extension_id> <label> -s key=value
  • extension_id is the extension to subscribe to. gofer extension list shows the ones your Gofer has.
  • label is a name you pick for this subscription. A pipeline can subscribe to the same extension more than once, for example to run every hour and also every Monday, and the label tells them apart.
  • -s key=value sets the subscription's parameters. Each extension has its own, listed on its page; repeat -s for more than one.

For example, to run the simple pipeline every five minutes with the interval extension:

gofer pipeline subscribe simple interval every-five-mins -s every="5m"

Or every Christmas at 1am with cron:

gofer pipeline subscribe simple cron yearly-on-xmas -s expression="0 1 25 12 * *"

Parameters can be secrets too, using the same pipeline_secret{{key}} or global_secret{{key}} strings as task variables. Gofer swaps in the real value when it sends the subscription to the extension.

Seeing and removing subscriptions

gofer fetch <pipeline_id> lists a pipeline's subscriptions along with each one's status, and so does the pipeline's page in the web UI (gofer web <pipeline_id>). active means it can trigger runs. error means the extension behind it isn't running, so nothing will fire until it's back; the reason column says what's wrong with the extension. To remove one:

gofer pipeline unsubscribe simple interval every-five-mins

To stop one from triggering runs for a while without losing its settings, disable it, and enable it again when you want it back:

gofer pipeline disable-subscription simple interval every-five-mins
gofer pipeline enable-subscription simple interval every-five-mins

What the extension passes to your run

When an extension starts a run it can pass in run variables, which your tasks receive as environment variables. The github extension, for example, passes in details about the event that triggered it. Each extension's page lists what it passes, and gofer task get <pipeline_id> <run_id> <task_id> shows exactly what a task received.

If an extension is stopped, disabled, or has failed, its subscriptions stay in place but nothing triggers them. They show as error until the extension is running again. If a pipeline that should be running on its own isn't, start with gofer fetch <pipeline_id>.