Mauveweb: Chopsticks - a Pythonic orchestration library

In my current role, we've been using Ansible for our orchestration and configuration management. Ansible is okay, but after several months wrestling with its extension API and being frustrated by YAML syntax ideas started popping into my head for something better.

Chopsticks represents my vision of Pythonic orchestration. It's not an orchestration framework in itself. It's more the transport layer for other orchestration systems. It's a remote procedure call (RPC) system that relies on no agent on the remote host: the agent is built dynamically on the remote host by feeding code over the SSH tunnel to the system Python.

For example, you can create an SSH Tunnel to a remote host:

fromchopsticks.tunnelimportTunneltun=Tunnel('ns1.example.com')

Then you can pass a function (any pickleable Python function), to be called on the remote host. Here I'm just calling the standard time.time() function.

importtimeprint('Time on %s:'%tun.host,tun.call(time.time))

Of course, you might want to do this in parallel on a number of hosts, and this is also built-in to Chopsticks:

fromchopsticks.groupimportGroupfromchopsticks.factsimportipgroup=Group(['web1.example.com','web2.example.com','web3.example.com',])forhost,tingroup.call(time.time).items():print('%s time:'%host,t)forhost,addringroup.call(ip).items():print('%s ip:'%host,addr)

Note that the code for chopsticks.facts does not need to be installed on the remote hosts. They will load it on demand from the orchestration host.

Effectively, Chopsticks gives you the ability to write Python programs that straddle a number of machines, all sharing a single codebase.

See the README for a summary of how this works.

SSH tunnels are not the only connection type Chopsticks supports. It communicates over stdin/stdout pipes, so can work with any system that supports these without inteference. Such as Docker (this is on Github, but not PyPI yet):

fromchopsticks.tunnelimportDockerfromchopsticks.groupimportGroupfromchopsticks.factsimportpython_versiongroup=Group([Docker('worker-1',image='python:3.4'),Docker('worker-2',image='python:3.5'),Docker('worker-3',image='python:3.6'),])forhost,python_versioningroup.call(python_version).items():print('%s Python version:'%host,python_version)