The choice of Go for my last project was driven by its ability to cross-compile code into static binary. A script pushes stable versions on Github releases or Bintray and anyone can wget the package and use it right away.
One of the important distinctions between Influx and some other time series solutions is that it doesn’t require any other software to install and run. This is one of the many wins that Influx gets from choosing Go as its implementation language. – Paul Dix
This “static linking” awesomeness has a cost though. No evaluation at runtime, every piece of features are frozen once compiled. However a developer might happen to need more flexibility. In this post, we will study several use-cases and implementations where Go dynamic extensions unlock great features for your projects.
Configuration
Gulp is a great example of the benefits of configuration as code (more control, easier to extend). Thanks to gopher-lua, we’re going to implement this behavior. Being our first step, let’s write a skeleton for our investigations.
package main
import (
"log"
"github.com/yuin/gopher-lua"
)
// LuaPlayground exposes a bridge to Lua.
type LuaPlayground struct {
VM *lua.LState
}
func main() {
// initialize lua VM 5.1 and compiler
L := lua.NewState()
defer L.Close()
}Gopher-lua let us call Lua code from Go and share information between each environments. The idea is to define the app configuration as a convenient scripting language like the one below.
-- save as conf.lua
print("[lua] defining configuration")
env = os.getenv("ENV")
log = "debug"
plugins = { "plugin.lua" }
Now we can read those variables from Go.
// DummyConf is a fake configuration we want to fill
type DummyConf struct {
Env string
LogLevel string
Plugins *lua.LTable
}
// Config evaluates a Lua script to build a configuration structure
func (self *LuaPlayground) Config(filename string) *DummyConf {
if err := self.VM.DoFile(filename); err != nil {
panic(err)
}
return &DummyConf{
Env: self.VM.GetGlobal("env").String(),
LogLevel: self.VM.GetGlobal("log").String(),
Plugins: self.VM.GetGlobal("plugins").(*lua.LTable),
}
}
func main() {
// [...]
playground := LuaPlayground{ VM: L }
conf := playground.Config("conf.lua")
log.Printf("loaded configuration: %vn", conf)
}Using a high level scripting language gives us great flexibility to initialize an application. While we only exposed simple assignments, properties could be fetched from services or computed at runtime.
Scripting
Heka ‘s sandbox constitutes a broader approach to Go plugins. It offers an isolated environment where developers have access to specific methods and data to control Heka’s behavior.
This strategy exposes an higher level interface to contributors without recompilation. The following code snippet extends our existing LuaPlayground structure with such skills.
// Log is a go function lua will be able to run
func Log(L *lua.LState) int {
// lookup the first argument
msg := L.ToString(1)
log.Println(msg)
return 1
}
// Scripting exports Go objects to Lua sandbox
func (self *LuaPlayground) Scripting(filename string) {
// expose the log function within the sandbox
self.VM.SetGlobal("log", self.VM.NewFunction(Log))
if err := self.VM.DoFile(filename); err != nil {
panic(err)
}
}
func main() {
// [...]
playground.Scripting("script.lua")
}
Lua code are now able to leverage the disruptive Go function Log.
-- save as script.lua
log("Hello from lua !")
This is obviously a scarce example intended to show the way. Following the same idiom, gopher-lua let us export to Lua runtime complete modules, channels, Go structures. Therefor we can hide and compile implementation details as a Go library, while business logic and data manipulation is left to a productive and safe scripting environment.
This idea leads us toward another pattern : hooks. As an illustration, Git is able to execute arbitrary scripts when such files are found under a specific directory, on specific events (like running tests before pushing code). In the same spirit, we could program a routine to list and execute files in a pre-defined directory. Moving a script in this folder, therefore, would activate a new hook. This is also the strategy Dokku leverages.
Extensions
This section takes things upside down. The next piece of code expects a Lua script to define its methods. Those components become plug-and-play extensions or components one could replace, activate or deactivate.
// [...]
// Call executes a function defined in Lua namespace
func (self *LuaPlayground) Call(method string, arg string) string {
if err := self.VM.CallByParam(lua.P{
Fn: self.VM.GetGlobal(method),
NRet: 1,
Protect: true,
}, lua.LString(arg) /* method argument */ ); err != nil {
panic(err)
}
// returned value
ret := self.VM.Get(-1)
// remove last value
self.VM.Pop(1)
return ret.String()
}
// Extend plugs new capabilities to this program by loading the given script
func (self *LuaPlayground) Extend(filename string) {
if err := self.VM.DoFile(filename); err != nil {
panic(err)
}
log.Printf("Identity: %vn", self.Call("lookupID", "mario"))
}
func main() {
// [...]
playground.Extend("extension.lua")
}An interesting use-case for such feature would be swappable backend. A service discovery application, for example, might use a key/value storage. One extension would perform requests against Consul, while another one would fetch data from etcd. This setup would allow an easier integration into existing infrastructures.
Alternatives
Executing arbitrary code at runtime brings the flexibility we can expect from language like Python or Node.js, and popular projects developed their own framework.
Hashicorp reuses the same technic throughout its Go projects. Plugins are standalone binaries only a master process can run. Once launched, both parties use RPC to communicate data and commands. This approach proved to be a great fit in the open-source community, enabling experts to contribute drivers for third-party services.
An other take on Go plugins was recently pushed by InfluxDB with Telegraf, a server agent for reporting metrics. Much closer to OOP, plugins must implement an interface provided by the project. While we still need to recompile to register new plugins, it eases development by providing a dedicated API.
Conclusion
The release of Docker A.7 and previous debates show the potential of Go extensions, especially in open-source projects where author wants other developers to contribute features in a manageable fashion.
This article skimmed several approach to bypass static go binaries and should feed some further ideas. Being able to just drop-in an executable and instantly use a new tool is a killer feature of the language and one should be careful if scripts became dependencies to make it work. However dynamic code execution and external plugins keep development modular and ease developers on-boarding. Having those trade-off in mind, the use-cases we explored could unlock worthy features for your next Go project.
About the Author
Xavier Bruhiere is a Lead Developer at AppTurbo in Paris, where he develops innovative prototypes to support company growth. He is addicted to learning, hacking on intriguing hot techs (both soft and hard), and practicing high intensity sports.
