In this article by Joseph Hall, the author of the book Mastering SaltStack Second Edition, we learn about the basics of Thorium and how it helps us in a Salt ecosystem. We are also introduced to the Salt API and how to set up its components as well as creating security certificates.

(For more resources related to this topic, see here.)

Using Thorium

The Thorium system is another component of Salt with the ability to watch the event bus and react based on what it sees there. But the ideas behind it are much different than with the reactor.

A word on engines

Thorium is one of the engines that started shipping with Salt in version 2016.3. Engines are a type of long-running process that can be written to work with the master or minion. Like other module types, they have access to the Salt configuration and certain Salt subsystems.

Engines are separate processes that are managed by Salt. The event reactor runs inside the Salt processes themselves, which means that long-running reactor operations can affect the rest of Salt. Because Thorium is an engine, it does not suffer from this limitation.

Looking at Thorium basics

Like the reactor, Thorium watches the event bus. But unlike the reactor, which is configured entirely via SLS files, Thorium uses its own subsystem of modules (which are written in Python) and SLS files. Because these modules and SLS files use the state compiler, much of the functionality has been carried over.

In order to use Thorium, there are a few steps that you must complete. These steps work together to form the basis of your Thorium setup, so be careful not to skip any.

Enabling Thorium

First, as an engine, you need to enable Thorium in the master configuration file using the engines directive:

engines:
  - thorium: {}

Because Thorium is so heavily configured using its own files, no configuration needs to be passed in at this point. However, engines do need a dictionary of some sort passed in, so we pass in an empty one.

Setting up the Thorium directory tree

With Thorium configured, we need to create a directory to store Thorium SLS files in. By default, this is /srv/thorium/. Go ahead and create that:

# mkdir /srv/thorium/

If you’d like to change this directory, you may do so in the master configuration file:

thorium_roots:
  base:
    - /srv/thorium-alt/

Like the state system, Thorium requires a top.sls file. This is the first of many similarities you’ll find between the two subsystems. As with /srv/salt/top.sls, you need to specify an environment, a target, and a list of SLS files:

base:
  '*':
    - thorium_test

To be honest, the environment and target really don’t mean much; they are artifacts from the state system, which weren’t designed to do anything special inside of Thorium. That said, the target does actually have some useful purposes.

The target here doesn’t refer to any minions. Rather, it refers to the master that this top file applies to. For example, if your master’s ID is moe and you set a target of curly, then this top file won’t be evaluated for that master. If you were to check the /var/log/salt/master file, you would find this:

Sound confusing? In a single-master, non-syndicated environment, it probably is. In such an environment, go ahead and set the target to *. But in an environment in which multiple masters are present, you may wish to divide the workload between them. Take a look at this top.sls file:

base:
  'monitoring-master':
    - alerts
    - graphs
  'packaging-master':
    - redhat-pkgs
    - debian-pkgs

In this multi-master environment, the masters may work in concert to manage jobs among the minions, but one master will also be tasked with looking for monitoring-related events and processing them, while the other will handle packaging-related events.

There is a component of Thorium that we haven’t discussed yet called the register. We’ll get to it in a moment, but this is a good time to point out that the register is not shared. This means that if you assign two different masters to handle the same events, any actions performed by one master will be invisible to the other. The right hand won’t know what the left is doing, as it were.

As you may expect, the list following each target specifies a set of SLS files to be evaluated. But whereas state SLS files are only evaluated when you kick off a state run (state.highstate, for instance), Thorium SLS files are evaluated at regular intervals. By default, these intervals are set to every half second. You can change that interval in the master configuration file:

thorium_interval: 0.5

Once you have your top.sls file configured, it’s time to set up some SLS files.

Writing Thorium SLS files

Let’s go ahead and create /srv/thorium/thorium-test.sls with the following content in it:

shell_test:
  local.cmd:
    - tgt: myminion
    - func: cmd.run
    - arg:
      - echo 'thorium success' > /tmp/thorium.txt

I wouldn’t restart your master yet if I were you. First, let’s talk about what we’re looking at here.

This should look very familiar to you, with a few differences. As you would expect, shell_test is the ID of this code block; local.cmd refers to the module and function that will be used, and everything that follows is arguments to that function.

The local module is a Thorium-specific module. Execution, state, runner, and other modules are not available in Thorium without using a Thorium module that wraps them. The local module is one such wrapper, which provides access to execution modules. As such, tgt refers to the target that the module will be executed on, func is the module and function that will be executed, and arg is a list of ordered arguments. If you like, you may use kwarg instead to specify keyword arguments.

Because state modules are accessed via the state execution module, local.cmd would also be used to kick those off; runner.cmd is also available to issue commands using the runner subsystem.

Now, why did I tell you not to restart your master yet? Because if you did, this SLS file would run every half second, writing out to /tmp/thorium.txt over and over again. In order to keep it from running so often, we need to gate it somehow.

Using requisites

Because Thorium uses the state compiler, all state requisites are available, and they all function as you would expect. Let’s go ahead and add another code block and alter our first one a little bit:

checker:
  check.event:
    - value: trigger
    - name: salt/thorium/*/test

shell_test:
  local.cmd:
    - tgt: myminion
    - func: cmd.run
    - arg:
      - echo 'thorium success' > /tmp/thorium.txt
    - require:
      - check: checker

The check module has a number of functions that are designed to compare a piece of data against an event and return True if the specified conditions are met.

In this case, we’re using check.event, which looks at a given tag and returns True if an event comes in and matches it. The tag that we are looking for is salt/thorium/*/test, which is intended to look for salt/thorium/<minion_id>/test. The event must also have a variable called checker in its payload, with a value of trigger.

We have also added a require requisite to the shell_test code block, which will prevent that block from running unless the right event comes in. Now that we’re set up, go ahead and restart the master and a minion called myminion, and issue the following command from the minion:

# salt-call event.fire_master '{"checker":"trigger"}' 'salt/thorium/myminion/test'
local:
    True

You may need to wait a second or two for the event to be processed and the command from shell_test to be sent. But then you should be able to see a file called /tmp/thorium.txt and read its contents:

# cat /tmp/thorium.txt
thorium success

This particular SLS, as it is now, mimics the functionality of the reactor system, albeit with a slightly more complex setup. Let’s take a moment now to go beyond the reactor.

Using the register

Thorium isn’t just another reactor. Even if it were, just running in its own process space makes it more valuable than the old reactor. But the true value of Thorium comes with the register.

The register is Thorium’s own in-memory database, which persists across executions. A value that is placed in the register at one point in time is will still be there a half hour later unless the master is restarted.

Is the register really that fragile? At the moment, yes. And as I stated before, it’s also not shared between systems. However, it is possible to make a copy of the register on disk by adding the following code block:

myregisterfile:
  file.save

This will cause the register to be written to a file called myregisterfile at the following location:

/var/cache/salt/master/thorium/saves/myregisterfile

At the time of writing this, that file will not be reloaded into memory when the master restarts.

We’re going to go ahead and alter our SLS file. The shell_test code block doesn’t need to change, but the checker code block will. Remove the name field and change the function from check.event to check.contains:

checker:
  check.contains:
    - value: trigger

We’re still looking for a payload with a variable called checker and a value called checker, but we’re going to look at the tag somewhere else:

myregister:
  reg.set:
    - add: checker
    - match: salt/thorium/*/test

In this code block, myregister is the name of the register that you’re going to write to. The reg.set function will add a variable to that register that contains the specified piece of the payload. In this case, it will grab the variable from the payload called checker and add its associated value. However, it will only add this information to the registry if the tag on the event in question matches the match specification (salt/thorium/*/test).

Go ahead and restart the master, and then fire the same event to the master:

# salt-call event.fire_master '{"checker":"trigger"}' 'salt/thorium/myminion/test'
local:
    True

If you’ve added the file.save code block from before, we can go ahead and take a look at the register:

# cat /var/cache/salt/master/thorium/saves/myregisterfile
{"myregister": {"val": "set(['trigger'])"}}

Looking forward

The Thorium system is pretty new, so it’s still filling out. The value of the registry is that data can be aggregated to it and analyzed in real time. Unfortunately, at the time of writing this, the functions to analyze that data do not yet exist.

Understanding the Salt API

We’ve spent some time looking at how to send requests, but many users would argue that receiving requests is just as important, if not more so. Let’s take a moment to understand the Salt API.

What is the Salt API?

Very simply, the Salt API is a REST interface wrapped around Salt. But that doesn’t tell you the whole story. The salt command is really just a command-line interface for Salt. In fact, each of the other Salt commands (salt-call, salt-cloud, and so on) is really just a way to access various parts of Salt from the command line.

The Salt API provides a way to access Salt from a different interface: HTTP (or HTTPS, preferably). Because web protocols are so ubiquitous, the Salt API allows software, written in any language that has the capability of interacting with web servers, to take advantage of it.

Setting up the Salt API

Being a REST interface, the Salt API acts as a web server over and above Salt. But it doesn’t actually provide the server interface itself. It uses other web frameworks to provide those services and then acts as more of a middleman between them and Salt. The modules that are supported for this are:

• CherryPy
• Tornado
• WSGI

These modules are set up in the master configuration file. Each has its own set of configuration parameters and possible dependencies. Let’s take a look at each one.

CherryPy

CherryPy is a minimalist web framework that is designed to be very Pythonic. Because it is based around creating web code in the same way that other Python code is created, it is said to result in code that is much smaller and more quickly developed. It has a mature codebase and a number of notable users. It has also been the de facto module of the Salt API for some time.

This module does require that the CherryPy package (usually called python-cherrypy) be installed.

The basic setup for CherryPy doesn’t involve much configuration. At a minimum, you should have the following:

rest_cherrypy:
  port: 8080
  ssl_crt: /etc/pki/tls/certs/localhost.crt
  ssl_key: /etc/pki/tls/certs/localhost.key

We’ll discuss creating certificates in a moment, but first let’s talk about configuration in general. There are a number of configuration parameters available for this module, but we’ll focus on the more common ones here:

• port: This is required. It’s the port for the Salt API to listen on.
• host: Normally, the Salt API listens on all available interfaces (0.0.0.0). If you are in an an environment where you need to provide services only to one interface, then provide the IP address (that is, 10.0.0.1) here.
• ssl_crt: This is the path to your SSL certificate. We’ll cover this in a moment.
• ssl_key: This is the path to the private key for the SSL certificate. Again, we’ll cover this in a moment.
• debug: If you are setting up the Salt API for the first time, setting this to True can be very helpful. But once you are up and running, make sure to remove this option or explicitly set it to False.
• disable_ssl: It is highly recommended that the default value of False be used here. Even when just getting started, self-signed certificates are better than setting this to True. Why? Because nothing is as permanent as temporary, and at least self-signed certificates will remind you each time that you need to get a real set of certificates in place. Don’t be complacent for the sake of learning.
• root_prefix: Normally, the Salt API will serve from the root path of the server (that is, https://saltapi.example.com/), but if you have several applications that you’re serving from the same host or you just want to be more specific, you can change this. The default is /, but you could set it to /exampleapi in order to serve REST services from https://saltapi.example.com/exampleapi, for example.
• webhook_url: If you are using webhooks, they need their own entry point. By default, this is set to /hook, which in our example would serve from https://saltapi.example.com/hook.
• webhook_disable_auth: Normally, the Salt API requires authentication, but this is quite commonly not possible with third-party applications that need to call it over a webhook. This allows webhooks to not require authentication. We’ll go more in depth on this in a moment.

Tornado

Tornado is a somewhat newer framework that was written by Facebook. It is also newer than Salt but is quickly becoming the web framework of choice inside Salt itself. In fact, it is used so much inside Salt that it is now considered a hard dependency for Salt and will be available on all newer installations.

Tornado doesn’t have as many configuration options inside the Salt API as CherryPy. The ones that are supported (as defined in the CherryPy section) are:

• port
• ssl_crt
• ssl_key
• debug
• disable_ssl

While the Tornado module doesn’t support nearly as much functionality as the CherryPy module just yet, keep an eye on it; it may become the new de facto Salt API module.

WSGI

WSGI, or Web Server Gateway Interface, is a Python standard, defined in PEP 333. Direct support for it ships with Python itself, so no external dependencies are required, but this module is also pretty basic. The only configuration option to worry about here is port.

However, this module is useful in that it allows the Salt API to be run under any WSGI-compliant web server, such as Apache with mod_wsgi or Nginx with FastCGI. Because this module does not provide any sort of SSL-based security, it is recommended that one of these options be used, with those third-party web servers being properly configured with the appropriate SSL settings.

Creating SSL certificates

It is highly advisable to use an SSL certificate for the Salt API even if you currently only plan to use it on a local, secured network. You should probably also purchase a certificate that is signed by a certificate authority (CA). When you get to this point, the CA will provide instructions on how to create one using their system. However, for now, we can get by with a self-signed certificate.

There are a number of guides online for creating self-signed certificates, but finding one that is easy to understand is somewhat more difficult. The following steps will generate both an SSL certificate and the key to use it on a Linux system:

First, we’ll need to generate the key. Don’t worry about the password—just enter one for now, take note of it, and we’ll strip it out in a moment.

# openssl genrsa -des3 -out server.key 2048
Generating RSA private key, 2048 bit long modulus
................++++++
..............................................++++++
e is 65537 (0x10001)
Enter pass phrase for server.key:
Verifying - Enter pass phrase for server.key:

Once you have the key, you need to use it to generate a certificate signing request, or CSR. You will be asked a number of questions about you that are important if you want a certificate signed by a CA. On your internal network, it’s somewhat less important.

# openssl req -new -key server.key -out server.csr
Enter pass phrase for server.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Utah
Locality Name (eg, city) []:Salt Lake City
Organization Name (eg, company) [Internet Widgits Pty Ltd]:My Company, LLC
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:[email protected]
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

At this point, we can go ahead and strip the password from the key.

# cp server.key server.key.org
# openssl rsa -in server.key.org -out server.key
Enter pass phrase for server.key.org:	
writing RSA fkey

Finally, we’ll create a self-signed certificate.

# openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
Signature ok
subject=/C=US/ST=Utah/L=Salt Lake City/O=My Company, LLC/[email protected]
Getting Private key

At this point, you will have four files:

• server.crt
• server.csr
• server.key
• server.key.org

Copy server.crt to the path specified for ssl_crt and server.key to the path specified for ssl_key.

Summary

In this article, we have learned how to setup the Thorium directory tree, using requisites and also using the register. We have also studied how to setup the Salt API. The modules that are supported for Salt API are CherryPy, Tornado, and WSGI. We have also gone through creating SSL certificates that is highly advisable to use an SSL certificate for the Salt API.

Resources for Article:

---

Further resources on this subject:

• Salt Configuration [article]
• Diving into Salt Internals [article]
• Introducing Salt [article]

---