In this article, Joseph Hall, the author of the book Extending SaltStack, explains thatwhile setting static configuration is fine and well, it can be very useful to be able to supply that data from an external source. You’ll learn about:
- Writing dynamic grains and external pillars
- Troubleshooting grains and pillars
- Writing and using SDB modules
(For more resources related to this topic, see here.)
Setting grains dynamically
Grains hold variables that describe certain aspects of a Minion. This could be information about the operating system, hardware, network, and so on. It can also contain statically-defined user data, which is configured either in /etc/salt/minion or /etc/salt/grains. It is also possible to define grains dynamically using grains modules.
Setting some basic grains
Grains modules are interesting in that as long as the module is loaded, all public functions will be executed. As each function is executed, it will return a dictionary, which contains items to be merged into the Minion’s grains.
Let’s go ahead and set up a new grains module to demonstrate. We’ll prepend the names of the return data with a z so that it is easy to find.
'''
Test module for Extending SaltStack
This module should be saved as salt/grains/testdata.py
'''
def testdata():
'''
Return some test data
'''
return {'ztest1': True}Go ahead and save this file as salt/grains/testdata.py, and then use salt-call to display all of the grains, including this one:
# salt-call --local grains.items
local:
----------
...
virtual:
physical
zmqversion:
4.1.3
ztest1:
TrueKeep in mind that you can also use grains.item to display only a single grain:
# salt-call --local grains.item ztest
local:
----------
ztest1:
TrueIt may not look like this module is very good, since this is still just static data that could be defined in the minion or grains files. But keep in mind that as with other modules, grains modules can be gated using a __virtual__() function. Let’s go ahead and set this up, along with a flag of sorts that will determine whether or not this module will load in the first place:
import os.path
def __virtual__():
'''
Only load these grains if /tmp/ztest exists
'''
if os.path.exists('/tmp/ztest'):
return True
return FalseNow run the following commands to see this in action:
# salt-call --local grains.item ztest
local:
----------
ztest:
# touch /tmp/ztest
# salt-call --local grains.item ztest
local:
----------
ztest:
TrueThis is very useful for gating the return data from an entire module, whether dynamic or, as this module currently is, static.
You may be wondering why the example checked for the existence of a file rather than checking the existing Minion configuration. This is to illustrate that the detection of certain system properties is likely to dictate how grains are set. If you want to just set a flag inside the minion file, you can pull it out of __opts__. Let’s go ahead and add this to the __virtual__() function:
def __virtual__():
'''
Only load these grains if /tmp/ztest exists
'''
if os.path.exists('/tmp/ztest'):
return True
if __opts__.get('ztest', False):
return True
return FalseRemove the old flag, and set the new one:
# rm /tmp/ztest
# echo 'ztest: True' >> /etc/salt/minion
# salt-call --local grains.item ztest
local:
----------
ztest:
TrueLet’s set up this module to return dynamic data as well. Because YAML is so prevalent in Salt, let’s go ahead and set up a function that returns the contents of a YAML file:
import yaml
import salt.utils
def yaml_test():
'''
Return sample data from /etc/salt/test.yaml
'''
with salt.utils.fopen('/etc/salt/yamltest.yaml', 'r') as fh_:
return yaml.safe_load(fh_)Save your module, and then issue the following commands to see the result:
# echo 'yamltest: True' > /etc/salt/yamltest.yaml
# salt-call --local grains.item yamltest
local:
----------
yamltest:
True(Not) Cross-calling execution modules
You may be tempted to try and cross-call an execution module from inside a grains module. Unfortunately, this won’t work. The __virtual__() function in many execution modules relies heavily on grains. Allowing grains to cross-call execution modules before Salt has decided whether or not to even call the execution module in the first place would cause circular dependencies.
Just remember: grains are loaded first, then pillars, and then execution modules. If you have code in which you plan to use two or more of these types of modules, consider setting up a library for it in the salt/utils/ directory.
The final grains module
With all of the code we’ve put together, the resulting module should look like the following:
'''
Test module for Extending SaltStack.
This module should be saved as salt/grains/testdata.py
'''
import os.path
import yaml
import salt.utils
def __virtual__():
'''
Only load these grains if /tmp/ztest exists
'''
if os.path.exists('/tmp/ztest'):
return True
if __opts__.get('ztest', False):
return True
return False
def testdata():
'''
Return some test data
'''
return {'ztest1': True}
def yaml_test():
'''
Return sample data from /etc/salt/test.yaml
'''
with salt.utils.fopen('/etc/salt/yamltest.yaml', 'r') as fh_:
return yaml.safe_load(fh_)Creating external pillars
As you know, pillars are like grains but with a key difference: grains are defined on the Minion, while pillars are defined for individual Minions, from the Master.
As far as users are concerned, there’s not a whole lot of difference here, except that pillars must be mapped to targets on the Master using the top.sls file in pillar_roots. One such mapping might look like this:
# cat /srv/pillar/top.sls
base:
'*':
- testIn this example, we would have a pillar called test defined, which might look like this:
# cat /srv/pillar/test.sls
test_pillar: TrueDynamic pillars are still mapped in the top.sls file, but that’s where the similarities end as far as configuration is concerned.
Configuring external pillars
Unlike dynamic grains, which will run so long as their __virtual__() function allows them to do so, pillars must be explicitly enabled in the master configuration file or, if running in local mode as we will be, in the minion configuration file. Let’s go ahead and add the following lines to the end of /etc/salt/minion:
ext_pillar:
- test_pillar: TrueIf we were testing this on the Master, we would have needed to restart the salt-master service. However, since we’re testing in local mode on the Minion, this will not be required.
Adding an external pillar
We’ll also need to create a simple external pillar to get started with. Go ahead and create salt/pillar/test_pillar.py with the following content:
'''
This is a test external pillar
'''
def ext_pillar(minion_id, pillar, config):
'''
Return the pillar data
'''
return {'test_pillar': minion_id}Save your work, and then test it to make sure it works:
# salt-call --local pillar.item test_pillar
local:
----------
test_pillar:
dufresneLet’s go over what’s happened here. First off, we have a function called ext_pillar(). This function is required in all external pillars. It is also the only function that is required. Any others, whether named with a preceding underscore or not, will be private to this module.
This function will always be passed three pieces of data. The first is the ID of the Minion that is requesting this pillar. You can see this in our example already: the minion_id parameter where the above example was run was dufresne. The second is a copy of the static pillars defined for this Minion. The third is an extra piece of data that was passed to this external pillar in the master (or in this case, minion) configuration file.
Let’s go ahead and update our pillar to show us what each component looks like. Change your ext_pillar() function to look like this:
def ext_pillar(minion_id, pillar, command):
'''
Return the pillar data
'''
return {'test_pillar': {
'minion_id': minion_id,
'pillar': pillar,
'config': config,
}}Save it, and then modify the ext_pillar configuration in your minion (or master) file:
ext_pillar:
- test_pillar: Alas, poor Yorik. I knew him, Horatio.Take a look at your pillar data again:
# salt-call --local pillar.item test_pillar
local:
----------
test_pillar:
----------
config:
Alas, poor Yorik. I knew him, Horatio.
minion_id:
dufresne
pillar:
----------
test_pillar:
TrueYou can see the test_pillar method that we referenced a couple of pages ago. And, of course, you can see the minion_id method, just like before. The important part here is the config.
This example was chosen to make clear where the config argument came from. When an external pillar is added to the ext_pillar list, it is entered as a dictionary, with a single item as its value. The item that is specified can be a string, boolean, integer, or float. It cannot be a dictionary or a list.
This argument is normally used to pass arguments to the pillar from the configuration file. For instance, the cmd_yaml pillar that ships with Salt uses it to define a command that is expected to return data in the YAML format, as follows:
ext_pillar:
- cmd_yaml: cat /etc/salt/testyaml.yamlIf the only thing that your pillar requires is to be enabled; then, you can just set this to True and then ignore it. However, you must still set it! Salt will expect this data to be there, and you will receive an error like this if it is not:
[CRITICAL] The "ext_pillar" option is malformedWhile minion_id, pillar, and config are all passed into the ext_pillar() function (in that order), Salt doesn’t actually care what you call the variables in your function definition. You could call them emeril, mario, and alton if you wanted (not that you would). But whatever you call them, they must all still be there.
Another external pillar
Let’s put together another external pillar so that it doesn’t get confused with our first one. This one’s job is to check the status of a web service. First, let’s write our pillar code:
'''
Get status from HTTP service in JSON format.
This file should be saved as salt/pillar/http_status.py
'''
import salt.utils.http
def ext_pillar(minion_id, pillar, config):
'''
Call a web service which returns status in JSON format
'''
comps = config.split()
key = comps[0]
url = comps[1]
status = salt.utils.http.query(url, decode=True)
return {key: status['dict']}Save this file as salt/pillar/http_status.py. Then, go ahead and update your ext_pillar configuration to point to it. For now, we’ll use GitHub’s status URL:
ext_pillar
- http_status: github https://status.github.com/api/status.jsonSave the configuration, and then test the pillar:
# salt-call --local pillar.item github
local:
----------
github:
----------
last_updated:
2015-12-02T05:22:16Z
status:
goodIf you need to be able to check the status on multiple services, you can use the same external pillar multiple times but with different configurations. Try updating your ext_pillar definition to contain these two entries:
ext_pillar
- http_status: github https://status.github.com/api/status.json
- http_status: github2 https://status.github.com/api/status.jsonNow, this can quickly become a problem. GitHub won’t be happy with you if you’re constantly hitting their status API. So, as nice as it is to get real-time status updates, you may want to do something to throttle your queries. Let’s save the status in a file and return it from there. We will check the file’s timestamp to make sure it doesn’t get updated more than once a minute.
Let’s go ahead and update the entire external pillar:
'''
Get status from HTTP service in JSON format.
This file should be saved as salt/pillar/http_status.py
'''
import json
import time
import datetime
import os.path
import salt.utils.http
def ext_pillar(minion_id, # pylint: disable=W0613
pillar, # pylint: disable=W0613
config):
'''
Return the pillar data
'''
comps = config.split()
key = comps[0]
url = comps[1]
refresh = False
status_file = '/tmp/status-{0}.json'.format(key)
if not os.path.exists(status_file):
refresh = True
else:
stamp = os.path.getmtime(status_file)
now = int(time.mktime(datetime.datetime.now().timetuple()))
if now - 60 >= stamp:
refresh = True
if refresh:
salt.utils.http.query(url, decode=True, decode_out=status_file)
with salt.utils.fopen(status_file, 'r') as fp_:
return {key: json.load(fp_)}Now we’ve set a flag called refresh, and the URL will only be hit when that flag is True. We’ve also defined a file that will cache the content obtained from that URL. The file will contain the name given to the pillar, so it will end up having a name like /tmp/status-github.json. The following two lines will retrieve the last modified time of the file and the current time in seconds:
stamp = os.path.getmtime(status_file)
now = int(time.mktime(datetime.datetime.now().timetuple()))And comparing the two, we can determine whether the file is more than 60 seconds old. If we wanted to make the pillar even more configurable, we could even move that 60 to the config parameter and pull it from comps[2].
Troubleshooting grains and pillars
While writing grains and pillars, you may encounter some difficulties. Let’s take a look at the most common problems you might have.
Dynamic grains not showing up
You may find that when you issue a grains.items command from the Master, your dynamic grains don’t show up. This can be difficult to track down, because grains are evaluated on the Minion, and any errors aren’t likely to make it back over the wire to you.
When you find that dynamic grains aren’t showing up as you expect, it’s usually easiest to log in to the Minion directly to troubleshoot. Open up a shell and try issuing a salt-call command to see whether any errors manifest themselves. If they don’t immediately, try adding –log-level=debug to your command to see whether any errors have been hiding at that level. Using a trace log level might also be necessary.
External pillars not showing up
External pillars can be a little more difficult to pick out. Using salt-call is effective for finding errors in grains because all of the code can be executed without starting up or contacting a service. But pillars come from the Master, unless you’re running salt-call in local mode.
If you are able to install your external pillar code on a Minion for testing, then the steps are the same as for checking for grains errors. But if you find yourself in a situation where the Master’s environment cannot be duplicated on a Minion, you will need to use a different tactic:
- Stop the salt-master service on the Master and then start it back up in the foreground with a debug log level:
# salt-master --log-level debug - Then, open up another shell and check the pillars for an affected Minion:
# salt <minionid> pillar.items
Any errors in the pillar code should manifest themselves in the window with salt-master running in the foreground.
Writing SDB modules
SDB is a relatively new type of module, and is ripe for development. It stands for Simple Database, and it is designed to allow data to be simple to query, using a very short URI. The underlying configuration could be as complex as necessary so long as the URI that is used to query it is as simple as possible.
Another design goal of SDB is that URIs can mask sensitive pieces of information to prevent them being stored directly inside a configuration file. For instance, passwords are often required for other types of modules, such as the mysql modules, but it is a poor practice to store passwords in files that are then stored inside a revision control system such as Git.
Using SDB to look up passwords on the fly allows references to the passwords to be stored but not the passwords themselves. This makes it much safer to store files that reference sensitive data inside revision control systems.
There is one supposed function that could be tempting to use SDB for: storing encrypted data on the Minion that cannot be read by the Master. It is possible to run agents on a Minion that require local authentication, such as typing in a password from the Minion’s keyboard or using a hardware encryption device. SDB modules can be made that make use of these agents, and due to their very nature, the authentication credentials themselves cannot be retrieved by the Master.
The problem is that the Master can access anything that a Minion subscribing to it can. While the data may be stored in an encrypted database on the Minion and while its transfer to the Master is certainly encrypted, once it gets to the Master, it can still be read in plaintext.
Getting SDB data
There are only two public functions that are used for SDB: get and set. And in truth, the only important one of these is get, since set can usually be done outside of Salt entirely. Let’s go ahead and take a look at get.
For our example, we’ll create a module that reads a JSON file and then returns the requested key from it. First, let’s set up our JSON file:
{
"user": "larry",
"password": "123pass"
}Save this file as /root/mydata.json. Then, edit the minion configuration file and add a configuration profile:
myjson:
driver: json
json_file: /root/mydata.jsonWith these two things in place, we’re ready to start writing our module. JSON has a very simple interface, so there won’t be much here:
'''
SDB module for JSON
This file should be saved as salt/sdb/json.py
'''
from __future__ import absolute_import
import salt.utils
import json
def get(key, profile=None):
'''
Get a value from a JSON file
'''
with salt.utils.fopen(profile['json_file'], 'r') as fp_:
json_data = json.load(fp_)
return json_data.get(key, None)You’ve probably noticed that we’re added a couple of extra things outside of the necessary JSON code. First, we imported something called absolute_import. This is because this file is called json.py, and it’s importing another library called json. Without absolute_import, the file would try to import itself and be unable to find the necessary functions from the actual json library.
We’ve also imported salt.utils so that we can make use of the fopen() function that ships with Salt. This is a wrapper around Python’s own open() built-in, which adds some extra error handling that Salt makes use of.
The get() function takes two arguments: key and profile. The key argument refers to the key that will be used to access the data that we need, while profile is a copy of the profile data that we save as myjson in the minion configuration file.
The SDB URI makes use of these two items. When we build this URI, it will be formatted like this:
sdb://<profile_name>/<key>For instance, if we were to use the sdb execution module to retrieve the value of key1, our command would look like this:
# salt-call --local sdb.get sdb://myjson/user
local:
larryWith this module and profile in place, we can now add lines to the minion configuration (or to grains, pillars, or even the master configuration) that look like this:
username: sdb://myjson/user
password: sdb://myjson/passwordWhen a module that uses config.get comes across an SDB URI, it will automatically translate it on the fly to the appropriate data.
Before we move on, let’s update this function a little bit to perform some error handling. If the user makes a typo in the profile (such as json_fle instead of json_file), the file being referenced doesn’t exist, or the JSON isn’t formatted correctly, then this module will start spitting out traceback messages. Let’s go ahead and handle all of this using Salt’s own CommandExecutionError:
from __future__ import absolute_import
from salt.exceptions import CommandExecutionError
import salt.utils
import json
def get(key, profile=None):
'''
Get a value from a JSON file
'''
try:
with salt.utils.fopen(profile['json_file'], 'r') as fp_:
json_data = json.load(fp_)
return json_data.get(key, None)
except IOError as exc:
raise CommandExecutionError (exc)
except KeyError as exc:
raise CommandExecutionError ('{0} needs to be configured'.format(exc))
except ValueError as exc:
raise CommandExecutionError (
'There was an error with the JSON data: {0}'.format(exc)
)The IOError method will catch problems with a path that doesn’t point to a real file. The KeyError method will catch errors with missing profile configuration (which would happen if one of the items were misspelled). The ValueError method will catch problems with an improperly formatted JSON file. This is what errors look like originally:
Traceback (most recent call last):
File "/usr/bin/salt-call", line 11, in <module>
salt_call()
File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 333, in salt_call
client.run()
File "/usr/lib/python2.7/site-packages/salt/cli/call.py", line 58, in run
caller.run()
File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 133, in run
ret = self.call()
File "/usr/lib/python2.7/site-packages/salt/cli/caller.py", line 196, in call
ret['return'] = func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/salt/modules/sdb.py", line 28, in get
return salt.utils.sdb.sdb_get(uri, __opts__)
File "/usr/lib/python2.7/site-packages/salt/utils/sdb.py", line 37, in sdb_get
return loaded_db[fun](query, profile=profile)
File "/usr/lib/python2.7/site-packages/salt/sdb/json_sdb.py", line 49, in get
with salt.utils.fopen(profile['json_fil']) as fp_:
KeyError: 'json_fil'Our code will turn them into something like this:
Error running 'sdb.get': 'json_fil' needs to be configuredSetting SDB data
The function used for set may look strange because set is a Python built-in. This means that the function cannot be called set(); it must be called something else and then given an alias using the __func_alias__ dictionary. Let’s now create a function that does nothing except returning the value to be set:
__func_alias__ = {
'set_': 'set'
}
def set_(key, value, profile=None):
'''
Set a key/value pair in a JSON file
'''
return valueThis will be enough for your purposes with read-only data, but in our case, we’re going to modify the JSON file. First, let’s look at the arguments that are passed to our function.
You already know that key points to the data to be referenced and that profile contains a copy of the profile data from the minion configuration file. And you can probably guess that value contains a copy of the data to be applied.
The value doesn’t change the actual URI; that will always be the same whether you’re getting or setting data. The execution module itself is what accepts the data to be set and then sets it. You can see that with this command:
# salt-call --local sdb.set sdb://myjson/password 321pass
local:
321passWith that in mind, let’s go ahead and make our module read the JSON file, apply the new value, and then write it back out again. For now, we’ll skip error handling in order to make it easier to read:
def set_(key, value, profile=None):
'''
Set a key/value pair in a JSON file
'''
with salt.utils.fopen(profile['json_file'], 'r') as fp_:
json_data = json.load(fp_)
json_data[key] = value
with salt.utils.fopen(profile['json_file'], 'w') as fp_:
json.dump(json_data, fp_)
return get(key, profile)This function reads the JSON file as before, updates the specific value (creating it if necessary), and then writes the file back out. When it’s finished, it returns the data using the get() function so that the user knows whether it was set properly. If it returns the wrong data, then the user will know that something went wrong. It won’t necessarily tell them what went wrong, but it will raise a red flag.
Let’s add some error handling to help the user know what went wrong. We’ll add in the error handling from the get() function now too:
def set_(key, value, profile=None): # pylint: disable=W0613
'''
Set a key/value pair in a JSON file
'''
try:
with salt.utils.fopen(profile['json_file'], 'r') as fp_:
json_data = json.load(fp_)
except IOError as exc:
raise CommandExecutionError (exc)
except KeyError as exc:
raise CommandExecutionError ('{0} needs to be configured'.format(exc))
except ValueError as exc:
raise CommandExecutionError (
'There was an error with the JSON data: {0}'.format(exc)
)
json_data[key] = value
try:
with salt.utils.fopen(profile['json_file'], 'w') as fp_:
json.dump(json_data, fp_)
except IOError as exc:
raise CommandExecutionError (exc)
return get(key, profile)Because we did all of this error handling when reading the file, by the time we get to writing it back again, we already know that the path is value, the JSON is valid, and there are no profile errors. However, there could still be errors in saving the file. Try the following:
# chattr +i /root/mydata.json
# salt-call --local sdb.set sdb://myjson/password 456passYou’ll get this output:
Error running 'sdb.set': [Errno 13] Permission denied: '/root/mydata.json'We’ve changed the attribute of the file to make it immutable (read-only), and we can no longer write to the file. Without the IOError method, we would get an ugly traceback message just like before. Removing the immutable attribute will allow our function to run properly:
# chattr -i /root/mydata.json
# salt-call --local sdb.set sdb://myjson/password 456pass
local:
456passSummary
The three areas of Salt configuration that can be hooked into using the loader system are dynamic grains, external pillars, and SDB. Grains are generated on the Minion, pillars are generated on the Master, and SDB URIs can be configured at either place.
SDB modules allow configuration to be stored outside, but referenced from, the various parts of the Salt configuration. When accessed from execution modules, they are resolved on the Minion. When accessed from Salt Cloud, they are resolved on whichever system is running Salt Cloud.
Resources for Article:
Further resources on this subject:
- Introducing Salt [article]
- Diving into Salt Internals [article]
- Networking [article]
