In this article by Jaime Soriano Pastor and Allesandro Frachezi, the authors of Extending Puppet – Second Edition, you will learn that the modules reusability is a topic that has got more and more attention in the past few years; as more people started to use Puppet, more evident became the need of having some common and shared code to manage common things.
The main characteristics of reusable modules are:
- They can be used by different people without the need to modify their content
- They support different OSes, and allow easy extension to new ones
- They allow users to override the default files provided by the module
- They might have an opinionated approach to the managed resources but don’t force it
- They follow a single responsibility principle and should manage only the application they are made for
- Reusability, we must underline, is not an all-or-nothing feature; we might have different levels of reusability to fulfill the needs of a variant percentage of users.
- For example, a module might support Red Hat and Debian derivatives, but not Solaris or AIX; Is it reusable? If we use the latter OSes, definitively not, if we don’t use them, yes, for us it is reusable.
- I am personally a bit extreme about reusability, and according to me, a module should also:
- Allow users to provide alternative classes for eventual dependencies from other modules, to ease interoperability
- Allow any kind of treatment of the managed configuration files, be that file- or setting-based
- Allow alternative installation methods
- Allow users to provide their own classes for users or other resources, which could be managed in custom and alternative ways
- Allow users to modify the default settings (calculated inside the module according to the underlining OS) for package and service names, file paths, and other more or less internal variables that are not always exposed as parameters.
- Expose parameters that allow removal of the resources provided by the module (this is a functionality feature more than a reusability one)
- Abstract monitoring and firewalling features so that they are not directly tied to specific modules or applications
Managing files
Everything is a file in UNIX, and Puppet most of the times manages files.
A module can expose parameters that allow its users to manipulate configuration files and it can follow one or both the file/setting approaches, as they are not alternative and can coexist.
To manage the contents of a file, Puppet provides different alternative solutions:
- Use templates, populated with variables that come from parameters, facts, or any scope (argument for the File type: content => template(‘modulename/path/templatefile.erb’)
- Use static files, served by the Puppet server
- Manage the file content via concat (https://github.com/puppetlabs/puppetlabs-concat) a module that provides resources that allow to build a file joining different fragments.
- Manage the file contents via augeas, a native type that interfaces with the Augeas configuration editing tool (http://augeas.net/)
- Manage the contents with alternative in-file line editing tools
- For the first two cases, we can expose parameters that allow to define the module’s main configuration file either directly via the source and content arguments, or by specifying the name of the template to be passed to the template() function:
class redis ( $config_file = $redis::params::file, $config_file_source = undef, $config_file_template = undef, $config_file_content = undef, ) {
- Manage the configuration file arguments with:
$managed_config_file_content = $config_file_content ? { undef => $config_file_template ? { undef => undef, default => template($config_file_template), }, default => $config_file_content, }
- The $managed_config_file_content variable computed here takes the value of the $config_file_content, if present; otherwise, it uses the template defined with $config_file_template. If also this parameter is unset, the value is undef:
if $redis::config_file { file { 'redis.conf': path => $redis::config_file, source => $redis::config_file_source, content => $redis::managed_config_file_content, } } }
- In this way, users can populate redis.conf via a custom template (placed in the site module):
class { 'redis': config_file_template => 'site/redis/redis.conf.erb', }
- Otherwise, they can also provide the content attribute directly:
class { 'redis': config_file_content => template('site/redis/redis.conf.erb'), }
- Finally, they can also provide a fileserver source path:
class { 'redis': config_file_source => 'puppet:///modules/site/redis/redis.conf', }
- In case users prefer to manage the file in other ways (augeas, concat, and so on), they can just include the main class, which, by default, does not manage the configuration file’s contents and uses whatever method to alter them:
class { 'redis': }
A good module could also provide custom defines that allow easy and direct ways to alter configuration files’ single lines, either using Augeas or other in-file line management tools.
Managing configuration hash patterns
If we want a full infrastructure as data setup and be able to manage all our configuration settings as data, we can follow two approaches, regarding the number, name, and kind of parameters to expose:
- Provide a parameter for each configuration entry we want to manage
- Provide a single parameter that expects a hash where any configuration entry may be defined
The first approach requires a substantial and ongoing effort, as we have to keep our module’s classes updated with all the current and future configuration settings our application may have.
Its benefit is that it allows us to manage them as plain and easily readable data on, for example, Hiera YAML files. Such an approach is followed, for example, by the OpenStack modules (https://github.com/stackforge) where the configurations of the single components of OpenStack are managed on a settings-based approach, which is fed by the parameters of various classes and subclasses.
For example, the Nova module (https://github.com/stackforge/puppet-nova) has many subclasses where the parameters that map to Nova’s configuration entries are exposed and are applied via the nova_config native type, which is a basically a line editing tool that works line by line.
An alternative and quicker approach is to just define a single parameter, like config_file_options_hash that accepts any settings as a hash:
class openssh (
$config_file_options_hash = { },
}Then, manage in a custom template the hash, either via a custom function, like the hash_lookup() provided by the stdmod shared module (https://github.com/stdmod/stdmod):
# File Managed by Puppet
[...]
Port <%= scope.function_hash_lookup(['Port','22']) %>
PermitRootLogin <%= scope.function_hash_lookup(['PermitRootLogin','yes']) %>
UsePAM <%= scope.function_hash_lookup(['UsePAM','yes']) %>
[...]Otherwise, refer directly to a specific key of the config_file_options_hash parameter:
Port <%= scope.lookupvar('openssh::config_file_options_hash')['Port'] ||= '22' %>
PermitRootLogin <%= scope.lookupvar('openssh::config_file_options_hash')['PermitRootLogin'] ||= 'yes' %>
UsePAM <%= scope.lookupvar('openssh::config_file_options_hash')['UsePAM'] ||= 'yes' %>
[...]Needless to say that Hiera is a good place to define these parameters; on a YAML-based backend, we can set these parameters with:
---
openssh::config_file_template: 'site/openssh/sshd_config.erb'
openssh::config_file_options_hash:
Port: '22222'
PermitRootLogin: 'no'Otherwise, if we prefer to use an explicit parameterized class declaration:
class { 'openssh':
config_file_template => 'site/openssh/sshd_config.erb'
config_file_options_hash => {
Port => '22222',
PermitRootLogin => 'no',
}
}Managing multiple configuration files
An application may have different configuration files and our module should provide ways to manage them. In these cases, we may have various options to implement in a reusable module:
- Expose parameters that let us configure the whole configuration directory
- Expose parameters that let us configure specific extra files
- Provide a general purpose define that eases management of configuration files
To manage the whole configuration directory these parameters should be enough:
class redis (
$config_dir_path = $redis::params::config_dir,
$config_dir_source = undef,
$config_dir_purge = false,
$config_dir_recurse = true,
) {
$config_dir_ensure = $ensure ? {
'absent' => 'absent',
'present' => 'directory',
}
if $redis::config_dir_source {
file { 'redis.dir':
ensure => $redis::config_dir_ensure,
path => $redis::config_dir_path,
source => $redis::config_dir_source,
recurse => $redis::config_dir_recurse,
purge => $redis::config_dir_purge,
force => $redis::config_dir_purge,
}
}
}Such a code would allow providing a custom location, on the Puppet Master, to use as source for the whole configuration directory:
class { 'redis':
config_dir_source => 'puppet:///modules/site/redis/conf/',
}Provide a custom source for the whole config_dir_path and purge any unmanaged config file; all the destination files not present on the source directory would be deleted. Use this option only when we want to have complete control on the contents of a directory:
class { 'redis':
config_dir_source => [
"puppet:///modules/site/redis/conf--${::fqdn}/",
"puppet:///modules/site/redis/conf-${::role}/",
'puppet:///modules/site/redis/conf/' ],
config_dir_purge => true,
}Consider that the source files, in this example, placed in the site module according to a naming hierarchy that allows overrides per node or role name, can only be static and not templates.
If we want to provide parameters that allow direct management of alternative extra files, we can add parameters such as the following (stdmod compliant):
class postgresql (
$hba_file_path = $postgresql::params::hba_file_path,
$hba_file_template = undef,
$hba_file_content = undef,
$hba_file_options_hash = { } ,
) { […] }Finally, we can place in our module a general purpose define that allows users to provide the content for any file in the configuration directory.
Here is an example https://github.com/example42/puppet-pacemaker/blob/master/manifests/conf.pp
The usage is as easy as:
pacemaker::conf { 'authkey':
source => 'site/pacemaker/authkey',
}Managing users and dependencies
Sometimes a module has to create a user or have some prerequisite packages installed in order to have its application running correctly.
These are the kind of “extra” resources that can create conflicts among modules, as we may have them already defined somewhere else in the catalog via other modules.
For example, we may want to manage users in our own way and don’t want them to be created by an application module, or we may already have classes that manage the module’s prerequisite.
There’s not a universally defined way to cope with these cases in Puppet, if not the principle of single point of responsibility, which might conflict with the need to have a full working module, when it requires external prerequisites.
My personal approach, which I’ve not seen being used around, is to let the users define the name of alternative classes, if any, where such resources can be managed.
On the code side, the implementation is quite easy:
class elasticsearch (
$user_class = 'elasticsearch::user',
) { [...]
if $elasticsearch::user_class {
require $elasticsearch::user_class
}Also, of course, in elasticsearch/manifests/user.pp, we can define the module’s default elasticsearch::user class.
Module users can provide custom classes with:
class { 'elasticsearch':
user_class => 'site::users::elasticsearch',
}Otherwise, they decide to manage users in other ways and unset any class name:
class { 'elasticsearch':
user_class => '',
}Something similar can be done for a dependency class or other classes.
In an outburst of a reusability spree, in some cases, I added parameters to let users define alternative classes for the typical module classes:
class postgresql (
$install_class = 'postgresql::install',
$config_class = 'postgresql::config',
$setup_class = 'postgresql::setup',
$service_class = 'postgresql::service',
[… ] ) { […] }Maybe this is really too much, but, for example, giving users the option to define the install class to use, and have it integrated in the module’s own relationships logic, may be useful for cases where we want to manage the installation in a custom way.
Managing installation options
Generally, it is recommended to always install applications via packages, eventually to be created onsite when we can’t find fitting public repositories.
Still, sometimes, we might need to, have to, or want to install an application in other ways; for example just downloading its archive, extracting it, and eventually compiling it.
It may not be a best practice, but still it can be done, and people do it.
Another reusability feature a module may provide is alternative methods to manage the installation of an application. Implementation may be as easy as:
class elasticsearch (
$install_class = 'elasticsearch::install',
$install = 'package',
$install_base_url = $elasticsearch::params::install_base_url,
$install_destination = '/opt',
) {These options expose both the install method to be used, the name of the installation class (so that it can be overridden), the URL from where to retrieve the archive, and the destination at which to install it.
In init.pp, we can include the install class using the parameter that sets its name:
include $install_classIn the default install class file (here install.pp) manage the install parameter with a case switch:
class elasticsearch::install {
case $elasticsearch::install {
package: {
package { $elasticsearch::package:
ensure => $elasticsearch::managed_package_ensure,
provider => $elasticsearch::package_provider,
}
}
upstream: {
puppi::netinstall { 'netinstall_elasticsearch':
url => $elasticsearch::base_url,
destination_dir => $elasticsearch::install_destination,
owner => $elasticsearch::user,
group => $elasticsearch::user,
}
}
default: { fail('No valid install method defined') }
}
}The puppi::netinstall defined in the preceding code comes from a module of mine (https://github.com/example42/puppi) and it’s used to download, extract, and eventually execute custom commands on any kind of archive.
Users can therefore define which installation method to use with the install parameter and they can even provide another class to manage in a custom way the installation of the application.
Managing extra resources
Many times, we have in our environment some customizations that cannot be managed just by setting different parameters or names. Sometimes, we have to create extra resources, which no public module may provide as they are too custom and specific.
While we can place these extra resources in any class, we may include in our nodes; it may be useful to link this extra class directly to our module, providing a parameter that lets us specify the name of an extra custom class, which, if present, is included (and contained) by the module:
class elasticsearch (
$my_class = undef,
) { [...]
if $elasticsearch::my_class {
include $elasticsearch::my_class
Class[$elasticsearch::my_class] ->
Anchor['elasticsearch::end']
}
}Another method to let users create extra resources by passing a parameter to a class is based on the create_resources function. We have already seen it; it creates all the resources of a given type from a nested hash where their names and arguments can be defined. Here is an example from https://github.com/example42/puppet-network:
class network (
$interfaces_hash = undef,
[…] ) { […]
if $interfaces_hash {
create_resources('network::interface', $interfaces_hash)
}
}In this case, the type used is network::interface (provided by the same module) and it can be fed with a hash. On Hiera, with the YAML backend, it could look like this:
---
network::interfaces_hash:
eth0:
method: manual
bond_master: 'bond3'
allow_hotplug: 'bond3 eth0 eth1 eth2 eth3'
eth1:
method: manual
bond_master: 'bond3'
bond3:
ipaddress: '10.10.10.3'
netmask: '255.255.255.0'
gateway: '10.10.10.1'
dns_nameservers: '8.8.8.8 8.8.4.4'
bond_mode: 'balance-alb'
bond_miimon: '100'
bond_slaves: 'none'Summary
As we can imagine, the usage patterns that such a function allows are quite wide and interesting. Being able to base, on pure data, all the information we need to create a resource may definitively shift most of the logic and the implementation that is done with Puppet code and normal resources to the data backend.
