In this article by Thomas Uphill, author of the book Troubleshooting Puppet, we will look at how the various parts of a module may cause issues. As a Puppet developer or a system administrator, modules are how you deliver your code to the nodes. Modules are great for organizing your code into manageable chunks, but modules are also where you’ll see most of your problems when troubleshooting. Most modules contain classes in a manifests directory, but modules can also include custom facts, functions, types, providers, as well as files and templates. Each of these components can be a source of error. We will address each of these components in the following sections, starting with classes.

(For more resources related to this topic, see here.)

In Puppet, the namespace of classes is referred to as the scope. Classes can have multiple nested levels of subclasses. Each class and subclass defines a scope. Each scope is separate. To refer to variables in a different scope, you must refer to the fully scoped variable name. For instance, in the following example, we have a class and two subclasses with similar names defined within each of the classes:

class leader {
notify {'Leader-1': }
}
class autobots {
   include leader
}
class autobots::leader {
notify {'Optimus Prime': }
}
class decepticons {
include leader
}
class decepticons::leader {
notify {'Megatron': }
}

We then include the leader, autobots, and decepticons classes in our node, as follows:

include leader
include autobots
include decepticons

When we run Puppet, we see the following output:

t@mylaptop ~ $ puppet apply leaders.pp

Notice: Compiled catalog for mylaptop.example.net in environment production in 0.03 seconds

Notice: Optimus Prime

Notice: /Stage[main]/Autobots::Leader/Notify[Optimus Prime]/message: defined 'message' as 'Optimus Prime'

Notice: Leader-1

Notice: /Stage[main]/Leader/Notify[Leader-1]/message: defined 'message' as 'Leader-1'

Notice: Megatron

Notice: /Stage[main]/Decepticons::Leader/Notify[Megatron]/message: defined 'message' as 'Megatron'

Notice: Finished catalog run in 0.06 seconds

If this is the output that you expected, you can safely move on. If you are a little surprised, then read on. The problem here is the scope. Although we have a top scope class named leader, when we include leader from within the autobots and decepticons classes, the local scope is searched first. In both cases, a local match is found first and used. Instead of the three ‘Leader-1’ notifications, we see only one ‘Leader-1’, one ‘Megatron’, and one ‘Optimus Prime’. If your normal procedure is to have the leader class defined and you forgot to do so, then you can end up being slightly confused. Consider the following modified example:

class leader {
notify {'Leader-1': }
}
class autobots {
include leader
}
include autobots

Now, when we apply this manifest, we see the following output:

t@mylaptop ~ $ puppet apply leaders2.pp

Notice: Compiled catalog for mylaptop.example.net in environment production in 0.02 seconds

Notice: Leader-1

Notice: /Stage[main]/Leader/Notify[Leader-1]/message: defined 'message' as 'Leader-1'

Notice: Finished catalog run in 0.04 seconds

Since the leader class was not available in the scope within the autobot class, the top scope leader class was used. Knowing how Puppet evaluates scope can save you time when your issues turn out to be namespace-related. This example is contrived. The usual situation where people run into this problem is when they have multiple modules organized in the same way. The problem manifests itself when you have many different modules with subclasses in different modules with the same names. For example, two modules named myappa and myappb with config subclasses, myappa::config and myappb::config. This problem occurs when the developer forgets to write the myappc::config subclass and there is a top scope config module available.

Metaparameters

Metaparameters are parameters that are used by Puppet to compile the catalog but are not used when modifying the target system. Some metaparameters, such as tag, are used to specify or mark resources. Other metaparameters, such as before, require, notify, and subscribe, are used to specify the order in which the resources should be applied to a node. When the catalog is compiled, the resources are evaluated based on their dependencies as opposed to how they are defined in the manifests. The order in which the resources are evaluated can be a little confusing for a person who is new to Puppet. A common paradigm when creating files is to create the containing directory before creating the resource. Consider the following code:

class apps {
file {'/apps':
   ensure => 'directory',
   mode => '0755',
}
}
class myapp {
file {'/apps/myapp/config':
   content => 'on = true',
   mode => '0644',
}
file {'/apps/myapp':
   ensure => 'directory',
   mode => '0755',
}
}
include myapp
include apps

When we apply this manifest, even though the order of the resources is not correct in the manifest, the catalog applies correctly, as follows:

[root@trouble ~]# puppet apply order.pp

Notice: Compiled catalog for trouble.example.com in environment production in 0.13 seconds

Notice: /Stage[main]/Apps/File[/apps]/ensure: created

Notice: /Stage[main]/Myapp/File[/apps/myapp]/ensure: created

Notice: /Stage[main]/Myapp/File[/apps/myapp/config]/ensure: defined content as '{md5}1090eb22d3caa1a3efae39cdfbce5155'

Notice: Finished catalog run in 0.05 seconds

Recent versions of Puppet will automatically use the require metaparameter for certain resources. In the case of the preceding code, the ‘/apps/myapp’ file has an implied require of the ‘/apps’ file because directories autorequire their parents. We can safely rely on this autorequire mechanism but, when debugging, it is useful to know how to specify the resource order precisely. To ensure that the /apps directory exists before we try to create the /apps/myapp directory, we can use the require metaparameter to have the myapp directory require the /apps directory, as follows:

classmyapp {
file {'/apps/myapp/config':
   content => 'on = true',
   mode => '0644',
   require => File['/apps/myapp'],
}
file {'/apps/myapp':
   ensure => 'directory',
   mode => '0755',
   require => File['/apps'],
}
}

The preceding require lines specify that each of the file resources requires its parent directory.

Autorequires

Certain resource relationships are ubiquitous. When the relationship is implied, a mechanism was developed to reduce resource ordering errors. This mechanism is called autorequire. A list of autorequire relationships is given in the type reference documentation at https://docs.puppetlabs.com/references/latest/type.html.

When troubleshooting, you should know that the following autorequire relationships exist:

• A cron resource will autorequire the specified user. An exec resource will autorequire both the working directory of the exec as a file resource and the user as which the exec runs.
• A file resource will autorequire its owner and group.
• A mount will autorequire the mounts that it depends on (a mount resource of /apps/myapp will autorequire a mount resource of /apps).
• A user resource will autorequire its primary group.

Autorequire relationships only work when the resources within the relationship are specified within the catalog. If your catalog does not specify the required resources, then your catalog will fail if those resources are not found on the node. For instance, if you have a mount resource of /apps/myapp but the /apps directory or mount does not exist, then the mount resource will fail. If the /apps mount is specified, then the autorequire mechanism will ensure that the /apps mount is mounted before the /apps/myapp mount.

Explicit ordering

When you are trying to determine an error in the evaluation of your class, it can be helpful to use the chaining arrow syntax to force your resources to evaluate in the order that you specified. For instance, if you have an exec resource that is failing, you can create another exec resource that outputs the information used within your failing exec. For example, we have the following exec code:

file {'arrow':
path => '/tmp/arrow',
ensure => 'directory',
}
exec {'arrow_debug_before':
command => 'echo debug_before',
path => '/usr/bin:/bin',
}
exec {'arrow_example':
command => 'echo arrow',
path => '/usr/bin:/bin',
require => File['arrow'],
}
exec {'arrow_debug_after':
command => 'echo debug_after',
path => '/usr/bin:/bin',
}

Now, when you apply this catalog, you will see that the arrow_before and arrow_after resources are not applied in the order that we were expecting:

[root@trouble ~]# puppet agent -t

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Loading facts

Info: Caching catalog for trouble.example.com

Info: Applying configuration version '1431872398'

Notice: /Stage[main]/Main/Node[default]/Exec[arrow_debug_before]/returns: executed successfully

Notice: /Stage[main]/Main/Node[default]/Exec[arrow_debug_after]/returns: executed successfully

Notice: /Stage[main]/Main/Node[default]/File[arrow]/ensure: created

Notice: /Stage[main]/Main/Node[default]/Exec[arrow_example]/returns: executed successfully

Notice: Finished catalog run in 0.23 seconds

To enforce the sequence that we were expecting, you can use the chaining arrow syntax, as follows:

exec {'arrow_debug_before':
command => 'echo debug_before',
path => '/usr/bin:/bin',
}->
exec {'arrow_example':
command => 'echo arrow',
path => '/usr/bin:/bin',
require => File['arrow'],
}->
exec {'arrow_debug_after':
command => 'echo debug_after',
path => '/usr/bin:/bin',
}

Now, when we apply the agent this time, the order is what we expected:

[root@trouble ~]# puppet agent -t

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Loading facts

Info: Caching catalog for trouble.example.com

Info: Applying configuration version '1431872778'

Notice: /Stage[main]/Main/Node[default]/Exec[arrow_debug_before]/returns: executed successfully

Notice: /Stage[main]/Main/Node[default]/Exec[arrow_example]/returns: executed successfully

Notice: /Stage[main]/Main/Node[default]/Exec[arrow_debug_after]/returns: executed successfully

Notice: Finished catalog run in 0.23 seconds

A good way to use this sort of arrangement is to create an exec resource that outputs the environment information before your failing resource is applied. For example, you can create a class that runs a debug script and then use chaining arrows to have it applied before your failing resource. If your resource uses variables, then creating a notify that outputs the values of the variables can also help with debugging.

Defined types

Defined types are great for reducing the complexity and improving the readability of your code. However, they can lead to some interesting problems that may be difficult to diagnose.

In the following code, we create a defined type that creates a host entry:

define myhost ($short,$ip) {
host {"$short":
   ip => $ip,
   host_aliases => [
     "$title.example.com",
     "$title.example.net",
     "$short"
   ],
}
}

In this define, the namevar for the host resource is an argument of the define, the $short variable.

In Puppet, there are two important attributes of any resource—the namevar and the title. The confusion lies in the fact that, sometimes, both of these attributes have the same value. Both values must be unique, but they are used differently. The title is used to uniquely identify the resource to the compiler and need not be related to the actual resource. The namevar uniquely identifies the resource to the agent after the catalog is compiled. The namevar is specific to each resource. For example, the namevar for a package is the package name and the namevar for a file is the full path to the file.

The problem with the preceding defined type is that you can end up with a duplicate resource that is difficult to find. The resource is defined within the defined type. So, when Puppet reports the duplicate definition, it will report it as though it were defined on the same line. Let’s create the following node definition with two myhost resources:

node default {
$short = "trb"
myhost {'trouble': short => 'trb',ip => '192.168.50.1' }
myhost {'tribble': short => "$short",ip => '192.168.50.2' }
}

Even though the two myhost resources have different titles, when we run Puppet, we see a duplicate definition, as follows:

[root@trouble~]# puppet agent -t

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Loading facts

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate declaration: Host[trb] is already declared in file /etc/puppet/environments/production/modules/myhost/manifests/init.pp:5; cannot redeclare at /etc/puppet/environments/production/modules/myhost/manifests/init.pp:5 on node trouble.example.com

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

Tracking down this issue can be difficult if we have several myhost definitions throughout the node definition.

To make this problem a lot easier to solve, we should use the title attribute of the defined type as the title attribute of the resources within the define method. The following rewrite shows this difference:

define myhost ($short,$ip) {
host {"$title":
   ip => $ip,
   host_aliases => [
     "$title.example.com",
     "$title.example.net",
     "$short"
   ],
}
}

Custom facts

When you define custom facts within your modules (in the lib/facter directory), they are automatically transferred to your node via the pluginsync method. The issue here is that the facts are synced to the same directory. So, if you created two facts with the same filename, then it can be difficult to determine which fact will be synced down to your node.

Facter is run at the beginning of a Puppet agent run. The results of Facter are used to compile the catalog. If any of your facts take longer than the configured timeout (config_timeout in the [agent] section of puppet.conf) in Puppet, then the agent run will fail. Instead of increasing this timeout, when designing your custom facts keep them simple enough so that they will take no longer than a few seconds to run.

You can debug Facter from the command line using the -d switch. To load custom facts that are synced from Puppet, add the -p option as well. If you are having trouble with the output of your fact, then you can also have the output formatted as a JSON document by adding the -j option. Combining all of these options, the following is a good starting point for the debugging of your Facter output:

[root@puppet ~]# facter -p -d -j |more

Found no suitable resolves of 1 for ec2_metadata

value for ec2_metadata is still nil

Found no suitable resolves of 1 for gce

value for gce is still nil
...
{
"lsbminordistrelease": "6",
"puppetversion": "3.7.5",
"blockdevice_sda_vendor": "ATA",
"ipaddress_lo": "127.0.0.1",
...

Having Facter output to a JSON file is helpful because the returned values are wrapped in quotes. So, any trailing spaces or control characters will be visible.

The easiest way to debug custom facts is to run them through Ruby directly. To run a custom fact through Ruby, start with the custom fact in the directory and use the irb command to run interactive Ruby, as follows:

[root@puppetfacter]# irb -r facter -r iptables_version.rb
irb(main):001:0> puts Facter.value("iptables_version")
1.4.7
=>nil

This displays the value of the iptables_version fact. From within IRB, you can check the code line-by-line to figure out your problem.

The preceding command was executed on a Linux host. Doing this on a Windows host is not so easy, but it is possible.

Locate the irb executable on your system. For the Puppet Enterprise installation, this should be in C:Program Files (x86)/Puppet Labs/Puppet Enterprise/sys/ruby/bin. Run irb and then alter the $LOAD_PATH variable to add the path to facter.rb (the Facter library), as follows:

irb(main):001:0>$LOAD_PATH.push("C:/Program Files (x86)/Puppet Labs/Puppet Enterprise/facter/lib")

Now require the Facter library, as follows:

irb(main):002:0> require 'facter'

=>true

Finally, run Facter.value with the name of a fact, which is similar to what we did in the previous example:

irb(main):003:0>Facter.value("uptime")

=> "0:08 hours"

Pry

When debugging any Ruby code, using the Pry library will allow you to inspect the Ruby environment that is running at any breakpoint that you define. In the earlier iptables_version example, we could use the Pry library to inspect the calculation of the fact. To do so, modify the fact definition and comment out the setcode section (the breakpoint definition will not work within a setcode block). Then define a breakpoint by adding binding.pry to the fact at the point that you wish to inspect, as follows:

Facter.add(:iptables_version) do
confine :kernel => :linux
#setcode do
version = Facter::Util::Resolution.exec('iptables --version')
if version
version.match(/d+.d+.d+/).to_s
else
nil
end
binding.pry
#end
end

Now run Ruby with the Pry and Facter libraries on the iptables_version fact definition, as follows:

root@mylaptop # ruby -r pry -r facteriptables_version.rb

From: /var/lib/puppet/lib/facter/iptables_version.rb @ line 10 :

     5:     if version
     6:       version.match(/d+.d+.d+/).to_s
     7:     else
     8:       nil
     9:     end
=> 10:   binding.pry
   11:   #end
   12: end

This will cause the evaluation of the iptables_version fact to halt at the binding.pry line. We can then inspect the value of the version variable and execute the regular expression matching ourselves to verify that it is working correctly, as follows:

[1] pry(#<Facter::Util::Resolution>)> version
=> "iptables v1.4.21"
[2] pry(#<Facter::Util::Resolution>)>version.match(/d+.d+.d+/).to_s
=> "1.4.21"
ok

Environment

When developing custom facts, it is useful to make your Ruby fact file executable and run the Ruby script from the command line. When you run custom facts from the command line, the environment variables defined in your current shell can affect how the fact is calculated. This can result in different values being returned for the fact when it is run through the Puppet agent. One of the most common variables that cause this sort of problem is JAVA_HOME. This can also be a problem when testing the exec resources. Environment variables and shell aliases will be available for exec when it is run interactively. When run through the agent, these customizations will not be available, which has the potential to cause inconsistency.

Files

Files are transferred between the master and the node via Puppet’s internal fileserver. When working with files, it is important to remember that all the files that are served via Puppet are read into memory by the Puppet Server. Transferring large files via Puppet is inefficient. You should avoid transferring large and/or binary files. Most of the problems with files are related to path and URL syntax errors. The source parameter contains a URL with the following syntax:

source => "puppet:///path/to/file"

In the preceding syntax, the three slashes specify the beginning of the URL location and the Puppet Server that should be contacted. The following is also valid:

source => "puppet://myserver/path/to/file"

The path from which we can to download a file depends on the context of the manifest. If the manifest is found within the manifest directory or the manifest is the site.pp manifest, then the path to the file is relative to this location starting at the files subdirectory. If the manifest is found within a module, then the path should start with the modules path; then the files will be found within the files directory of the module.

Templates

ERB templates are written in Ruby. The current releases of Puppet also support EPP Puppet templates, which are written in Puppet. The debugging of ERB templates can be done by running the templates through Ruby. To simply check the syntax, use the following code:

$ erb -P -x -T '-' template.erb |ruby -c
Syntax OK

If your template does not pass the preceding test, then you know that your syntax is incorrect. The usual error type that you will see is as follows:

-:8: syntax error, unexpected end-of-input, expecting keyword_end

The problem with the preceding command is that the line number is in the evaluated code that is returned by the erb script, not the original file. When checking for the syntax error, you will have to inspect the intermediate code that is generated by the erb command.

Unfortunately, doing anything more than checking simple syntax is a problem. Although the ERB templates can be evaluated using the ERB library, the <%= block markers that are used in the Puppet ERB templates break the normal evaluation. The simplest way to evaluate Ruby templates is by creating a simple manifest with a file resource that applies the template. As an example, the resolv.conf template is shown in the following code:

# resolv.conf built by Puppet
domain<%= @domain %>
search<% searchdomains.each do |domain| -%>
<%= domain -%><% end -%><%= @domain %>
<% nameservers.each do |server| -%>
nameserver<%= server %>
<% end -%>

This template is then saved into a file named template.erb. We then create a file resource using this template.erb file, as shown in the following code:

$searchdomains = ['trouble.example.com','packt.example.com']
$nameservers = ['8.8.8.8','8.8.4.4']
$domains = 'example.com'

file {'/tmp/test':
content => template('/tmp/template.erb')
}

We then use puppet apply to apply this template and create the /tmp/test file, as follows:

$ puppet apply file.pp

Notice: Compiled catalog for mylaptop.example.net in environment production in 0.20 seconds

Notice: /Stage[main]/Main/File[/tmp/test]/ensure: defined content as '{md5}4d1c547c40a27c06726ecaf784b99e84'

Notice: Finished catalog run in 0.04 seconds

The following are the contents of the /tmp/test file:

# resolv.conf built by Puppet
domainexample.net
search trouble.example.com packt.example.com example.net
nameserver 8.8.8.8
nameserver 8.8.4.4

Debugging templates

Templates can also be used in debugging. You can create a file resource that uses a template that outputs all the defined variables and their values. You can include the following resource in your node definition:

file { "/tmp/puppet-debug.txt":
content =>inline_template("<% vars = scope.to_hash.reject { |k,v| !( k.is_a?(String) &&v.is_a?(String) ) }; vars.sort.each do |k,v| %><%= k %>=<%= v %>n<% end %>"),
}

This uses an inline template, which may make it slightly hard to read. The template loops through the output of the scope function and prints the values if the value is a string. Focusing only on the inner loop, this can be shown as follows:

vars = scope.to_hash.reject
{ |k,v| !( k.is_a?(String) &&
v.is_a?(String) ) };

vars.sort.each do |k,v|
k=vn
end

Summary

In this article, we examined metaparameters and how to deal with resource ordering issues. We built custom facts and defines and discussed the issues that may arise when using them. We then moved on to templates and showed how to use templates as an aid in debugging.

Resources for Article:

---

Further resources on this subject:

• My First Puppet Module[article]
• Puppet Language and Style[article]
• Puppet and OS Security Tools [article]

---