17 min read

In this article by Martin Alfke, the author of the book Puppet Essentials – Third Edition, we will following topics:

  • The Puppet server
  • Setting up the Puppet Agent

(For more resources related to this topic, see here.)

The Puppetserver

Many Puppet-based workflows are centered on the server, which is the central source of configuration data and authority. The server hands instructions to all the computer systems in the infrastructure (where agents are installed). It serves multiple purposes in the distributed system of Puppet components.

The server will perform the following tasks:

  • Storing manifests and compiling catalogs
  • Serving as the SSL certification authority
  • Processing reports from the agent machines
  • Gathering and storing information about the agents

As such, the security of your server machine is paramount. The requirements for hardening are comparable to those of a Kerberos Key Distribution Center.

During its first initialization, the Puppet server generates the CA certificate. This self-signed certificate will be distributed among and trusted by all the components of your infrastructure. This is why its private key must be protected very carefully. New agent machines request individual certificates, which are signed with the CA certificate.

The terminology around the master software might be a little confusing. That’s because both the terms, Puppet Master and Puppet Server, are floating around, and they are closely related too. Let’s consider some technological background in order to give you a better understanding of what is what.

Puppet’s master service mainly comprises a RESTful HTTP API. Agents initiate the HTTPS transactions, with both sides identifying each other using trusted SSL certificates. During the time when Puppet 3 and older versions were current, the HTTPS layer was typically handled by Apache. Puppet’s Ruby core was invoked through the Passenger module. This approach offered good stability and scalability.

Puppet Inc. has improved upon this standard solution with a specialized software called puppetserver. The Ruby-based core of the master remains basically unchanged, although it now runs on JRuby instead of Ruby’s own MRI. The HTTPS layer is run by Jetty, sharing the same Java Virtual Machine with the master.

By cutting out some middlemen, puppetserver is faster and more scalable than a Passenger solution. It is also significantly easier to set up.

Setting up the server machine

Getting the puppetserver software onto a Linux machine is just as simple as the agent package. Packages are available on Red Hat Enterprise Linux and its derivatives,derivatives, Debian and Ubuntu and any other operating system which is supported to run a Puppet server.

Until now the Puppet server must run on a Linux based Operating System and can not run on Windows or any other U*ix.U*ix.

A great way to get Puppet Inc. packages on any platform is the Puppet Collection. Shortly after the release of Puppet 4, Puppet Inc. created this new way of supplying software. This can be considered as a distribution in its own right. Unlike Linux distributions, it does not contain a Kernel, system tools, and libraries. Instead, it comprises various software from the Puppet ecosystem. Software versions that are available from the same Puppet Collection are guaranteed to work well together.

Use the following commands to install puppetserver from the first Puppet Collection (PC1) on a Debian 7 machine. (The Collection for Debian 8 has not yet received a puppetserver package at the time of writing this.)

root@puppetmaster# wget http://apt.puppetlabs.com/puppetlabs-release-pc1-jessie.debhttp://apt.puppetlabs.com/puppetlabs-release-pc1-jessie.deb
root@puppetmaster# dpkg -i puppetlabs-release-pc1-jessie.debpuppetlabs-release-pc1-jessie.deb
root@puppetmaster# apt-get update
root@puppetmaster# apt-get install puppetserver

The puppetserver package comprises only the Jetty server and the Clojure API, but the all-in-one puppet-agent package is pulled in as a dependency.

The package name, puppet-agent, is misleading. This AIO package contains all the parts of Puppet including the master core, a vendored Ruby build, and the several pieces of additional software.

Specifically, you can use the puppet command on the master node. You will soon learn how this is useful. However, when using the packages from Puppet Labs, everything gets installed under /opt/puppetlabs. It is advisable to make sure that your PATH variable always includes the /opt/puppetlabs/bin directory so that the puppet command is found here.

Regardless of this, once the puppetserver package is installed, you can start the master service:

root@puppetmaster# systemctl start puppetserver

Depending on the power of your machine, the startup can take a few minutes. Once initialization completes, the server will operate very smoothly though. As soon as the master port 8140 is open, your Puppet master is ready to serve requests.

If the service fails to start, there might be an issue with certificate generation. (We observed such issues with some versions of the software.) Check the log file at /var/log/puppetlabs/puppetserver/puppetserver-daemon.log. If it indicates that there are problems while looking up its certificate file, you can work around the problem by temporarily running a standalone master as follows:
puppet master –-no-daemonize.

After initialization, you can stop this process. The certificate is available now, and puppetserver should now be able to start as well.

Another reason for start failures is insufficient amount of memory. The Puppet server process needs 2 GB of memory.

Creating the master manifest

The master compiles manifests for many machines, but the agent does not get to choose which source file is to be used—this is completely at the master’s discretion. The starting point for any compilation by the master is always the site manifest, which can be found in /opt/puppetlabs/code/environments/production/manifests/.

Each connecting agent will use all the manifests found here. Of course, you don’t want to manage only one identical set of resources on all your machines. To define a piece of manifest exclusively for a specific agent, put it in a node block. This block’s contents will only be considered when the calling agent has a matching common name in its SSL certificate. You can dedicate a piece of the manifest to a machine
with the name of agent, for example:

node 'agent' {
  $packages = [ 'apache2', 
    'libapache2-mod-php5', 
    'libapache2-mod-passenger', ] 
  package { $packages:
    ensure => 'installed', 
    before => Service['apache2'], 
  } 
  service { 'apache2': 
    ensure => 'running', 
    enable => true, 
  }
}

Before you set up and connect your first agent to the master, step back and think about how the master should be addressed. By default, agents will try to resolve the unqualified puppet hostname in order to get the master’s address. If you have a default domain that is being searched by your machines, you can use this as a default and add a record for puppet as a subdomain (such as puppet.example.net). Otherwise, pick a domain name that seems fitting to you, such as master.example.net or adm01.example.net. What’s important is the following:

  • All your agent machines can resolve the name to an address
  • The master process is listening for connections on that address
  • The master uses a certificate with the chosen name as CN or DNS Alt Names

The mode of resolution depends on your circumstances—the hosts file on each machine is one ubiquitous possibility. The Puppet server listens on all the available addresses by default.

This leaves the task of creating a suitable certificate, which is simple. Configure the master to use the appropriate certificate name and restart the service. If the certificate does not exist yet, Puppet will take the necessary steps to create it. Put the following setting into your /etc/puppetlabs/puppet/puppet.conf file on the master machine:

[main]
[main]
certname=puppetmaster.example.net

In Puppet versions before 4.0, the default location for the configuration file is /etc/puppet/puppet.conf.

Upon its next start, the master will use the appropriate certificate for all SSL connections. The automatic proliferation of SSL data is not dangerous even in an existing setup, except for the certification authority. If the master were to generate a new CA certificate at any point in time, it would break the trust of all existing agents.

Make sure that the CA data is neither lost nor compromised. All previously signed certificates become obsolete whenever Puppet needs to create a new certification authority. The default storage location is /etc/puppetlabs/puppet/ssl/ca for Puppet 4.0 and higher, and /var/lib/puppet/ssl/ca for older versions.

Inspecting the configuration settings

All the customization of the master’s parameters can be made in the puppet.conf file. The operating system packages ship with some settings that are deemed sensible by the respective maintainers. Apart from these explicit settings, Puppet relies on defaults that are either built-in or derived from the :

root@puppetmaster # puppet master --configprint manifest
/etc/puppetlabs/code/environments/production/manifests

Most users will want to rely on these defaults for as many settings as possible. This is possible without any drawbacks because Puppet makes all settings fully transparent using the –configprint parameter. For example, you can find out where the master manifest files are located.

To get an overview of all available settings and their values, use the following command:

root@puppetmaster# puppet master --configprint all | less

While this command is especially useful on the master side, the same introspection is available for puppet apply and puppet agent. 

Setting specific configuration entries is possible with the puppet config command:

root@puppetmaster # puppet config set –-section main certname puppetmaster.example.net

Setting up the Puppet agent

As was explained earlier, the master mainly serves instructions to agents in the form of catalogs that are compiled from the manifest. You have also prepared a node block for your first agent in the master manifest.

The plain Puppet package that allows you to apply a local manifest contains all the required parts in order to operate a proper agent.

If you are using Puppet Labs packages, you need not install the puppetserver package. Just get puppet-agent instead.

After a successful package installation, one needs to specify where Puppet agent can find the puppetserver:

root@puppetmaster # puppet config set –-section agent server puppetmaster.example.net

Afterwards the following invocation is sufficient for an initial test:

root@agent# puppet agent --test
Info: Creating a new SSL key for agent
Error: Could not request certificate: getaddrinfo: Name or service not known
Exiting; failed to retrieve certificate and waitforcert is disabled

Puppet first created a new SSL certificate key for itself. For its own name, it picked agent, which is the machine’s hostname. That’s fine for now. An error occurred because the puppet name cannot be currently resolved to anything. Add this to /etc/hosts so that Puppet can contact the master:

root@agent# puppet agent --test
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for agent
Info: Certificate Request fingerprint (SHA256): 52:65:AE:24:5E:2A:C6:17:E2:5D:0A:C9: 86:E3:52:44:A2:EC:55:AE:3D:40:A9:F6:E1:28:31:50:FC:8E:80:69
Error: Could not request certificate: Error 500 on SERVER: Internal Server Error: java.io.FileNotFoundException: /etc/puppetlabs/puppet/ssl/ca/requests/agent.pem (Permission denied)
Exiting; failed to retrieve certificate and waitforcert is disabled

Note how Puppet conveniently downloaded and cached the CA certificate. The agent will establish trust based on this certificate from now on.

Puppet created a certificate request and sent it to the master. It then immediately tried to download the signed certificate. This is expected to fail—the master won’t just sign a certificate for any request it receives. This behavior is important for proper security.

There is a configuration setting that enables such automatic signing, but users are generally discouraged from using this setting because it allows the creation of arbitrary numbers of signed (and therefore, trusted) certificates to any user who has network access to the master.

To authorize the agent, look for the CSR on the master using the puppet cert command:

root@puppetmaster# puppet cert --list
"agent" (SHA256) 52:65:AE:24:5E:2A:C6:17:E2:5D:0A:C9:86:E3:52:44:A2:EC:55:AE: 3D:40:A9:F6:E1:28:31:50:FC:8E:80:69

This looks alright, so now you can sign a new certificate for the agent:

root@puppetmaster# puppet cert --sign agent
Notice: Signed certificate request for agent
Notice: Removing file Puppet::SSL::CertificateRequest agent at '/etc/puppetlabs/ puppet/ssl/ca/requests/agent.pem'

When choosing the action for puppet cert, the dashes in front of the option name can be omitted—you can just use puppet cert list and puppet cert sign.

Now the agent can receive its certificate for its catalog run as follows:

root@agent# puppet agent --test
Info: Caching certificate for agent
Info: Caching certificate_revocation_list for ca
Info: Caching certificate for agent
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Caching catalog for agent
Info: Applying configuration version '1437065761'
Notice: Applied catalog in 0.11 seconds

The agent is now fully operational. It received a catalog and applied all resources within. Before you read on to learn how the agent usually operates, there is a note that is especially important for the users of Puppet 3.

Since this is the common name in the master’s certificate, the preceding command will not even work with a Puppet 3.x master. It works with puppetserver and Puppet 4 because the default puppet name is now included in the certificate’s Subject Alternative Names by default.

It is tidier to not rely on this alias name, though. After all, in production, you will probably want to make sure that the master has a fully qualified name that can be resolved, at least inside your network. You should therefore, add the following to the main section of puppet.conf on each agent machine:

[agent]
[agent]
server=master.example.net

In the absence of DNS to resolve this name, your agent will need an appropriate entry in its hosts file or a similar alternative way of address resolution.

These steps are necessary in a Puppet 3.x setup. If you have been following along with a Puppet 4 agent, you might notice that after this change, it generates a new Certificate Signing Request:

root@agent# puppet agent –test
Info: Creating a new SSL key for agent.example.net
Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for agent.example.net
Info: Certificate Request fingerprint (SHA256): 85:AC:3E:D7:6E:16:62:BD:28:15:B6:18: 12:8E:5D:1C:4E:DE:DF:C3:4E:8F:3E:20:78:1B:79:47:AE:36:98:FD
Exiting; no certificate found and waitforcert is disabled

If this happens, you will have to use puppet cert sign on the master again. 

The agent will then retrieve a new certificate.

The agent’s life cycle

In a Puppet-centric workflow, you typically want all changes to the configuration of servers (perhaps even workstations) to originate on the Puppet master and propagate to the agents automatically. Each new machine gets integrated into the Puppet infrastructure with the master at its center and gets removed during the decommissioning, as shown in the following diagram:

Insert image 6566_02_01.png

The very first step—generating a key and a certificate signing request—is always performed implicitly and automatically at the start of an agent run if no local SSL data exists yet. Puppet creates the required data if no appropriate files are found. There will be a short description on how to trigger this behavior manually later in this section.

The next step is usually the signing of the agent’s certificate, which is performed on the master. It is a good practice to monitor the pending requests by listing them on the console:

root@puppetmaster# puppet cert list
oot@puppetmaster# puppet cert sign '<agent fqdn>'

From this point on, the agent will periodically check with the master to load updated catalogs. The default interval for this is 30 minutes. The agent will perform a run of a catalog each time and check the sync state of all the contained resources. The run is performed for unchanged catalogs as well, because the sync states can change between runs.

Before you manage to sign the certificate, the agent process will query the master in short intervals for a while. This can avoid a 30 minute delay if the certificate is not ready, right when the agent starts up.

Launching this background process can be done manually through a simple command:

root@agent# puppet agent

However, it is preferable to do this through the puppet system service.

When an agent machine is taken out of active service, its certificate should be invalidated. As is customary with SSL, this is done through revocation. The master adds the serial number of the certificate to its certificate revocation list. This list, too, is shared with each agent machine. Revocation is initiated on the master through the puppet cert command:

root@puppetmaster# puppet cert revoke agent

The updated CRL is not honored until the master service is restarted. If security is a concern, this step must not be postponed.

The agent can then no longer use its old certificate:

root@agent# puppet agent --test
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect SYSCALL returned=5 errno=0 state=unknown state
[...]
Error: Could not retrieve catalog from remote server: SSL_connect SYSCALL returned=5 errno=0 state=unknown state
[...]

Renewing an agent’s certificate

Sometimes, it is necessary during an agent machine’s life cycle to regenerate its certificate and related data—the reasons can include data loss, human error, or certificate expiration, among others.

Performing the regeneration is quite simple: all relevant files are kept at /etc/puppetlabs/puppet/ssl (for Puppet 3.x, this is /var/lib/puppet/ssl) on the agent machine.

Once these files are removed (or rather, the whole ssl/ directory tree), Puppet will renew everything on the next agent run. Of course, a new certificate must be signed. This requires some preparation—just initiating the request from the agent will fail:

root@agent# puppet agent –test
Info: Creating a new SSL key for agent
Info: Caching certificate for ca
Info: Caching certificate for agent.example.net
Error: Could not request certificate: The certificate retrievedfrom the master does not match the agent's private key.
Certificate fingerprint: 6A:9F:12:C8:75:C0:B6:10:45:ED:C3:97:24:CC:98:F2:B6:1A:B5: 4C:E3:98:96:4F:DA:CD:5B:59:E0:7F:F5:E6

The master still has the old certificate cached. This is a simple protection against the impersonation of your agents by unauthorized entities.

To fix this, remove the certificate from both the master and the agent and then start a Puppet run, which will automatically regenerate a certificate.

On the master:

puppet cert clean agent.example.net

On the agent:

On most platforms:

find /etc/puppetlabs/puppet/ssl -name agent.example.net.pem 
–delete

On Windows:

del "/etc/puppetlabs/puppet/ssl/agent.example.net.pem" /f
puppet agent –t
Exiting; failed to retrieve certificate and waitforcert is disabled

Once you perform the cleanup operation on the master, as advised in the preceding output, and remove the indicated file from the agent machine, the agent will be able to successfully place its new CSR:

root@puppetmaster# puppet cert clean agent
Notice: Revoked certificate with serial 18
Notice: Removing file Puppet::SSL::Certificate agent at '/etc/puppetlabs/ puppet/ssl/ca/signed/agent.pem'

Removing file Puppet::SSL::Certificate agent at ‘/etc/puppetlabs/ puppet/ssl/certs/agent.pem’.

The rest of the process is identical to the original certificate creation. The agent uploads its CSR to the master, where the certificate is created through the puppet cert sign command.

Running the agent from cron

There is an alternative way to operate the agent. We covered starting one long-running puppet agent process that does its work in set intervals and then goes back to sleep. However, it is also possible to have cron launch a discrete agent process in the same interval. This agent will contact the master once, run the received catalog, and then terminate. This has several advantages as follows:

  • The agent operating system saves some resources
  • The interval is precise and not subject to skew (when running the background agent, deviations result from the time that elapses during the catalog run), and distributed interval skew can lead to thundering herd effects
  • Any agent crash or an inadvertent termination is not fatal

Setting Puppet to run the agent from cron is also very easy to do—with Puppet! You can use a manifest such as the following:

service { 'puppet': enable => false }
cron { 'puppet-agent-run':
  user    => 'root',
  command =>
    'puppet agent --no-daemonize --onetime --logdest=syslog',
  minute  => fqdn_rand(60),
  hour    => absent,
}

The fqdn_rand function computes a distinct minute for each of your agents. Setting the hour property to absent means that the job should run every hour.

Summary

In this article we have learned about the Puppet server and also learned how to setting up the Puppet agent. 

Resources for Article:

Further resources on this subject:

Packt

LEAVE A REPLY

Please enter your comment!
Please enter your name here