What is SaltStack?
SaltStack is an extremely fast, scalable, and powerful remote execution engine and configuration management tool created to control distributed infrastructure, code, and data efficiently. At the heart of SaltStack, or “Salt”, is its remote execution engine, which is a bi-directional, secure communication system administered through the use of a Salt Master daemon. This daemon is used to control Salt Minion daemons, which receive commands from the remote Salt Master.
A major component of Salt’s approach to configuration management is Salt Cloud, which was made to manage Salt Minions in cloud environments. The main purpose of Salt Cloud is to spin up instances on cloud providers, install a Salt Minion on the new instance using Salt’s Bootstrap Script, and configure the new minion so it can immediately get to work. Salt Cloud makes it easy to get an infrastructure up and running quickly and supports an array of cloud providers such as OpenStack, Digital Ocean, Joyent, Linode, Rackspace, Amazon EC2, and Google Compute Engine to name a few. Here is a full list of cloud providers supported by SaltStack and the automation features supported for each.
What is cloud auto scaling?
One of the most formidable benefits of cloud application hosting and data storage is the cloud infrastructure’s capacity to scale as demand fluctuates. Many cloud providers offer auto scaling features that automatically increase or decrease the number of instances that are up and running in a user’s cloud at any given time. These components generate new instances as needed to ensure optimal performance as activity escalates, while during idle periods, instances are destroyed to reduce costs.
To harness the power of cloud auto-scaling technologies, SaltStack provides two reactor formulas that integrate Salt’s configuration management and remote execution capabilities for either Amazon EC2 Auto Scaling or Rackspace Auto Scale.
The Salt Cloud Reactor
Salt Formulas can be very helpful in the rapid build out of management frameworks for cloud infrastructures. Formulas are pre-written Salt States that can be used to configure services, install packages, or any other common configuration management tasks. The Salt Cloud Reactor is a formula that allows Salt to interact with supported Salt Cloud providers who provide cloud auto scaling features. (Note: at the time this article was written, the only supported Salt Cloud providers with cloud auto scaling capabilities were Rackspace Auto Scale and Amazon EC2 Auto Scaling. The Salt Cloud Reactor can also be used directly with EC2 Auto Scaling, but it is recommended that the EC2 Autoscale Reactor be used instead, as discussed in the following section.)
The Salt Cloud Reactor allows SaltStack to know when instances are spawned or destroyed by the cloud provider. When a new instance comes online, a Salt Minion is automatically installed and the minion’s key is accepted by the Salt Master. If the configuration for the minion contains the appropriate startup state, it will configure itself and start working on its tasks. Accordingly, when an instance is deleted by the cloud provider, the minion’s key is removed from the Salt Master.
In order to use the Salt Cloud Reactor, the Salt Master must be configured appropriately. In addition to applying all necessary settings on the Salt Master, a Salt Cloud query must be executed on a regular basis. The query polls data from the cloud provider to collect changes in the auto scaling sequence, as cloud providers using the Salt Cloud Reactor do not directly trigger notifications to Salt upon instance creation and deletion. The cloud query must be issued via a scheduling system such as cron or the Salt Scheduler. Once the Salt Master has been configured and query scheduling has been implemented, the reactor will manage itself and allow the Salt Master to interact with any Salt Minions created or destroyed by the auto scaling system.
The EC2 Autoscale Reactor
Salt’s EC2 Autoscale Reactor enables Salt to collaborate with Amazon EC2 Auto Scaling. Similarly to the Salt Cloud Reactor, the EC2 Autoscale Reactor will bootstrap a Salt Minion on any newly created instances and the Salt Master will automatically accept the new minion’s key. Additionally, when an EC2 instance is destroyed, the Salt Minion’s key will be automatically removed from the Salt Master.
However, the EC2 Auto Scale Reactor formula differs from the Salt Cloud Reactor formula in one major way. Amazon EC2 provides notifications directly to the reactor when the EC2 cloud is scaled up or down, making it easy for Salt to immediately bootstrap new instances with a Salt Minion, or to delete old Salt Minion keys from the master. This behavior, therefore, does not require any kind of scheduled query to poll EC2 for changes in scale like the Salt Cloud Reactor demands. Changes to the EC2 cloud can be acted upon by the Salt Master immediately, whereas changes in clouds using the Salt Cloud Reactor may experience a delay in the instance being created and the Salt Master bootstrapping the instance with a new minion.
Configuring the EC2 Autoscale Reactor
Both of the cloud auto scaling reactors were only recently added to the SaltStack arsenal, and as such, the Salt develop branch is required to set up auto any scaling capabilities. To get started, clone the Salt repository from GitHub onto the machine serving as the Salt Master:
git clone https://github.com/saltstack/saltDepending on the operating system you are using, there are a few dependencies that also need to be installed to run SaltStack from the develop branch. Check out the Installing Salt for Development documentation for OS-specific instructions.
Once Salt has been installed for development, the Salt Master needs to be configured. First, create the default salt directory in /etc :
mkdir /etc/saltThe default Salt Master configuration file resides in salt/conf/master. Copy this file into the new salt directory:
cp path/to/salt/conf/master /etc/salt/master The Salt Master configuration file is completely commented out, as the default configuration for the master will work on most systems. However, some additional settings must be configured to enable the EC2 Autoscale Reactor to work with the Salt Master. Under the external_auth section of the master configuration file, replace the commented out lines with the following:
external_auth:
pam:
myuser:
- .*
- ‘@runner’
- ‘@wheel’
rest_cherrypy:
port: 8080
host: 0.0.0.0
webhook_url: /hook
webhook_disable_auth: True
reactor:
- ‘salt/netapi/hook/ec2/autoscale’:
- ‘/srv/reactor/ec2-autoscale.sls’
ec2.autoscale:
provider: my-ec2-config
ssh_username: ec2-userThese settings allow the Salt API web hook system to interact with EC2. When a web request is received from EC2, the Salt API will execute an event for the reactor system to respond to. The final ec2.autoscale setting points the reactor to the corresponding Salt Cloud provider configuration file.
If authenticity problems with the reactor’s web hook occur, an email notification from Amazon will be sent to the user. To configure the Salt Master to connect to a mail server, see the example SMTP settings in the EC2 Autoscale Reactor documentation.
Next, the Salt Cloud provider configuration file must be created. First, create the cloud provider configuration directory:
mkdir /etc/salt/cloud.providers.d In /etc/salt/cloud.providers.d, create a file named ec2.conf, and set the following configurations according to your Amazon EC2 account:
my-ec2-config:
id: <my aws id>
key: <my aws key>
keyname: <my aws key name>
securitygroup: <my aws security group>
private_key: </path/to/my/private_key.pem>
location: us-east-1
provider: ec2
minion:
master: saltmaster.example.comThe last line, master: saltmaster.example.com, represents the location of the Salt Master so the Salt Minions know where to connect once it’s up and running.
To set up the actual reactor, create a new reactor directory, download the ec2-autoscale-reactor formula, and copy the reactor formula into the new directory, like so:
mkdir /srv/reactor
cp path/to/downloaded/package/ec2-autoscale.sls /srv/reactor/ec2-autoscale.slsThe last major configuration step is to configure all of the appropriate settings on your EC2 account. First, log in to your AWS account and set up SNS HTTP(S) notifications by selecting SNS (Push Notification Service) from the AWS Console. Click Create New Topic, enter a topic name and a display name, and click the Create Topic button. Then, inside the Topic Details area, click Create Subscription. Choose HTTP or HTTPS as needed and enter the web hook for the Salt API. Assuming your Salt Master is set up at https://saltmaster.example.com, the final web hook endpoint will be: https://saltmaster.example.com/hook/ec2/autoscale. Finally, click Subscribe.
Next, set up the launch configurations by choosing EC2 (Virtual Servers in the Cloud) from the AWS Console. Then, select Launch Configurations on the left-hand side. Click Create Launch Configuration and follow the prompts to define the appropriate settings for your cloud. Finally, on the review screen, click Create Launch Configuration to save your settings.
Once the launch configuration is set up, click Auto Scaling Groups from the left-hand navigation menu to create auto scaling variables such as the minimum and maximum number of instances your cloud should contain. Click Create Auto Scaling Group, choose Create an Auto Scaling group from an existing launch configuration, select the appropriate configuration, and then click Next Step. From there, follow the prompts until you reach the Configure Notifications screen. Click Add Notification and choose the notification setting that was configured during the SNS configuration step. Finally, complete the rest of the prompts.
Congratulations! At this point, you should have successfully configured SaltStack to work with EC2 Auto Scaling!
Salt Scheduler
As mentioned in the Salt Cloud Reactor section, some type of scheduling system must be implemented when using the Salt Cloud Reactor formula. SaltStack provides its own scheduler, which can be used by adding the following state to the Salt Master’s configuration file:
schedule:
job1:
function: cloud.full_query
seconds: 300Here, the seconds setting ensures that the Salt Master will perform a salt-cloud –full-query command every 5 minutes. A minimum value of 300 seconds or greater is recommended, however, the value can be changed as necessary.
Salting instances from the web interface
Another exciting quality of Salt’s auto-scale reactor formulas is once a reactor is configured, the respective cloud provider web interface can be used to spin up new instances that are automatically “Salted”. Since the reactor integrates with the web interface to automatically install a Salt Minion on any new instances, it will perform the same operations when instances are created manually via the web interface. The same functionality is true for manually deleting instances: if an instance is manually destroyed on the web interface, the corresponding minion’s key will be removed from the Salt Master.
More resources
For troubleshooting, more configuration options, or SaltStack specifics, SaltStack has many helpful resources such as SaltStack, Salt Cloud, Salt Cloud Reactor, and EC2 Autoscale Reactor documentation. SaltStack also has a thriving, active, and friendly open source community.
About the Author
Nicole Thomas is a QA Engineer at SaltStack, Inc. Before coming to SaltStack, she wore many hats from web and Android developer to contributing editor to working in Environmental Education. Nicole recently graduated Summa Cum Laude from Westminster College with a degree in Computer Science. Nicole also has a degree in Environmental Studies from the University of Utah.