News

Welcome to End Point’s blog

Ongoing observations by End Point people

Setting a server role in Salt (comparing Puppet and Salt)

There are many ways to solve a given problem, and this is no truer than with configuration management. Salt (http://www.saltstack.com) is a fairly new tool in the configuration management arena joining the ranks of Puppet, Chef, and others. It has quickly gained and continues to grow in popularity, boasting its scalable architecture and speed. And so with multiple tools and multiple ways to use each tool, it can get a little tricky to know how best to solve your problem.

Recently I've been working with a client to convert their configuration management from Puppet to Salt. This involved reviewing their Puppet configs and designs and more-or-less mapping them to the equivalent for Salt. Most features do convert pretty easily. However, we did run into something that didn't at first- assigning a role to a server.

We wanted to preserve the "feeling" of the configs where possible. In Puppet they had developed and used a convention for using some custom variables in their configs to assign an "environment" and a "role" for each server. These variables were assigned in the node and role manifests. But in Salt we struggled to find a similar way to do that, but here is what we learned.

In Puppet, once a server's "role" and "environment" variables were set, then they could be used in other manifest files to select the proper source for a given config file like so:

    file    {
        "/etc/rsyslog.conf":
            source  =>
            [
                "puppet:///rsyslog/rsyslog.conf.$hostname", 
                "puppet:///rsyslog/rsyslog.conf.$system_environment-$system_role", 
                "puppet:///rsyslog/rsyslog.conf.$system_role", 
                "puppet:///rsyslog/rsyslog.conf.$system_environment", 
                "puppet:///rsyslog/rsyslog.conf"
            ],
            ensure  => present,
            owner   => "root",
            group   => "root",
            mode    => "644"
    }

Puppet will search the list of source files in order and use the first one that exists. For example, if $hostname = 'myniftyhostname' and $system_environment = 'qa' and $system_role = 'sessiondb', then it will use rsyslog.conf.myniftyhostname if it exists on the Puppet master, or if not then use rsyslog.conf.qa-sessiondb if it exists, or if not then rsyslog.conf.sessiondb if it exists, or if not then rsyslog.conf.qa if it exists, or if not then rsyslog.conf.

In Salt, environment is built into the top.sls file, where you match your servers to their respective state file(s), and can be used within state files as {{ env }}. Salt also allows for multiple sources for a managed file to be listed in order and it will use the first one that exists in the same way as Puppet. We were nearly there; however, setting the server role variable was not as straight forward in Salt.

We first looked at using Jinja variables (which is the default templating system for Salt), but soon found that setting a Jinja variable in one state file does not carry over to another state file. Jinja variables remain only in the scope of the file they were created in, at least in Salt.

The next thing we looked at was using Pillar, which is a way to set custom variables from the Salt master to given hosts (or minions). Pillar uses a structure very similar to Salt's top.sls structure- matching a host with its state files. But since the hostnames for this client vary considerably and don't lend themselves to pattern matching easily, this would be cumbersome to manage both the state top.sls file and the Pillar top.sls file and keep them in sync. It would require basically duplicating the list of hosts in two files, which could get out of sync over time.

We asked the salt community on #salt on Freenode.net how they might solve this problem, and the recommended answer was to set a custom grain. Grains are a set of properties for a given host, collected from the host itself- such as, hostname, cpu architecture, cpu model, kernel version, total ram, etc. There are multiple ways to set custom grains, but after some digging we found how to set them from within a state file. This meant that we could do something like this in a "role" state file:

# sessiondb role
# {{ salt['grains.setval']('server_role','sessiondb') }}

include:
  - common
  - postgres

And then within the common/init.sls and postgres/init.sls state files we could use that server_role custom grain in selecting the right source file, like this:

/etc/rsyslog.conf:
  file.managed:
    - source:
      - salt://rsyslog/files/rsyslog.conf.{{ grains['host'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ env }}-{{ grains['server_role'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ grains['server_role'] }}
      - salt://rsyslog/files/rsyslog.conf.{{ env }}
      - salt://rsyslog/files/rsyslog.conf
    - mode: 644
    - user: root
    - group: root

This got us to our desired config structure. But like I said earlier, there are probably many ways to handle this type of problem. This may not even be the best way to handle server roles and environments in Salt, if we were more willing to change the "feeling" of the configs. But given the requirements and feedback form our client, this worked fine.

3 comments:

Igor said...

Grains and pillar are basically the same (grains will be shared, pillar are only known by the minion and the master).

So if you create your custom grain or your custom pillar, that's a question of your taste :)

What I don't understand:
You said your client already used the hostname. Have you seen, that there is already a existing grain for host/hostname (which should also be the minion id per default)?

So there's nothing to change for you, just use the right variable.

But thanks for pointing out that you can use multiple source files, that's a nice idea.


PS: You may want to read https://groups.google.com/forum/#!msg/salt-users/CgAjdk1l_7M/tygfYX7yqlEJ

One problem is that the minion id may contain dots, but you can create your custom grain which will replace dots... very easy.

Spencer Christensen said...

Thanks Igor for that Google Group thread- no I hadn't seen that before. There are two interesting approaches in there as well.

As for grains and pillars- pillar data is set on the master, grains come from the minion by default but custom ones can be added from the master. You can compare the two for a given minion 'centos-server-5-b.local' like this:

On the salt master run:
* salt 'centos-server-5-b*' grains.items
* salt 'centos-server-5-b*' pillar.items

Or on the minion run these commands:
* salt-call grains.items
* salt-call pillar.items

I'm not sure I understand your question about hostname. We use hostname (well, id really) in top.sls like so:

base:
'daffy.example.com':
- roles.memcached
'bunny.example.com':
- roles.webapp
'elmer.example.com':
- roles.sessiondb

Then in the roles/ dir, we have a state file for each role. Like this one:

# Memcached
# {{ salt['grains.setval']('server_role','memcached') }}
include:
- packages.common
- packages.memcached

Then in the packages/ dir we have common/init.sls, which contains things for all roles. In order for it to be used by all roles, it uses the grain 'server_role' to pull the right source for the config file where needed. That way memcached servers get the right yum repos installed and the right rsyslog config, and iptables just for that role.

Does this make more sense?

Like I said, there's more than one way to do it. Thanks for the feedback.

David Douard said...

Hi.
what I prefer is to describe my system in the pillar area and have the states be able to apply the setup according the pillars. As such, there are very fiew states that depends on the host name. This is the idea behind salt formulas. The states are written in a way they aim at applying what is described in a declarative manner in the pillar system. Pillars describe the how the system should be, states describe how to apply this declarative description on the machines.

Unless I'm wrong, your approach is to mix both the declarative part and the formulas in the states system. I would not recomend it.

David