Salt-Cloud – Executing an auto high-state

Saltstack now includes the wonderful salt-cloud tool by default. This tools allows you to easily provision virtual machines with a multitude of Cloud providers. One issue with it though, is that you have to run a second command to execute a high-state. While this might seem like a trivial task, it adds complication when you would like to integrate the provisioning into some type of automation. The goal of this post is to have a fully provisioned server, including high-state, with a single command either from the CLI or API.

Detailed below are several methods of executing an automatic high-state and their pros/cons.

1. Startup states: Run a high-state when the minion starts.

In the minion config file, you can add an entry for startup_states. This will execute any states when the minion starts including a high-state. You can either manage the minion config file with Salt or in the case of salt-cloud, include it in the map file.

fedora_small:
  - web1:
    minion:
      startup_states: highstate

PRO: You will run a high-state each and every time your minion starts, which means they will run if the server is rebooted.

CON: The obvious downside here is the chicken and egg problem. You have to have the entry in either the map file or the minion config file prior to provisioning the server. Also, you cannot use custom grains or access mine functions as they have not be set yet or synced.

2. Start action: Salt-cloud will run a highstate during provisioning .

In the cloud config file, /etc/salt/cloud, you can define the start_action option. This will the defined action during the salt-cloud provisioning process.

start_action: state.highstate

PRO: Seamlessly integrates into salt-cloud and dead simple to configure. Provides exactly what we want. When the salt command returns, the server is 100% complete.

CON: This is currently considered to be experimental functionality, and may not work well with all providers. Especially with providers that do not give you root access and require using sudo initially. I am looking at you Amazon! Same grains and mine problem as above.

3. Salt Reactor: Use the reactor to run the high-state

Salt outlines a very nice example of using the reactor to run a high-state. When your server is created, salt-cloud will fire off an event to the Salt reactor. You can set up ‘listeners’ for specific events and have various states executed based on the matches.

# /etc/salt/master
reactor:
  - 'salt/cloud/*/created':
    - '/srv/reactor/startup_highstate.sls'

# /srv/reactor/startup_highstate.sls
reactor_highstate:
  local.state.highstate:
    - tgt: {{ data['name'] }}

PRO: Its tried and true and will work will all providers. You can run multiple states on a server, but currently the order is not guaranteed.

CON: More complicated to set up and has the same mine/grains issue.

4. Salt Reactor + Orchestration: Use the reactor to run an orchestration.

One things Salt doesn’t explain well in the documentation is you can do so much more with the reactor system. Why run a single state when you can run a full orchestration!

# /etc/salt/master
reactor:
  - 'salt/cloud/*/created':
    - '/srv/reactor/startup_orchestration.sls'

# /srv/reactor/startup_orchestration.sls
startup_orchestrate:
  runner.state.orchestrate:
    - mod: orchestration.startup

# /srv/salt/orchestration/startup.sls
saltutil.sync_all:
  salt.function:
    - tgt: '*'
    - reload_modules: True

saltutil.refresh_pillar:
  salt.function:
    - tgt: '*'

mine.update:
  salt.function:
    - tgt: '*'

highstate_run:
  salt.state:
    - tgt: '*'
    - highstate: True

PRO: You can run nearly anything you want in an orchestration! Lets breakdown the example above and the reasoning behind it.

  1. sync_all: This will sync all of our custom modules, states, and grains. Everything except pillar, across all of our servers.
  2. refresh_pillar: This will refresh our pillar data across all of our servers. Not sure why sync_all doesn’t do this…
  3. mine.update: Update the mine functions across all of our servers. This will allow us to access any mine functions and have current data. This is extremely important if you need a server to find another server. For example, you have a web server that needs to know information about a database server.
  4. highstate: And finally, we run our highstate. Complete with fresh pillar data, mine data and evertyhing that goes with it.

CON: Its a bit to set up and depending on your version of Saltstack, the documentation is a bit murky.

6 thoughts on “Salt-Cloud – Executing an auto high-state

  1. When I try option #4, I’m getting an error (watching ‘salt-run state.event pretty=True’):

    Exception occurred in runner state.orchestrate: Traceback (most recent call last):
    File \”/usr/lib/python2.7/dist-packages/salt/client/mixins.py\”, line 293, in low
    expected_extra_kws=CLIENT_INTERNAL_KEYWORDS
    File \”/usr/lib/python2.7/dist-packages/salt/utils/__init__.py\”, line 816, in format_call
    used_args_count\nSaltInvocationError: orchestrate takes at least 1 argument (0 given)

    Any thoughts?

    • Honestly, that one took quite a bit to get working. Unfortunately, the runner and orchestration functions have changed quite a bit between Salt versions. When I put this together, the existing documentation was not correct and required me talking to people in the SaltStack channel to get it working.

  2. Howdy. Good stuff here, getting to understand events and reactors. Curious why you use data[‘name’] in example 3 vs. data[‘id’] for referencing the minion?

    Like Fred, seeing the same error on example 4, but I’m also in 2015.3.0 (trusty) build. Hoping the PPA’s get sorted for 2015.8.0 to update my 14.04 LTS box.

    Thanks for documenting this!

  3. Found the error in example 4. runner.state.orchestrate: requires “mods:” not “mod:”, at least in 2015.3.0. Changed that and the orchestrate sls file is getting fired off by the reactor.

  4. So no pointing fingers at Amazon.. I can vouch that the centos AMI in the marketplace has root login in the sshd_config on, but cloud-init has root disable..

    Little trick is to use: userdata_file: /etc/salt/my-userdata-file inside your provisioner file. In the userdata file add the following lines. We use this trick with our packer builds

    #cloud-config
    # vim:syntax=yaml
    debug: True
    disable_root: false

    Now you can log in as root.

    Unfortunately for me the start_action: state.highstate is still now working 🙁

Leave a Reply

Your email address will not be published. Required fields are marked *