Category Archives: Foreman

How to use the new virsh provider in Foreman 1.4

This morning I decided to play with a new Foreman 1.4 feature : TFTP, DHCP and DNS provider for my local workstation : virsh.
Virsh provider allow you to manage DHCP and DNS libvirt’s network (via dnsmasq) for some local development. It allow to have a full provisioning workflow without having to install bind, tftpd and dhcpd.

This post is hugely inspired from Foreman 1.4 manual.

Libvirt configuration

The first thing to do is to define a persistent virtual network in libvirt.
Copy in a file named net-defintion.xml. You can off course change the network name, ip range, domain name, etc …

$ cat net-defintion.xml
<network>
 <name>default</name>
 <uuid>16b7b280-7462-428c-a65c-5753b84c7545</uuid>
 <forward mode='nat'>
 <nat>
 <port start='1024' end='65535'/>
 </nat>
 </forward>
 <bridge name='virbr0' stp='on' delay='0'/>
 <mac address='52:54:00:b2:fa:27'/>
 <domain name='fitzdsl.local'/>
 <ip address='192.168.122.1' netmask='255.255.255.0'>
 <tftp root='/tftp'/>
 <dhcp>
 <range start='192.168.122.2' end='192.168.122.254'/>
 <bootp file='pxelinux.0'/>
 </dhcp>
 </ip>
</network>

Then, you need to create and start the default network on libvirt:

# virsh net-define --file net-definition.xml
# virsh net-start default

We need to setup the TFTP directory (from Foreman manual for Fedora) :

mkdir -p /var/tftproot/{boot,pxelinux.cfg}
yum -y install syslinux
cp /usr/share/syslinux/{pxelinux.0,menu.c32,chain.c32} /var/tftproot
chgrp -R nobody /var/tftproot
find /var/tftproot/ -type d | xargs chmod g+s

Smart-Proxy configuration

We need now to configure a local smart-proxy to manage TFTP, DNS and DHCP:
We should now configure the local smart-proxy to use this new provider:
Set the following:

:tftp: true
:tftproot: /var/tftproot
:tftp_servername: 192.168.122.1
:dns: true
:dns_provider: virsh
:dhcp: true
:dhcp_vendor: virsh
:virsh_network: default

Finaly make sure your smart-proxy can have sudo rights :

Defaults !requiretty
foreman-proxy ALL=/usr/bin/virsh

Foreman configuration

First you need to add your proxy or refresh the feature list:

In Infrastructure:
New proxy : http://localhost:8443

Then you need to create a new domain and subnet :

  • Create a new domain : name it accordingly to your “domain name” on the net-defintion.xml.
  • Create a new subnet accordingly to you net-definition file.

In my case:

Name: Home
Network address: 192.168.122.0
Netmask: 255.255.255.0
Start IP Range: 192.168.122.2
Stop IP Range: 192.168.122.255
  • In Domains tab check the domain you just created.
  • In Proxies tab select your new proxy for DHCP, TFTP and DNS.

Create a new VM

When creating a new host, take care to select in “Virtual Machine” Tab on Network Interfaces:

  • Network Type => Virtual (NAT)
  • Network => “default”

You have now the ability to setup in local full provisioning environment. The only missing thing is that the PTR DNS record is not setup.

Great thanks to Lukas (@lzap) who implemented this new great feature !

Share

Foreman 1.3 has been released

What’s new in that release ?

Foreman 1.3 has just been released, let’s have a look to the content of that new version:

  1. The installer is now based on Kafo project. I didn’t test it because I always install Foreman with a git checkout
  2. The Hammer project (the Foreman CLI) is going on ! This is great because Foreman was lacking a good CLI at the beginning. However Core Team still warn that CLI is still limited, to be continued so.
  3. On Compute Resource level, the most wanted Amazon EC2 VPC support (Virtual Private Cloud) has been included. On top of that a first shot for GCE (Google Compute Engine) has been released. It’s now quite limited as it doesn’t support VMs that requires persistant disk creation.
  4. Spice support for Libvirt Compute Resource is now available
  5. And Foreman allows now to transform a VM seen like a BareMetal host in Foreman as … a VM associated to a Compute Resource!
  6. There is some changes also on API side. The API v2 is still ‘experimental’ but supports now : REMOTE_USER, smart classes, the network interface management, the power management and boot devices. The API v1 is still the default one.
  7. A new foreman-rake command exists now. The goal is to provide a generic wrapper for all Foreman’s related rake commands.

Translations have been improved. Foreman is now translated into 6 languages. Don’t hesitate to participate on Foreman’s Transifex.

Important before migration

The Reports and Facts format structure have changed with removal of Puppet from Foreman Core.

  • You have to change your external node scripts (/etc/puppet/node.rb) on your puppet master and use the new one that you can find there.
  • You also need to change your report upload script on your puppet master and use the new one

The removal of Puppet from Foreman Core is a really good thing. It will let now the support of other Configuration Management System such as Chef or CFEngine. It should be noted that a first Chef support has begin. You can find the project here.

The official release note and changelog can be found here.
Let’s migrate !

Share

Power management of Bare Metal servers with Foreman

Power management of bare-metal servers is a new feature that comes with Foreman 1.2. You will need to have deployed Foreman and smart-proxy to 1.2 to enjoy this.
With that feature you will be able to provide a way to start, stop and reboot servers directly from Foreman’s Webinterface or from REST API.
This has been tested on DELL servers configuring DRAC/

Smart-proxy configuration

On your smart-proxy, you need:

* The rubyipmi gem installed

# gem install rubyipmi

* ipmitool installed:

# apt-get install ipmitool

Configuration in Foreman

You need to edit the host you want to manage with BMC:

  • Go on foreman/hosts/server.example.com/edit
  • Click on  Network tab
  • Click on ‘Add Interface’
  • Type = ‘BMC’
  • You need then to get the MAC address of your BMC interface. Connect to your server (server.example.com in my example) in ssh and then using ipmitool command :
# ipmitool lan print

Foreman will automaticaly add a DNS record and a DHCP lease for your interface.
However, there is a bug in 1.2.0 where Foreman doesn’t append correctly the domain to the name of the card (see the open bug ). As a woraround, you can set in Name field the FQDN of your interface name (ie: server-ipmi.example.com)

  • Eventualy choose a user/password then save.

IPMI configuration

On the server, you still need to configure it to allow Foreman to use it :
You can use the following ipmitool commands :

# ipmitool lan set 1 ipsrc dhcp
# ipmitool lan set 1 access on
# ipmitool user set name 3 your_user
# ipmitool user set password 3 your_password
# ipmitool channel setaccess 1 3 callin=on ipmi=on link=on privilege=4
# ipmitool user enable 3

How to use that feature

On the host page, if all went well, you should be able to manage from Foreman the power of the server :

bmc

A new feature to integrate more deeply Foreman with all your servers. Foreman is getting better and better !

Share

Speed up Foreman with memcached

Even if I have Foreman running with a quite new version of apache and passenger, the speed
is not the first quality of that very good application.
That’s the reason why I tested a plugin written by Ohad Levy which add memcache support.
You can find the repo on github.
I installed it without problem in production on my Foremans 1.2.

I use git checkout to install Foreman, this may need adaptation
for you.
This method is quite similar to all plugin installation in Foreman : this plugin is only
a new gem to install.

You first need to install memcached on the same server :

# apt-get install memcached
# service memcached start

By default Foreman use a local memcached server.

# cd ~foreman
# echo "gem 'foreman_memcache', :git => \"http://github.com/ohadlevy/foreman_memcache.git\"" > bundler.d/memcache.rb
# bundle install --without postgresql sqlite test development --path vendor # you may have to adapt what options do you need with your bundle

Eventualy we need to restart Foreman :

# service apache2 restart 

That’s it, Foreman should respond quicker now !

Share

Howto integrate Puppet, Foreman and Mcollective

Since we deployed Foreman in production, we didn’t use the ‘Run puppet’ button
in Foreman’s interface because we run puppet with a crontab.

However Foreman 1.2 release changed that : now smart-proxy have
mcollective native integration.

This is how to setup that. I assume that you already have a working Foreman and Mcollective
setup.

In all your ‘puppet’ proxies you need to :
Install mcollective client and puppet plugin:

# apt-get install mcollective-client mcollective-puppet-client

You need to configure you mcollective client (/etc/mcollective/client.cfg). This configuration should be
quite similar to the one you have for your desktop.
You need then to grant the user foreman-proxy to run mcollective client :

# visudo 
Defaults:foreman-proxy !requiretty
foreman-proxy ALL = NOPASSWD: /usr/bin/mco puppet runonce *

In your proxy configuration :

:puppet: true
:puppet_provider: mcollective

Restart then your smart-proxy (I run it with apache/passenger):

# service apache2 restart

You should be able to test your new installation with a simple
curl command :

$  curl   -d "nodes=myserver.example.com" https://myproxy:8443/puppet/run

In order to be able to use the mcollective integration, I had to add in my mcollective daemon
configuration the following directive :

Dans /etc/mcollective/server.cfg

identity = myserver.example.com

Eventualy in Foreman settings, you
need to set ‘puppetrun’ directive to ‘true':

This should be good: you just need to click on ‘Run puppet’ button on your host page !

Share

How to run foreman-proxy with passenger

I recently decided to run my Foreman-Proxy daemon with Passenger instead of commonly used webrick.

As we will see, the setup is quite simple. I assume that you already have apache and passenger installed
(for Foreman, puppetmasted, …).

As I use Git for my setup, my smart-proxy is located in /opt, I let you fix your paths!
My apache configuration is (for apache2.4):

Listen 8444
<VirtualHost *:8444>
  ServerName foreman-proxy.example.com
  ServerAlias proxy1.example.com

  DocumentRoot /opt/smart-proxy/public

  RailsAutoDetect On
  PassengerTempDir /opt/smart-proxy/tmp
  AddDefaultCharset UTF-8
  HostnameLookups On

  SSLEngine on
  SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/proxy1.example.com.pem
  SSLCertificateFile /var/lib/puppet/ssl/certs/proxy1.example.com.pem
  SSLCACertificateFile /var/lib/puppet/ssl/certs/ca.pem

  <Directory /opt/smart-proxy/public>
     Require local
     Require ip 192.168.0.0/16 10.0.0.0/8
  </Directory>

  CustomLog ${APACHE_LOG_DIR}/foreman-proxy.example.com/access.log combined
  ErrorLog ${APACHE_LOG_DIR}/foreman-proxy.example.com/error.log
</VirtualHost>

I decided to use an other listenning port for apache, but you can use default 8443 port.

As you can see, the SSL configuration is done on apache level and not is smart-proxy anymore.

On proxy side configuration, it’s important to know, that “:trusted_hosts” directive raise a ‘500 Internal Error’.

The bug has been open there : http://projects.theforeman.org/issues/2259

Now, you only have to stop your webrick smart-proxy daemon and restart apache.

Be careful, if you changed your listenning port to update your smart-proxies configuration on Foreman.

Share

How to generate Puppet SSL certificate with “Alternative Name”

I needed to add DNS Alt name in order to setup a full SSL comunication between my 2 Foreman servers et their proxies.
My problem was that my Foreman servers are used in faillover (with a VIP) and the clients use a generic DNS record and not directly
their FQDN. This was a problem because the address didn’t match with the certificate’s CN.

In order to fix that, I seted up a Puppet certificate where CN is the FQDN of the server (ie: foreman1.example.com) and which have an
‘Subject Alternative Name’ with VIP address (ie: foreman.example.com).

This is really simple to do but not that easy to find on the internet:
You first need to revoke the certicate on the master and remove it on the client :
On the client (on Debian):

# rm -rf /var/lib/puppet/ssl

On the master:

# puppet cert clean foreman1.example.com

You should add to the client’s puppet.conf the following:

dns_alt_names = foreman.example.com

The you need to kick puppet on the client to force a new certificate generation and ask to the puppet master to sign it:

# puppet agent -t --report --pluginsync

On the master, you can see the certificate signing request and sign it:

# puppet cert list
  "foreman1.example.com" (SHA256) 2C:76:5B:85:67:28:1C:92:48:AA:10:22:44:C7:9B:A7:0D:9B:E2:A5:5F:10:71:87:B9:3F:46:E4:70:4B:43:6C (alt names: "DNS:foreman.example.com", "DNS:foreman1.example.com")
# puppet cert sign devshinken4.yzserv.com --allow-dns-alt-names

You now have a Puppet CA signed certificate with DNS Alt Name.

Share

Foreman migration without problems.

I just migrated my Foreman instances to 1.1 in production (I’ll writte later about nice new features on 1.1).

One of most important test I do before upgrading the production is the non regression of ENC output. What I mean is that I check that the new Foreman server sends the same YAML to the Puppet master during ENC lookup. I wrote a small ruby script (using the external controler script from Foreman community) wich compares YAML responses between 2 Foreman instances (ie: production and QA).

In order to support parameterized classes, Foreman changed a bit the YAML structure but this script supports this changement.
You can find it on my Github repo. You just have to change the 2 URLs of Foreman instances and set your login and password.

This script stop automaticaly if it founds a different node defintion between dev and production. This tool allow me to be more confident before a major Foreman’s migration.

Share

Fully automated deployement of webserver using Foreman

The aim of this post is to show the automation level for application server deployement on virtualization layer that is now possible with Foreman @ Yakaz.

This video shows how easy it is for us to create a new virtual machine and how Puppet manage it to fully configure the server to run our application. This video show the Compute Resource system that was introduced with Foreman 1.0.

Compared to deprecated virtualization support on Foreman before 1.0, the Compute Resources allow to :

  • Deploy new VMs with multiple NIC and disks
  • Direct access to VNC console on Foreman’s interface
  • ACL on Compute Resource use. This is usefull for us to be able to have an ‘open’ virtualization cluster for every developers that need VM and an isolated production cluster
  • VM power cycle management directly on Foreman
  • When a Compute Resource is deleted on Foreman, the VM is also deleted on virtualization cluster
For now, supported compute resources are: 
  • oVirt / RHEV-M
  • libvirt
  • Amazon EC2
  • VMware
  • Rackspace OpenCloud
As reminder, with ‘Unattended Installation’ feature, Foreman manage:
  • A and PTR DNS record for multiple domains
  • DHCP lease for the host
  • Puppet configuration with ENC
  • PuppetCA management : no need to sign and revoke certificates anymore
Once the VM is built, puppet configure it in order that it comply with configuration selected (in that case, our Webserver for Yakaz website)

This video shows that the ‘Operator time’ required for new server deployement is really reaaly short. On top of that, our Foreman integration give us a great flexibility and elasticity on our production cluster management.

Share

How to check Puppet run with Foreman and Nagios

I needed to make sure that Puppet was running smoothly on all production servers. For that purpose I needed to check 2 things :

– First that puppet was running every 30 minutes (I use cron and not Puppet daemon) . For that I simply use the nagios ‘check_file_age’ and I check the age of the “state.yaml” file. Here is my configuration on the command on Debian server:

/usr/lib/nagios/plugins/check_file_age -w 3780 -c 43200 -f /var/lib/puppet/state/state.yaml

– The first check make me sure that Puppet is running on a regular basis. However I am not sure that it run without problems. That’s the reason why I decided to use the Foreman Report status.

You can find my script on Github . To use it on a Debian server, you should install the dependencies :

# apt-get install libhttp-server-simple-perl libjson-perl
# wget http://search.cpan.org/CPAN/authors/id/M/MC/MCRAWFOR/REST-Client-243.tar.gz
# tar xvf REST-Client-243.tar.gz
# cd REST-Client-243
# make
# make install

To use it, it’s very simple:

$ /usr/lib/nagios/plugins/check_foreman_puppet_failure.pl -H webserver.example.com -F http://foreman.example.com -w 3 -c 5 -u username -p password

This command will check last reports of Puppet run. If the number of run with error state is greater than warning or critical then the nagios check will return the corresponding error.

Now, you can monitor your puppet run thanks to Nagios and Foreman !

Share