Check MK and Grafana

The graphs in Check MK may be quite nice but there is another way of taking this a step higher and make beautiful dashboards. You may even want to create public dashboards or put some info on wallboards.  There is a simple setting i Check MK that you may have missed. It looks like this:

I did not take any notice of it but once used it is fantastic. Ok, let´s start setting this up. I have used instructions on various places but mostly the documentation found here. Before starting make sure you have:

• A working Check MK Server
• An Ubuntu server with ssh access. Just install the basics.

Connect to your new server and update:

sudo apt-get update && sudo apt-get upgrade

Install what is needed. Answer no to remove the files

sudo apt-get install build-essential graphite-web graphite-carbon python-dev apache2 libapache2-mod-wsgi libpq-dev python-psycopg2

Configure Carbon

Edit the Carbon configuration file.

sudo vi /etc/default/graphite-carbon

Make the following change.

CARBON_CACHE_ENABLED=true

Start the service.

sudo service carbon-cache start

Install and configure PostgreSQL

By default Graphite will use SQLite. PostgreSQL is better.

Install PostgreSQL.

sudo apt-get install postgresql

Change to postgres user.

sudo su postgres

Create a database user. You will be prompted to supply a password. Remember this password as you’ll need it when configuring Graphite.

createuser graphite --pwprompt

Create the databases for Graphite and Grafana.

createdb -O graphite graphite
createdb -O graphite grafana

Then switch back to your user account. For example:

su - youruseraccountname

Configure Graphite

Update the database settings.

sudo vi /etc/graphite/local_settings.py

Update the database connection settings to use the PostgreSQL database. Use the password your specified for the graphite user created in PostgreSQL. For example:

DATABASES = {
'default': {
'NAME': 'graphite',
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'USER': 'graphite',
'PASSWORD': 'graphite',
'HOST': '127.0.0.1',
'PORT': ''
}
}

Within the same file, update the time zone. Uncomment the TIME_ZONE line and add the correct time zone which matches the server’s time zone.  I did got an error here when I initialized the database. In case you get that change this setting later. For example:

TIME_ZONE = 'Europe/Stockholm'

 

Within the same file, update the secret key to something long and complicated. Uncomment the SECRET_KEY line and add a random string of letters and numbers. For example:

SECRET_KEY = 'WH@T3v3RuW@nTG03$h3Re'

Initialize the database. You will be prompted to create a superuser account which will be used to access Graphite’s web interface. Leave the username as root and enter a password.

sudo graphite-manage syncdb

Configure Apache for Graphite

Apache is required for Graphite.

Copy Graphite’s Apache config template to Apache’s sites-available directory.

sudo cp /usr/share/graphite-web/apache2-graphite.conf /etc/apache2/sites-available/apache2-graphite.conf

Change Graphite’s port from 80 to 8080 (port 80 will be used by Grafana).

sudo nano /etc/apache2/sites-available/apache2-graphite.conf

 <VirtualHost *:8080>

Update which ports Apache is listening for.

sudo nano /etc/apache2/ports.conf

Add port 8080. For example:

Listen 80
Listen 8080

Disable Apache’s default site to avoid any conflicts.

sudo a2dissite 000-default

Enable Graphite’s virtual site.

sudo a2ensite apache2-graphite

Reload Apache to apply the changes.

sudo service apache2 reload

You should now be able to browse to Graphite’s web interface, http://servername:8080.

If you have configured the setting in Check MK and if it is working you should be able to see the host here now.

Install and configure Grafana

Add the following line to your /etc/apt/sources.list file.

deb https://packagecloud.io/grafana/stable/debian/ jessie main

Use the above line even if you are on Ubuntu or another Debian version. There is also a testing repository if you want beta or release candidates.

deb https://packagecloud.io/grafana/testing/debian/ jessie main

Then add the Package Cloud key. This allows you to install signed packages.

curl https://packagecloud.io/gpg.key | sudo apt-key add -

Update your Apt repositories and install Grafana

sudo apt-get update
sudo apt-get install grafana

Edit Grafana’s configuraiton file.

sudo nano /etc/grafana/grafana.ini

Update the database connection values to use PostgreSQL. For example:

[database]
type = postgres
host = 127.0.0.1:5432
name = grafana
user = graphite
password = graphite

Update the server configuration. For example:

[server]
protocol = http
http_addr = 127.0.0.1
http_port = 3000
domain = domain.com
enforce_domain = true
root_url = %(protocol)s://%(domain)s/

Update the security information. For example:

[security]
# default admin user, created on startup
admin_user = admin # default admin password, can be changed before first start of grafana, or in profile settings
admin_password = admin # used for signing
secret_key = SW2YcwsfssfsDSFDS8979869t

Enable anonymous viewing.

[auth.anonymous]
# enable anonymous access
enabled = true

Enable Apache’s proxy modules for reverse proxying. This will allow Apache to listen to port 80 and proxy the requests to Grafana’s webserver on port 3000.

sudo a2enmod proxy proxy_http xml2enc

Create an Apache configuration file for Grafana.

sudo nano /etc/apache2/sites-available/apache2-grafana.conf

Add the following to the file:

<VirtualHost *:80>
ProxyPreserveHost On
ProxyPass / http://127.0.0.1:3000/
ProxyPassReverse / http://127.0.0.1:3000/
ServerName segbggr01.white.local
</VirtualHost>

Enable the Grafana website in Apache.

sudo a2ensite apache2-grafana

Configure Grafana to run after boot.

sudo update-rc.d grafana-server defaults 95 10

Start the Grafana server.

sudo service grafana-server start

Restart Apache.

sudo service apache2 restart

Go to http://yourserver. If it shows Grafana it works. Also try the :8080 to Grahite again to make sure it works. This was the boring part. The next is the fun one.

Setting the datasource

In Grafana go to datasources.  For me it looks like this:

Press save & test and make sure it works. When I first installed Grafana via apt- get without adding the repository I ended up with an old version without any icon. So make sure you have installed the latest release. You can check this is the Grafana gui.

Dashboards with Check MK Data

Press the add row in a dashboard. Choose what type of panel you may want.

I choosed the graph. Edit the settings you want and save.

There are several cool panels that you may download from grafana. These are installed with the command below (the example is the pie chart) Restart Grafana after installing.

grafana-cli plugins install grafana-piechart-panel

In the end you may have a dashboard something like (it is just a demo I made quick) this with realtime data collected from Check MK.